Skip to content
Daniel Kliewer

The Skills Map for Building and Deploying AI Applications

By
9 min read
The Skills Map for Building and Deploying AI Applications

The single most useful thing someone handed me this year is a one-page map of the skills that make up building and deploying AI applications. It was built by analyzing a large number of job postings, structured expert interviews, and survey responses — so it reflects what the field actually rewards, not what a single company's blog thinks is fashionable.

The map is worth sitting with. AI Engineering sits at the top, and one of its four branches — Building and deploying AI applications — expands into six competencies:

  1. LLM foundations
  2. Grounding models with data
  3. Building agentic systems
  4. Evaluation-driven development
  5. Operating in production
  6. Machine learning foundations

That's the map. What follows is my field guide to each node — what the skill means, where to go to actually build it, and how the pieces fit together. I've linked the primary sources and tools I trust so you can go deeper on any branch.

The one idea underneath all six

Before the branches: the difference between AI applications and ordinary software is that an AI application's output is less predictable. You don't know in advance what an LLM will say, or what a supervised model will predict. That uncertainty changes the shape of the work.

Traditional software is planned. AI software is iterative. You build a piece, examine it, and decide what to try next — a sequence of steps driven by the intermediate results. Skilled AI engineers are good at that loop:

The core loop: build, examine, evaluate, decide, repeat

The six competencies are the toolkit you reach into at every step of that loop. Being able to skillfully decide what to do next is what lets you build reliable systems on top of unreliable components. That's the whole game.

1. LLM foundations

Understanding how large language models tokenize input and generate output is what lets you predict when to count on them and when they'll fail. This is not trivia — it's the difference between guessing and reasoning about:

  • tokenization and context windows — what fits, what gets truncated, and the cost of a cache miss vs. a cache hit (OpenAI's tokenizer guide and Anthropic's prompt engineering overview are the canonical practical references; Lilian Weng's survey is the deep one);
  • knowledge cutoff and reasoning effort — knowing a model can't know what it wasn't trained on, and how to spend inference budget;
  • sampling parameters — temperature, top-p, and why "more creative" is rarely what a production system wants;
  • special features — tool calling, multimodal input, and when a smaller specialized model beats a giant general one.

The Transformer architecture is the substrate for all of it (Vaswani et al., 2017 — Attention Is All You Need). When you understand the foundations you can make the real calls: which model, or which mix of models, and whether a specialized technique like fine-tuning or self-hosting is worth it (Hu et al., 2021 — LoRA; Dettmers et al., 2023 — QLoRA).

I've written about this from the sovereignty angle — when the model is local and you control the weights, "LLM foundations" becomes an operational discipline, not a vendor dependency. See Context Engineering: the real full-stack paradigm and The Model Is Not the Product.

2. Grounding models with data

LLMs need good input context to produce useful output. Vector-search RAG was the early answer, but the toolkit has grown well past it. The actual decision is what to include in the prompt versus what to let the model retrieve on demand, and which representation fits the data and the query:

Choosing a data representation for grounding

Whatever you pick, you still have to turn documents — text, PDFs, HTML, images — into LLM-ready inputs, and engineer pipelines that keep that data clean and fresh. Stale grounding is worse than no grounding. For the framing of "RAG is one branch, not the tree," Huyen Chip's GenAI Platform design is the clearest industry write-up, and her AI Engineering field guide is worth the read alongside the skills map.

3. Building agentic systems

Agentic systems run from workflows that chain a fixed sequence of LLM calls to a harness that lets the model pick its own next step. The engineering decisions:

The field is moving fast — voice agents, computer-use agents, generative UI. Each is a specialization of the same harness. I've been building these fully local and giving local systems hands via a real execution environment.

4. Evaluation-driven development

This is the one I'd pick if I could only keep a single skill. The engineers who are genuinely good at building AI systems are the ones who can drive a disciplined evals + error-analysis loop. It focuses effort on directions that actually pay off, instead of random thrashing.

Building good evals is its own deep skill:

  • exploratory analysis of traces and outputs, combined with product and business judgment, to decide what to measure;
  • the menu of eval types — deterministic (code-based) checks, LLM-as-a-judge, and human-in-the-loop — and when each applies (DeepEval, DSPy for programmatic optimization);
  • evaluating your evals — so the measurement itself keeps improving.

The right approach varies by project and by stage of the project, which is why it's hard to master. But once you have it, development becomes systematic rather than random — exactly the loop in the first diagram.

5. Operating in production

Operating AI software differs from traditional software because of its unpredictability, cost, and latency. The production skill set:

  • observability — understand real-world performance, detect drift, and respond fast to model failures and adversarial prompt injection (Arize Phoenix, Evidently AI for drift and evaluation);
  • statistical CI/CD — regression testing calibrated to the risk of a mistake, not a fixed suite;
  • cost and latency optimization — model-choice optimization, distillation, fine-tuning, and workflow simplification when you have many users.

This is where "the loop is the product" becomes literal: if the system runs continuously, observability is the operating system (If the loop is the product, observability becomes the OS — when that post is live; the theme runs through the whole observatory series).

6. Machine learning foundations

Modern LLMs are built with machine learning — supervised and reinforcement learning. Everyone I know who is good at building with LLMs also understands ML and deep learning at some depth, and many applications still need a model someone else trained or one you train yourself.

Foundations here mean knowing the popular models and their tradeoffs (accuracy, training speed, inference speed), and how to engineer the data to train and evaluate them. The mental frameworks — bias/variance, error analysis, engineering your data — are the same ones that let you navigate uncertain output everywhere else on this map:

Labeling and data-curation tooling like Label Studio and serving layers like MLServer round out the practical stack.

How to use the map

The map isn't a checklist to finish — it's a diagnostic. When a build is stuck, the six nodes tell you which missing skill is the bottleneck:

  • output is unpredictable in a way you can't reason about → LLM foundations.
  • the model keeps missing facts → grounding with data.
  • the workflow is brittle or can't scale to open-ended tasks → agentic systems.
  • you're changing things and hoping → evaluation-driven development.
  • it works in the demo and dies at scale → operating in production.
  • you can't train or tune the part that matters → ML foundations.

Every bit you learn makes the next system better. And the complement that holds the whole map together is plain software engineering — the discipline that turns a promising prototype into something safe to run. That's the next letter.


Sources & further reading

Related research

Jul 14, 2026

Synthesizing Memory with Agent: A Local-First Architecture for Persistent AI State

The prevailing paradigm in AI agent development treats memory as an external retrieval service, a failure mode that fragments intelligence across the model, the context window, and the vector store.

Oct 23, 2026

LLMjacking: Building the 'Skynet Mode' Recursive Attacker

A deep dive into 'LLMjacking'—where autonomous agents hijack exposed LLMs to drive their own attacks. We deconstruct the architecture of recursive agents, 'Skynet mode' autonomy, and how a stolen brain becomes the primary attacker.

Oct 23, 2026

Operation Ouroboros: The Recursive Intelligence Parasite

What happens when an attacker stops treating language models as individual services and starts treating them as a distributed pool of reasoning capacity? What happens when exposed inference endpoints, autonomous agents, poisoned retrieval systems, and compromised cloud identities become components of a single operational architecture?

Jul 15, 2026

The Sovereign Knowledge Compiler: Compile-Time Memory for Local-First AI Agents

A revised architecture for agent memory that treats cognition as something compiled once into inspectable artifacts rather than retrieved fresh on every query — grounded in how OpenAI's Agents SDK, Mem0, and Hindsight are already converging on distillation, and extended with decay, conflict resolution, and CRDT-based device sync.

Jul 14, 2026

The Recursive Research Compiler: Turning Compile-Time AI Inward with the Knowledge Compiler SDK

A case study in Compile-Time AI: how the Knowledge Compiler SDK turns Markdown into inspectable intermediate representations, and how pairing it with an agent like Hermes lets a codebase compile its own next generation of passes.

Jul 5, 2026

Retrieval Architecture: Memory Systems That Compound

Memory systems and retrieval architecture for sovereign AI. Sovereign Memory Bank, Dynamic Persona MoE RAG, Objective05, and GraphRAG — the subsystems that make retrieval compound over time.

More from the archive