Workload: Knowledge / RAG Assistant
(Grounded answers from a corpus or curated knowledge base — with explicit bars before authority)
Intent
Deliver answers, summaries, or explanations that are grounded in approved knowledge sources, with citations or provenance the user (or downstream system) can inspect. The system may be conversational, but its primary outcome is trustworthy information retrieval and synthesis — not open-ended actuation.
Retrieval-augmented generation was introduced specifically to address parametric models’ limits on knowledge-intensive tasks and to improve provenance and updatability of knowledge. (arxiv.org)
Typical requirements
- A corpus, wiki, or document set exists (or will be curated) as the primary evidence base.
- Answers should refuse or escalate when evidence is missing or conflicting.
- Freshness expectations are known (static handbook vs frequently updated policy).
- Write actions, if any, are secondary (e.g. open a ticket) and tightly gated.
- Auditability of “what evidence supported this answer” matters. RAG’s original motivation includes providing provenance for decisions. (arxiv.org)
Default shape
User question
↓
Bounded assistant (single control loop)
├── Retrieve / load knowledge (RAG, wiki, API, or SQL)
├── Assemble context with provenance tags
├── Generate candidate answer
↓
Epistemic gate
├── Grounding / citation check
├── Policy and refusal rules
└── (optional) human review for high-risk topics
↓
Authoritative answer to user
└── Optional: Tool Gateway → low-risk side effect (ticket, feedback)
Default control structure: one bounded agent (or a deterministic retrieve → generate → validate pipeline). Anthropic’s prompt-chaining pattern explicitly allows programmatic gates on intermediate steps before continuing. (anthropic.com) Add stages only if retrieval, synthesis, and verification need different models, permissions, or acceptance criteria.
Critical design decisions
1. Knowledge access
| Option | Prefer when | Avoid when |
|---|---|---|
| RAG (retrieve chunks) | Large unstructured corpus; questions vary widely | Corpus is tiny; answers must be exact structured facts |
| Curated wiki / memory | Stable institutional knowledge you own and version | Content churns daily with no editorial process |
| Live API | Authoritative entity state in a system of record | Knowledge is narrative/doc-shaped, not API-shaped |
| SQL / datastore | Exact filters, joins, or metrics over structured data | Model would need free-form exploration of sensitive tables |
| Stuff-in-context | Small, stable pack that always fits the budget | Pack grows; collisions and cost dominate |
Default: RAG for broad doc corpora; curated wiki for owned playbooks; API/SQL for live operational facts. Do not use RAG as a substitute for a system of record. Anthropic’s context-engineering guidance favours the smallest high-signal context and just-in-time retrieval over stuffing exhaustive material into the window. (anthropic.com)
Watch-outs: Retrieval miss looks like confident hallucination; models often fail to use information placed in the middle of long contexts as context grows. (arxiv.org) Untagged retrieved text is an instruction surface for indirect prompt injection. (genai.owasp.org; ncsc.gov.uk) Free-form SQL/tool access expands blast radius if the model is coerced. (genai.owasp.org)
2. Grounding bar before authority
| Option | Prefer when |
|---|---|
| Cite retrieved spans before showing the answer as final | User-facing Q&A; compliance needs evidence |
| Schema-validated structured answer grounded in API/SQL rows | Operational facts (balances, statuses) |
| Refuse / ask clarifying question when confidence or coverage is low | High cost of wrong answers |
Default: candidate answer remains non-authoritative until grounding checks pass. Untested fluency is not a Definition of Done. Anthropic’s agent-eval guidance emphasises unambiguous success criteria and graders that check behaviour and outcomes, not merely plausible text. (anthropic.com)
3. Write tools
| Option | Prefer when |
|---|---|
| None (answer only) | Pure knowledge workload |
| Side-effect tools via Tool Gateway (ticket, feedback) | Explicit product requirement; least privilege |
| Broad write access | Almost never for this workload |
Default: no write tools. If you add them, treat them as a different risk class with separate approvals. OWASP’s Excessive Agency guidance recommends minimising extensions, enforcing least privilege, and requiring human approval for high-impact actions. (genai.owasp.org) The NCSC likewise argues that impact reduction depends on deterministic constraints on tools/APIs, not on hoping the model cannot be confused. (ncsc.gov.uk)
4. Verification
| Option | Prefer when |
|---|---|
| Provenance + citation coverage checks | Doc-grounded answers |
| Deterministic validators on structured fields | API/SQL-backed answers |
| Second-model or human review | High-stakes domains (legal, medical, safety policy) |
Default: retrieval provenance mandatory; human or secondary review only where risk justifies latency and cost. Prefer deterministic graders for safety-critical properties; use LLM-as-judge carefully and calibrate it. (anthropic.com)
AAF review focus
| Lens | Why it pressures this workload |
|---|---|
| Context Optimization | Retrieval quality, chunking, provenance, and context budget dominate outcomes (anthropic.com; arxiv.org) |
| Reliability | Definition of Done is grounded correctness, not fluent narrative (anthropic.com) |
| Cost | Naive RAG + large context + retries scales poorly (anthropic.com) |
| Security | Corpus poisoning / indirect injection via retrieved text; over-broad SQL/API tools (genai.owasp.org; ncsc.gov.uk) |
| Autonomy & Outcome Governance | When may an answer be treated as authoritative advice? |
Also review Performance (retrieval latency) and Operations (index freshness, eval harness for grounding).
Dominant cross-pillar trades
No design maximises every lens. For this workload, the usual imbalances are:
- Cost × Accuracy × Speed: Deeper retrieval, citation checks, and stronger models buy grounded accuracy; they raise token/tool cost and add latency. Thin context and a fast small model buy speed/cost; they raise miss and hallucination risk. (platform.openai.com; anthropic.com)
- Reliability × Cost: Provenance + refusal paths improve Definition of Done; they add validators, retries, and sometimes human review. (anthropic.com)
- Reliability × Performance: Live API/SQL grounding can be more accurate than stale RAG for operational facts, at the price of round-trip latency.
- Security × Cost / Context: Stuffing large top-k sets or broad SQL tools “to be safe” often worsens injection surface, cost, and mid-context miss rates. (arxiv.org; genai.owasp.org)
- Autonomy governance × Performance: Treating answers as non-authoritative until grounding passes slows time-to-answer; skipping the gate speeds replies and spends Reliability/Security.
Typical starting bias for this workload: Accuracy / Reliability and provenance over raw Speed; keep Cost bounded with just-in-time retrieval rather than context stuffing.
Common failure modes
- Treating RAG hits as authoritative without citation or coverage checks. (arxiv.org)
- Using the LLM as a database for facts that belong in API/SQL.
- Index drift: answers cite outdated policy while the wiki has moved on (RAG helps updatability only if the index/source of truth is operated). (arxiv.org)
- Context stuffing “top-k everything” until cost and noise degrade answers; long contexts with mid-list evidence are especially fragile. (arxiv.org)
- Adding write tools “just in case,” expanding blast radius without product need. (genai.owasp.org)
- No refusal path: the system always answers.
Trade-off log
Answer these after design and AAF review:
- Which knowledge-access option did we choose for each evidence class (docs, live facts, playbooks), and why?
- What must be true before an answer is shown as final (grounding bar)?
- What did we explicitly refuse to put in context or tools?
- Where did we land on Cost × Accuracy × Speed (model tier, retrieval depth, validators), and what did we explicitly deprioritise?
- Which pillar pairs moved (e.g. Reliability↑ × Cost↑, Security↑ × Autonomy↓)? What is the fallback when retrieval or validators fail?
When this is not the pattern
- Dialogue plus account changes, refunds, or case management → Customer service chatbot.
- Assistive coding or ops over private repos with varying write risk → Internal copilot.
- Multi-step process completion with verifiable end state in enterprise systems → Workflow automation agent.
Sources
- Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
- Liu et al. — Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172
- Anthropic — Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- Anthropic — Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Anthropic — Building effective agents: https://www.anthropic.com/engineering/building-effective-agents
- OWASP — Excessive Agency: https://genai.owasp.org/llmrisk2023-24/llm08-excessive-agency/
- UK NCSC — Prompt injection is not SQL injection: https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection
- OpenAI — Model selection: https://platform.openai.com/docs/guides/model-selection