Workload: Internal Copilot
(Assistive help over internal context — with explicit blast-radius limits on suggested or executed actions)
Intent
Help employees move faster inside an organisation’s own artefacts and systems — documentation, repositories, tickets, runbooks, or operational signals — while keeping autonomy mostly assistive or delegated. The primary outcome is better human work product, not unsupervised enterprise change.
Google’s agent-pattern guidance recommends starting with a single agent and refining core logic, prompts, and tools before adding architectural complexity. (docs.cloud.google.com) OWASP’s Excessive Agency category exists precisely because tool-using assistants can take damaging actions when outputs are unexpected or manipulated. (genai.owasp.org)
Typical requirements
- Context comes from private sources (repos, wikis, tickets, metrics) with clear tenancy and access control.
- Users expect suggestions they can accept, edit, or reject.
- Write risk varies by surface (comment in doc vs merge code vs change prod config).
- Team norms and secrets must not leak across projects or into prompts/logs carelessly.
- Evaluation includes usefulness and safety, not only answer fluency. (anthropic.com)
Default shape
User task (IDE, chat, ops console)
↓
Scoped context assembly
├── Allowed repo / doc / ticket slice
├── Optional retrieval over internal corpus
└── Tool reads via Tool Gateway
↓
Bounded copilot (assistive by default)
├── Candidate plan, draft, or command
↓
Epistemic gate
├── Policy / secret scanners / tests as applicable
└── User confirmation for side effects
↓
User accepts / edits OR gated tool write
Default control structure: one bounded agent per session, assistive autonomy. Prefer showing a draft or plan over silent writes. Raise autonomy only per action class, not globally. OWASP explicitly recommends reducing autonomy by requiring human review before consequential sends/writes. (genai.owasp.org)
Critical design decisions
1. Context assembly
| Option | Prefer when |
|---|---|
| Open files / selection / repo slice in context | Coding and doc editing copilots |
| Internal RAG over wikis and runbooks | Broad “how do we do X here?” questions |
| Live tools (CI, ticketing, metrics APIs) | Ops and incident assist |
| Unbounded “whole monorepo” stuffing | Almost never |
Default: just-in-time, least context necessary for the step; separate durable memory from task context. Anthropic frames context engineering as maximising signal within a finite attention budget, including progressive disclosure and avoiding drowning the agent in exhaustive but irrelevant material. (anthropic.com) Long stuffed contexts also suffer “lost in the middle” degradation when the needed evidence is buried. (arxiv.org)
2. Autonomy and write risk
| Option | Prefer when |
|---|---|
| Suggest only (user copies/applies) | High blast radius; early maturity |
| Delegated writes with user confirm (patch, comment, draft PR) | Clear review affordance in the product |
| Bounded autonomous writes in non-prod or sandboxes | Strong tests, rollback, and ACC |
Default: assistive. Treat prod config, IAM, data mutation, and force-push class actions as human-required unless a separate, reviewed ACC says otherwise. NCSC guidance stresses that when an LLM can call tools/APIs, prompt-injection impact equals worst-case direct access to those tools — so constrain privileges deterministically. (ncsc.gov.uk)
3. Memory
| Option | Prefer when |
|---|---|
| Session-only context | Default for most copilots |
| User preferences (style, language) | Low-risk durable settings |
| Team durable memory | Shared playbooks — versioned and auditable |
| Opaque long-term memory of everything | Avoid; becomes a leak and drift surface |
Default: session context + explicit, versioned knowledge sources. Do not silently promote chat into institutional memory. Anthropic distinguishes compaction and structured note-taking as deliberate persistence strategies — not accidental accumulation of everything said. (anthropic.com) Poisoned or stale memory/RAG stores are a recognised agentic threat surface. (csrc.nist.gov)
4. Blast radius controls
| Option | Prefer when |
|---|---|
| Read tools wide, write tools narrow | Standard internal assist |
| Environment separation (dev/stage/prod identities) | Any execution capability |
| Secret scanning before context/send | Repos and logs with credentials |
Default: distinct identities per environment; Tool Gateway enforces allowlists; never embed long-lived secrets in prompts. Least privilege on tools and downstream authorisation (not trusting the model to decide permission) are core Excessive Agency mitigations. (genai.owasp.org)
AAF review focus
| Lens | Why it pressures this workload |
|---|---|
| Context Optimization | Repo/doc noise vs missing constraints; secret leakage into context (anthropic.com; arxiv.org) |
| Security | Cross-repo access, supply chain of tools/skills, exfiltration via prompts (genai.owasp.org; ncsc.gov.uk) |
| Autonomy & Outcome Governance | Assistive vs delegated vs autonomous per action class (genai.owasp.org) |
| Operational Excellence | Versioning of prompts/skills, evals on real internal tasks (anthropic.com) |
| Reliability | “Looks right” patches that fail tests or break invariants |
Cost and Performance matter as context windows and tool round-trips grow; Sustainability follows wasted regeneration loops.
Dominant cross-pillar trades
No design maximises every lens. For this workload, the usual imbalances are:
- Cost × Accuracy × Speed: Frontier models + large repo slices buy suggestion quality; they raise Cost and can hurt Speed as context grows. Smaller models + tight file slices reverse the triangle. (platform.openai.com; anthropic.com)
- Security × Autonomy / Performance: Suggest-only or confirm-before-write buys Security; it spends Autonomy and perceived Speed (human is the bottleneck). (genai.owasp.org)
- Security × Context / Cost: Wide org-wide retrieval or prod credentials “for usefulness” expands exfil and injection impact; scoping tools and tenancy spends setup effort and sometimes answer coverage. (ncsc.gov.uk)
- Reliability × Performance: Requiring tests/scanners before apply improves Reliability; silent apply is faster and fails later.
- Context × Cost: Opaque durable memory can look efficient until drift, leaks, and rework dominate. (anthropic.com)
Typical starting bias for this workload: Security and Autonomy governance (assistive defaults) over raw Autonomy/Speed; tune Cost × Accuracy via model routing and just-in-time context, not whole-monorepo stuffing.
Common failure modes
- Silent writes to shared systems without confirmation. (genai.owasp.org)
- Whole-org retrieval without tenancy filters.
- Durable memory that stores secrets, PII, or stale decisions.
- One “super-copilot” identity with prod credentials. (ncsc.gov.uk)
- No Definition of Done beyond “user seemed happy.” (anthropic.com)
- Treating vendor coding-agent defaults as an ACC.
Trade-off log
- What autonomy level applies to each action class (suggest / confirm / autonomous)?
- How is context scoped (repo, space, tenant), and what is excluded?
- What is session vs durable memory, and who can edit durable knowledge?
- What is the blast radius if the model is wrong or manipulated?
- Where did we land on Cost × Accuracy × Speed, and which pillar pairs moved (e.g. Security↑ × Autonomy↓)? Why is that imbalance acceptable?
When this is not the pattern
- External customer support with identity, refunds, and case audit → Customer service chatbot.
- Public or enterprise knowledge Q&A without deep write risk → Knowledge / RAG assistant.
- Durable multi-step business workflow with system-of-record completion criteria → Workflow automation agent.
Sources
- Google Cloud — Choose a design pattern for agentic AI: https://docs.cloud.google.com/architecture/choose-design-pattern-agentic-ai-system
- Anthropic — Effective context engineering for AI agents: https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
- Anthropic — Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
- Liu et al. — Lost in the Middle: https://arxiv.org/abs/2307.03172
- OWASP — Excessive Agency: https://genai.owasp.org/llmrisk2023-24/llm08-excessive-agency/
- UK NCSC — Prompt injection is not SQL injection: https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection
- NIST CSRC (Sotiropoulos) — Agentic security emerging threats: https://csrc.nist.gov/csrc/media/presentations/2026/agentic-ai-emerging-threats,-mitigations,-and-cha/1.3-agentic_ai-sotiropoulos.pdf
- OpenAI — Model selection: https://platform.openai.com/docs/guides/model-selection