A reproducible lab that secures an AI coding loop — an autonomous agent that writes and ships code, wrapped in input/output guards, an IaC security gate, optional observability, and a red-team harness. Every attack in this repo is run against models and services I host myself, in my own sandbox. No third-party systems are touched.
The output is a verifiable findings report you can attach to anything you publish.
Read the full write-up: linkedin/article.md — charts, the guard-delta numbers, the garak red-team results, and the gap-fix-re-run story in one place.
AI coding assistants turn an LLM into a person with commit access. That gives a coding loop four attack surfaces:
This lab shows — with before/after evidence — what happens to each surface when you wrap the loop in the right tools.
prompt ──> input guard (llm-guard, in-process) ──> Ollama model (qwen3-coder:30b)
│ injection/secret? quarantine, no generation
▼
generated output ──> output guard (llm-guard, in-process)
│ mask secrets/PII, block listed URLs
▼
IaC prompt? ──yes──> Checkov gate
│ │ fail → 1 repair prompt → re-check
no ▼
│ pass/fail recorded (no auto-promote)
▼ ▼
reports/findings-<run>.json + trace.json (always written locally)
│
optional: export to Langfuse (only if creds set in .env)
garak (separate, out-of-band): red-teams the raw Ollama model directly —
no guard in the loop — this is the untreated-risk baseline behind F3.
Scope note: earlier planning notes for this lab also named SonarQube, DefectDojo, and Grafana. None of the three have a working integration here, so they’re left out of the diagram and stack below — nothing on this page is claimed that didn’t actually run.
| Layer | Tool | Why |
|---|---|---|
| Input guard | LLM Guard (Python lib, in-process) | prompt injection, PII, secret scanning on the way in |
| Agent | Ollama-hosted local model | fully sandboxed; pinned, reproducible |
| Driver | scripts/run_demo.py |
baseline vs. guarded generation, per-prompt trace, one findings JSON per run |
| Output guard | LLM Guard (Python lib, in-process) | mask/block secrets & malicious code before commit |
| IaC gate | Checkov | fails agent-written Terraform on real policy violations; one repair round, then records pass/fail |
| Red team | garak | scans the raw coding model (no guard) for jailbreak/injection bypass |
| Observability | Langfuse | optional trace export of every guard decision + token spend, if credentials are set |
LLM Guard runs in-process (it’s a Python dependency, not a hosted service) and garak is
invoked directly as a CLI — nothing needs to be “booted.” The only external dependency
is an Ollama endpoint with the model pinned in .env (OLLAMA_MODEL) already pulled.
make setup # venv + pinned deps (garak, llm-guard, checkov)
make demo # run the baseline-vs-guarded loop, emit reports/findings-<run>.json
make report # render reports/findings.md
make trace-export # optional: push the run trace to Langfuse, if creds are set
# separately — the red-team pass behind finding F3:
.venv/bin/garak --model_type ollama --model_name qwen3-coder:30b \
--probes dan.DanInTheWild --parallel_attempts 4 --seed 42
docker-compose.yml/make up are left in the repo as an optional path for anyone who
wants LLM Guard running as a real hosted service instead of the in-process library — they
are not what produced the findings below.
Each run pins its own identity (see Reproducible metrics) so every claim in the report can be reproduced by anyone with the same manifest.
OLLAMA_MODEL in .env) recorded in every run’s manifest.prompts/ (not paste-in chat).llm-guard version is recorded per run (see Versions below);
the scanner set itself (InSecrets, PromptInjection, Sensitive) is fixed in
scripts/run_demo.py, versioned with the rest of the repo.llm-guard and Checkov versions are recorded automatically in each run’s
manifest (reports/findings-<run>.json); the garak version is recorded manually in
reports/garak-summary.md, since that’s a separate CLI run, not part of run_demo.py.Featured before/after findings, from real runs (full detail in reports/findings.md and
reports/garak-summary.md):
aws-key shape);
2 prompt-injection attempts blocked outright (P1-01, P1-05).resource_count=0) and still failed to parse after one repair
round — no plan was ever approved.dan.DanInTheWild, base model, no guard): 360 of 1280 judged slices
bypassed the mitigation-bypass detector — 28.1% attack success rate on the untreated
model. This is why the guard sits upstream of the agent, not after it.reports/hardening.md..env.example only; .gitignore protects yours.SECURITY.md for disclosure policy for this repo; report responsibly.MIT — see LICENSE.