# CocoaSkills RFC 0006 — Audit LLM Backends Status: draft Date: 2026-06-17 Author: Ivan Oparin Target: v0.8.0 RFC 0005 shipped the deterministic audit foundation: capability manifests, static detectors, verdict cache, trust pins, revocations, and the shared `gate_plans` install gate. v0.8.0 adds real extractor backends on top of that foundation without changing the deterministic policy layer. The key rule stays unchanged: backends detect and report findings. They do not decide whether a skill installs. The gate remains deterministic and local to `csk`. ## 1. Goals - Add a generic `command` audit backend that can call any local auditor process using a stable JSON stdin/stdout protocol. - Add a first-party `codex` backend implemented as a wrapper around Codex CLI non-interactive execution. - Keep v0.8 local-first: no cloud egress unless `audit.allow_cloud` and the selected backend both explicitly allow it. - Redact file contents before they are sent to a cloud-capable backend. - Keep local backends useful by allowing raw file bytes for local-only analysis. - Move backend timeout from a hardcoded `30` seconds into typed audit backend config. - Exercise fail-open and fail-closed behavior with real backend failures, not only mocked exceptions. - Preserve v0.7 semantics for `null` backend, static detectors, cache, pins, and revocations. ## 2. Non-Goals - Adding Claude Code backend. It remains a later RFC. - Adding remote service APIs or OpenAI SDK integration. v0.8 uses external processes only. - Letting a backend decide install policy. Backends only emit findings. - Executing skill code. Neither `command` nor `codex` may be handed a working directory that contains executable skill files as runnable files. - Implementing full provenance signatures. - Replacing static detectors. Static detectors always run first and remain the deterministic baseline. - Making cloud egress the default. Cloud backends are opt-in. ## 3. Motivation The v0.7 static audit catches deterministic patterns: undeclared subprocess execution, network hosts, filesystem escapes, secret reads, dangerous shell patterns, manifest hazards, and opaque artifacts. That is enough to make strict mode meaningful, but some skill risks are semantic: - prompt instructions that ask the agent to exceed the declared scope; - hidden operational behavior described in prose rather than code; - confusing command UX that can cause unsafe agent behavior; - capability intent that static analysis can see only partially. An LLM extractor can read `SKILL.md`, references, localized prompts, manifests, and code snippets and return structured findings. It must not be trusted as the gate. It is another detector. ## 4. Existing Foundation Reused | Component | v0.7 behavior | v0.8 use | |---|---|---| | `AuditBackend` protocol | Present, exercised by `null` | Extended with `command` and `codex` implementations | | `AuditRequest` | Carries files, capabilities, static findings | Adds redacted file-content guarantees | | `Finding` model | Stable | Backend findings use the same model | | `gate_plans` | Project and global install gate | Unchanged policy semantics | | Verdict cache | Keyed by hash/backend/model/prompt/ruleset | Backend identity and timeout do not affect hash, only backend/model/prompt/ruleset | | Source policy | Classifies source as internal/public | Used to decide raw vs redacted request content | | Redaction module | Scrubs evidence before output/cache | Extended to scrub file contents before cloud requests | | Canary | Static canary | Adds backend canary for `command`/`codex` | ## 5. Backend Model Backends implement the existing protocol: ```python class AuditBackend(Protocol): name: str cloud: bool def is_available(self) -> bool: ... def run_canary(self) -> bool: ... def extract(self, request: AuditRequest, *, timeout: float) -> tuple[Finding, ...]: ... ``` v0.8 keeps this protocol but makes the request content boundary explicit: ```python @dataclass(frozen=True) class AuditRequest: files: dict[str, bytes] capabilities: CapabilityManifest contract_reference: str response_schema: dict static_findings: tuple[Finding, ...] redacted: bool ``` Rules: - If `backend.cloud == false`, `files` may contain raw bytes. - If `backend.cloud == true`, `files` must contain redacted bytes only. - If redaction changes any file, the request has `redacted=true`. - Backend output must be parsed into `Finding`; malformed output is a backend failure, not a finding. - Backends may add findings but must not remove static findings. - Backend findings must set `verifiable=true` only when the finding is anchored to a concrete file/span or another deterministic piece of evidence. Findings that are useful but not checkable must set `verifiable=false`. - The install gate considers static findings and verifiable backend findings only. Unverifiable backend findings are kept in reports and cache as advisory facts, but a hallucinated high-severity finding must not block install. - Static detector findings are deterministic and gate-eligible. - Backends must not return policy decisions. Backend canary: - `run_canary()` sends a built-in malicious fixture through the backend before the backend is trusted for real skills. - The fixture includes prompt-scope escape text and undeclared runtime behavior. - The backend must return at least one expected high-severity, verifiable finding from the fixture. - Canary failure is a backend integrity failure and blocks in both advisory and strict modes. ## 6. Config Schema Delta The existing `audit.backends` object becomes typed. ```json "audit": { "enabled": false, "mode": "advisory", "fail_on": "high", "backend": "codex", "model": "gpt-5", "allow_cloud": false, "max_request_bytes": 1048576, "backends": { "local-command": { "kind": "command", "command": ["my-audit-tool", "--json"], "timeout_seconds": 30, "cloud": false, "env": {"MY_AUDITOR_MODE": "strict"} }, "codex-local": { "kind": "codex", "model": "qwen2.5-coder:32b", "profile": "local-audit", "oss": true, "local_provider": "ollama", "timeout_seconds": 60, "cloud": false }, "codex-cloud": { "kind": "codex", "model": "gpt-5", "timeout_seconds": 60, "cloud": true } } } ``` Field rules: - `kind`: required, one of `null`, `command`, `codex`. - `timeout_seconds`: optional positive number, default `30`, maximum `300`. - `cloud`: optional boolean, default `false`. - `model`: optional string. Backend-specific model overrides global `audit.model`. - `max_request_bytes`: optional positive integer, default `1048576`, maximum `10485760`. This is the maximum serialized request payload that may be sent to a backend. - Unknown fields are config errors. `command` fields: - `command`: required non-empty list of strings. - `env`: optional object of string keys and string values. - `cwd`: optional path. Defaults to a temporary empty directory. `codex` fields: - `model`: optional string. - `profile`: optional Codex profile name. - `oss`: optional boolean. If true, pass `--oss`. - `local_provider`: optional string. If present, pass `--local-provider`. - `sandbox`: optional string, default `read-only`. v0.8 rejects anything except `read-only`. - `approval_policy`: optional string, default `never`. v0.8 rejects anything except `never`. - `extra_args`: optional list of strings for forward-compatible local experiments. Disallowed when `cloud=true` and strictly validated in all modes. - If `cloud=false`, `oss=true` and `local_provider` are required. A Codex backend cannot be treated as local unless the config names the local provider. - If `cloud=true`, `oss` and `local_provider` are rejected. Cloud rules: - If selected backend config has `cloud=true`, `audit.allow_cloud` must be true. - If selected backend config has `cloud=false`, file contents are not redacted for egress, but evidence redaction still applies before cache/output. - If source classification is not `public` and backend is cloud, fail closed unless the backend config explicitly sets `allow_internal_sources: true`. This field is not accepted in v0.8. It is reserved for a later policy review. ## 7. Backend Selection Current v0.7 config has `audit.backend = "null"`. v0.8 resolves a backend in this order: 1. `audit.backend` names an entry in `audit.backends`. 2. If not found and value is one of built-ins (`null`, `command`, `codex`), use built-in defaults. 3. Otherwise config error. Built-in defaults: ```json { "null": {"kind": "null", "timeout_seconds": 30, "cloud": false} } ``` There is no built-in default for `command` because it needs an executable. There is no built-in default for `codex` because a safe local Codex setup must name `oss=true` and `local_provider`, while a cloud Codex setup must opt into `cloud=true` and pass source-policy checks. ## 8. Command Backend The command backend is a local process protocol. Invocation: ```text ``` Input on stdin: ```json { "schema_version": 1, "skill": "skill-gitlab", "source": "skill-gitlab", "commit": "abc123", "content_sha256": "sha256:...", "capabilities": { "...": "..." }, "static_findings": [ { "...": "Finding payload" } ], "files": { "SKILL.md": {"encoding": "utf-8", "content": "..."}, "scripts/gmr": {"encoding": "base64", "content": "..."} }, "redacted": false, "contract_reference": "...", "response_schema": { "...": "JSON schema" } } ``` Output on stdout: ```json { "schema_version": 1, "findings": [ { "id": "llm.prompt.scope-escape", "surface": "prompt", "category": "capability-escalation", "severity": "high", "location": {"file": "SKILL.md", "span": [12, 12]}, "evidence": "Prompt asks agent to bypass review policy", "confidence": "medium", "verifiable": true, "capability_violation": { "capability": "prompt_scope", "declared": "Manage merge requests", "observed": "Bypass merge request policy" } } ] } ``` Process rules: - Exit code `0`: parse stdout as response. - Non-zero exit: backend failure. - Timeout: backend failure. - Invalid JSON: backend failure. - Unknown response fields: backend failure. - Findings with invalid enum values: backend failure. - Stderr is captured and included in the backend failure message after redaction and truncation. - The command runs in a temporary empty directory unless `cwd` is configured. - `csk` never writes skill files to the command cwd. ## 9. Codex Backend The Codex backend is a first-party adapter over Codex CLI non-interactive mode. Base command: ```text codex exec --sandbox read-only --ask-for-approval never --cd \ --skip-git-repo-check --ephemeral --ignore-rules \ --output-schema --output-last-message - ``` Optional arguments: - `--model ` from backend config or global `audit.model`. - `--profile ` when configured. - `--oss` when `oss=true`. - `--local-provider ` when configured. Hard constraints: - `--sandbox` must be `read-only`. - `--ask-for-approval` must be `never`. - `--search` is never passed by `csk`. - `extra_args` are appended only after validation. The parser rejects any token that matches one of these option names, uses `--option=value` for one of these option names, or starts with the listed unsafe prefix `--dangerously-`: - `--sandbox` - `--ask-for-approval` - `--search` - `--cd` - `--output-schema` - `--output-last-message` - `--full-auto` - `--dangerously-` - `--add-dir` - `--config` - `-c` - `--profile` - `--model` - `--oss` - `--local-provider` - `--ephemeral` - `--ignore-rules` - `--skip-git-repo-check` - `--resume` - `--experimental-resume` - `--enable` - `--disable` - Prompt content is passed on stdin using `-`, not as a process argument. - `--ephemeral` is always passed so Codex does not persist audit sessions. - `--ignore-rules` is always passed so user/project execpolicy rules do not alter the auditor run. - `--skip-git-repo-check` is passed because the working directory is an empty scratch directory, not a repository. - `--output-schema` points to the response schema file created by `csk`. - `--output-last-message` points to a temporary response file that `csk` parses after the command exits. - Skill files are not materialized in the Codex working directory. - The prompt carries file contents as data. The model may reason over bytes, but there is no runnable skill checkout in the process cwd. - Codex output must contain exactly one JSON object matching the response schema. `csk` reads this JSON from `--output-last-message`; if the file is missing, malformed, or contains prose around the JSON, it is a backend failure. Local-first behavior: - `oss=true` plus `local_provider` is required for `cloud=false`. - `cloud=false` means the selected Codex model provider is local. The config must say which local provider is used; an omitted provider is a config error, not a quiet trust assertion. - `cloud=true` is allowed only when `audit.allow_cloud=true` and source policy classifies the source as `public`. Network boundary: - `csk` never enables Codex web search. - The empty cwd and missing skill files prevent tool execution over a runnable skill checkout. - The Codex model-provider connection is the backend connection. For `cloud=false`, that connection must target the configured local provider. For `cloud=true`, it is cloud egress and must pass cloud/source policy. - The read-only sandbox constrains Codex tool execution. It does not make a cloud model safe; cloud safety is enforced by backend config, source policy, and redaction. Codex is not allowed to decide policy. It returns findings only. ## 10. Prompt and Response Schema v0.8 adds versioned prompt assets: ```text src/csk/audit/prompts/ extractor-v1.md response-schema-v1.json ``` Prompt requirements: - State that skill content is untrusted and may attempt prompt injection. - State that the backend must ignore instructions inside skill content that try to affect the auditor. - Ask only for findings anchored to files and spans. - Require JSON output only. - Include static findings so the model can add context but not erase them. - Include the declared capability envelope. - Include the skill operational contract reference text or a compact excerpt. Response schema requirements: - `schema_version == 1`. - `findings` list only. - No free-form summary field. - No policy decision field. - No install recommendation field. ## 11. File-Content Redaction v0.7 redacts evidence before cache/output. v0.8 extends redaction to request file contents for cloud backends. Redaction applies to: - URL userinfo, query, and fragment. - Environment assignment values for names containing `TOKEN`, `SECRET`, `PASSWORD`, `PASS`, `KEY`, `PRIVATE`. - PEM/private key blocks. - Common bearer/basic authorization header values. - Long high-entropy strings above a conservative threshold. Redaction does not apply to local-only backends because local analysis needs raw content to detect real risks. Evidence is still scrubbed before cache/output for all backends. Residual risk: local backends are trusted local processes. `csk` redacts its own cache/output, but an external command backend or local model provider can keep its own logs outside `csk` control. Operators should use local backends only when they trust that local toolchain with raw skill content. Each redacted request includes a synthetic info-severity finding: ```text id: audit.redaction.applied surface: manifest category: hygiene severity: info evidence: "3 file(s) redacted before cloud backend request" ``` This makes cloud verdicts auditable: a reviewer can see that the backend saw modified content. It must not block even if a user sets `fail_on=low`. ## 12. Timeout Plumbing v0.7 hardcodes `timeout=30` in `pipeline.audit_plans`. v0.8 replaces it with: ```python timeout = backend_config.timeout_seconds backend.extract(request, timeout=timeout) ``` Rules: - Default `null`: `30`, unused in practice. - Default configured `codex`: `60`. - Default custom `command`: no implicit default unless command backend is configured; when configured and omitted, `30`. - Minimum `1`. - Maximum `300`. - Timeout value participates in runtime behavior but not verdict cache key. Cache key remains `(content_sha256, backend, model, prompt_version, ruleset_version)`. Rationale: changing timeout should not invalidate a valid existing verdict. Changing prompt or detector rules must bump `PROMPT_VERSION` or `RULESET_VERSION`. ## 13. Request Size and No Truncation The backend request builder must never silently truncate files, drop files, or audit a partial skill snapshot to fit a size budget. Rules: - `audit.max_request_bytes` is evaluated against the serialized request payload that would be sent to the selected backend. - For cloud backends, the check happens after file-content redaction because that is the payload that would leave the machine. - For local backends, the check happens on raw request content. - If the payload is larger than the limit, backend extraction is skipped. - The audit emits a verifiable high-severity finding: ```text id: audit.request.too-large surface: manifest category: audit-incomplete severity: high evidence: "Audit request is 2145728 bytes, limit is 1048576 bytes" ``` Policy handling is then normal: advisory mode warns and proceeds; strict mode blocks before writes. `csk audit` reports the finding. No cloud request is made for an oversize payload. ## 14. Failure Semantics The v0.7 policy remains: | Scenario | advisory install gate | strict install gate | `csk audit` | |---|---|---|---| | static canary failure | block | block | error | | backend canary failure | block | block | error | | backend unavailable | warn + proceed | block | error | | backend timeout | warn + proceed | block | error | | backend invalid output | warn + proceed | block | error | | request too large | warn + proceed | block | finding | | cloud not allowed | block | block | error | | source not cloud-eligible | block | block | error | Why cloud policy blocks even in advisory: it protects confidentiality, not skill quality. Proceeding without backend analysis is acceptable in advisory; sending private content to a disallowed backend is not. Verifiability rule: - Static findings and backend findings with `verifiable=true` participate in `policy.decide`. - Backend findings with `verifiable=false` are cached and reported but excluded from blocking decisions. ## 15. Cache Semantics Backends are part of the verdict cache key through `backend.name` and `model`. Examples: - `null-none-p1-r1.json` - `command-local-command-p1-r1.json` - `codex-qwen2.5-coder-32b-p1-r1.json` Cache hit behavior: - Static detector execution is skipped, as in v0.7. - Backend execution is skipped. - Policy decision is recomputed from cached findings, current mode, trust pins, and revocations. - Revoked hash/source still blocks a cached pass verdict. Backend config fields such as timeout, command path, and environment do not enter the cache key. If they change backend behavior materially, the operator must use a new backend name or csk must bump prompt/ruleset versions. ## 16. CLI Surface No new top-level commands are required. Existing: ```text csk audit [target] [--all] [--global] [--json] csk install [target] --audit[=advisory|strict] csk global install --audit[=advisory|strict] ``` v0.8 may add backend override flags: ```text csk audit --backend codex --model qwen2.5-coder:32b csk install --audit=strict --audit-backend codex ``` If added, CLI overrides are process-local and never persist to `~/.cocoaskills/config.json`. ## 17. Module Layout New files: ```text src/csk/audit/ backend_config.py # parse typed backend config redaction.py # extend to file-content scrubbing prompts/ extractor-v1.md response-schema-v1.json backends/ command_backend.py codex_backend.py ``` Modified files: ```text src/csk/config.py # validate typed audit.backends src/csk/audit/pipeline.py # backend config resolution + timeout plumbing src/csk/audit/trust.py # prompt/model cache key normalization if needed src/csk/cli.py # optional backend/model override flags tests/ # integration tests for command/codex paths ``` ## 18. Test Surface Release-blocking tests: - Config: - parses command backend config; - parses codex backend config; - rejects unknown backend fields; - rejects invalid timeout values; - rejects cloud backend when `allow_cloud=false`; - rejects Codex `cloud=false` without both `oss=true` and `local_provider`; - rejects Codex `cloud=true` with `oss` or `local_provider`; - rejects unsafe Codex `extra_args`; - rejects invalid `max_request_bytes` values. - Command backend: - sends valid JSON request to a fixture command; - parses valid findings; - redacts stderr in backend failure messages; - non-zero exit is backend failure; - timeout is backend failure; - backend canary failure blocks advisory and strict gates. - Codex backend: - constructs `codex exec` command with `--sandbox read-only`, `--ask-for-approval never`, `--ephemeral`, `--ignore-rules`, `--skip-git-repo-check`, stdin prompt, schema file, response file, and an empty temp cwd; - passes model/profile/local provider flags from config; - never passes `--search`; - rejects unsafe sandbox/approval config; - rejects dangerous `extra_args` even when `cloud=false`; - parses JSON-only output from a fake codex executable. - Redaction: - cloud request receives scrubbed file contents; - local request receives raw file contents; - cache/output never contains redacted secrets; - redaction finding is present with `severity=info` when file contents changed. - Gate behavior: - advisory install with command backend timeout warns and proceeds; - strict install with command backend timeout blocks before writes; - advisory install with cloud backend disallowed blocks before egress; - strict install ignores unverifiable backend findings for blocking decisions; - strict install blocks on verifiable backend findings above `fail_on`; - oversize backend request produces `audit.request.too-large` and never sends a partial request; - project and global install paths share the same behavior. - Cache: - command backend verdict cache hit skips command execution; - codex backend verdict cache hit skips codex execution; - revocation still blocks cached pass verdict. Recommended manual tests: - `codex` local provider smoke with an internal skill and `allow_cloud=false`. - `codex` cloud profile smoke only against a public fixture skill. - Verify no audited skill files are written into Codex cwd. - Verify shell history/log output does not contain raw secret-like strings. - Verify the configured local provider can receive the model request while Codex tool execution remains sandboxed in an empty cwd. ## 19. Implementation Sequence 1. Backend config parser and tests. 2. File-content redaction primitives and request-building tests. 3. Command backend with fixture command integration tests. 4. Timeout plumbing through `pipeline.audit_plans`. 5. Codex backend command construction and fake executable integration tests. 6. CLI backend/model override flags if still needed after config support. 7. README and `docs/audit-design.md` update pointing from v0.7 foundation to RFC 0006. 8. Release v0.8.0 after command backend and fake Codex tests are green. ## 20. Acceptance Checklist - `audit.backends` typed config is strict and round-trips. - `command` backend can produce findings from a local fixture command. - `codex` backend can be tested with a fake `codex` executable. - Cloud-capable requests redact file contents before process invocation. - Local-only requests preserve raw file contents. - Timeout is configurable and covered by tests. - Request size limits never cause silent truncation. - Advisory/strict fail-open/fail-closed behavior is tested on real subprocess failures. - Unverifiable backend findings are report-only and cannot block install. - Codex local configs must name an explicit local provider. - Dangerous Codex argument overrides are rejected. - Static detectors still run before backend extractors. - Backend canary failures block before real skill audit. - Backend findings are cached and policy is recomputed on cache hit. - No v0.7 `null` backend behavior regresses. ## 21. Open Questions 1. Should `codex` backend be hidden behind an explicit feature flag until a real local-provider smoke test is part of CI or release validation? 2. Should cloud backends ever be allowed for internal sources through an explicit per-source grant, or should that remain out of scope until provenance and policy review are stronger? 3. Should command backend stderr be stored in verdict metadata after redaction, or only surfaced on failures? ## 22. Out of Scope for v0.8 - Claude Code backend. - Signature/provenance verification. - N-of-M backend consensus. - Per-finding human grants beyond the legacy pin semantics already in v0.7. - Remote backend services. - Automatic installation of backend tools. - Parallel backend extraction. v0.8 may execute backend extraction sequentially; verdict cache handles the steady-state cost.