Command catalog contracts

Quirl has one composed command-intelligence graph but more than one source and storage lifecycle. Keeping those roles distinct prevents a generated artifact or an imported external fact from replacing an executable interface contract.

The native KDL database is not the CLI intelligence cache. It is a build artifact that contributes external command facts to composition. The CLI cache continues to hold the exact composed schema-v4 snapshot, normalized query rows, discovery state, and optional embeddings described by ADR 0021. Legacy catalog JSON v2/v3 exists only as a read-only migration input for that cache. KDL does not replace Catalog::builtin(), and neither SQLite database is human-authored.

The accepted trust and ownership decision is ADR 0024.

Strict KDL native command schema

One UTF-8 document contains exactly one catalog root, exactly one provenance child, and at least one root command. The following is a schema example, not a claim that a particular executable ships in the curated corpus:

catalog "native-tools" {
    provenance author="Example project" license="MIT" revision="v1.2.3" source="https://example.invalid/native-tools"

    command "copy" summary="Copy paths" description="Copy a source path to a destination path." {
        alias "cp"
        intent "duplicate a file"
        platform "linux"
        platform "macos"
        flag "--recursive" short="-r" summary="Copy recursively" description="Descend into source directories." {
            platform "linux"
        }
        flag "--target" value="directory" summary="Select a target" description="Place copied paths in this directory." action="directories"
        argument "source" summary="Source path" description="Read data from this path." required=#true action="files"
        argument "destination" summary="Destination path" description="Write data to this path." action="files"

        command "status" summary="Show copy status" description="Report the current bounded copy state." {
            intent "inspect copy progress"
        }
    }
}

KDL node and value type annotations are not part of the schema. Strings must be quoted KDL strings; booleans use #true or #false. Every retained string must be nonempty after trimming, contain no control character, and fit the configured UTF-8 byte limit.

Catalog and provenance

catalog takes one positional string and no properties. Its value is a stable catalog identifier made from lowercase ASCII letters, digits, hyphens, and underscores; its first byte must be a lowercase letter or digit. It requires a child block.

provenance takes no positional values or child block and requires exactly these four string properties:

The compiler stores these fields in the exact typed snapshot and the normalized provenance table. Provenance is attribution, not executable authority and not a substitute for required license notices.

Commands, subcommands, and aliases

Every command takes one positional name, required summary and description string properties, and an optional child block. A nested command is a subcommand; full paths are formed by joining canonical ancestor names with one ASCII space.

Command names and aliases use the catalog identifier grammar: lowercase ASCII letters, digits, hyphens, or underscores, beginning with a lowercase letter or digit. An alias takes exactly one such string and has no properties or child block. Names and aliases must be unique across one sibling scope: an alias may not collide with its command's canonical sibling name, another alias, or a different sibling command. Normalized full command paths must also be unique.

Each command may contain only these child nodes:

summary is concise completion- and list-facing text. description is the longer behavioral contract shown by detailed consumers and included in search. Both are required on commands, flags, and arguments. The compiler does not derive one from the other and does not interpret markup.

Flags

A flag has exactly one positional canonical spelling. A long spelling starts with --; its nonempty remainder uses lowercase ASCII letters, digits, and hyphens and begins with a lowercase letter or digit. A POSIX short-only spelling is - followed by one ASCII letter or digit. A Windows spelling starts with / and then uses one or more ASCII letters, digits, or hyphens. Flags may contain only platform "selector" children. Omission inherits the owning command's effective platform set. A declared flag scope is intersected with that command and must remain nonempty.

Canonical and short-alias spellings share one uniqueness namespace within a command on overlapping platforms. The same spelling may be declared more than once only when every declaration has a disjoint effective platform scope; this allows GNU and macOS meanings to coexist without leaking either variant onto the other host. The schema does not currently express conflicts, static enum values, defaults, deprecation, types, or dynamic providers; none may be inferred from upstream syntax.

Positional arguments

An argument has exactly one positional identifier, no child block, and the following properties:

Argument identifiers are unique within a command. Their source order is semantic and becomes their ordinal. All required arguments must precede every optional argument, and a repeatable argument must be last.

Platforms

The closed platform values are any, linux, macos, windows, and freebsd. any must appear alone. A root command with no platform nodes inherits any. A subcommand with no platform nodes inherits its parent. Specific child platforms are intersected with the parent's effective set and must leave at least one platform. Declaring any on a child means the inherited parent set; it does not widen support. Flags follow the same rule and inherit their command when they have no platform children.

Runtime selection filters commands and their flag, argument, and semantic projections by the requested host platform. Asking for platform any disables host filtering for platform-independent inspection; it does not assert that an upstream executable works everywhere.

Native completion actions

The action property accepts exactly:

These are declarations, not build-time or query-time callbacks. The catalog compiler and reader never execute an action. A runtime consumer must implement the selected action under its own cancellation, byte, result, filesystem, and latency policy. The action confers no ambient capability.

Strictness and diagnostics

The parser rejects input rather than ignoring it. Rejection includes:

• more than one top-level node or a root other than catalog;
• missing or duplicate provenance, or a catalog without commands;
• unknown catalog, command, or flag child nodes;
• unknown or duplicate properties;
• typed KDL nodes or values, incorrect property types, missing required strings, unexpected positional values, or child blocks on scalar nodes;
• empty, whitespace-only, control-containing, oversized, or malformed names and strings;
• duplicate aliases, intents, platforms, overlapping flag spellings, arguments, sibling tokens, or normalized paths;
• any combined with another platform, a child platform outside its parent, a completion action on a boolean flag, a required argument after an optional one, or a non-final repeatable argument; and
• every configured byte, count, depth, database, scan, or query limit crossed.

Diagnostics are inert quirl-catalog values. Each has a syntax, validation, resource_limit, io, or database kind; a caller-supplied source identity; a message; optional UTF-8 byte offset and length; actionable help; and bounded context. Resource diagnostics include the configured limit and observed usage when safe. The CLI or another effect owner maps these values to ShellError; the foundation parser does not perform rendering or I/O merely to report source errors.

Explicit resource bounds

Callers may lower defaults but cannot set zero or exceed the hard maximum. The same limit object applies to parse, typed validation, compilation, publication, opening, and query operations.

The checked-in embedded artifact uses the public NativeCatalogLimits::embedded profile: 2 MiB of database bytes, 2,048 commands, 8,192 flags, 8,192 arguments, and 24,576 semantic documents, with the other defaults below unchanged. Both xtask and the CLI use that one profile, so CI cannot accept an artifact that the executable will reject for exceeding a duplicated lower limit.

SQLite additionally limits one value to at most 4 MiB, SQL text to 64 KiB, 64 columns, expression depth 32, 16 compound selects, 100,000 virtual-machine operations, 32 function arguments, 4 KiB LIKE patterns, and 64 bound variables. Triggers, attached databases, and SQLite worker threads are disabled.

Canonical formatting and deterministic compilation

The formatter operation is a source review aid. Its contract is to produce stable KDL layout and property ordering without changing values, platform meaning, argument order, or provenance. Running it twice must be byte-idempotent. Import must never be the only way to format or validate curated source.

The parser canonicalizes meaning before compilation:

• root commands and subcommands sort by canonical name;
• aliases and intent phrases sort lexically after duplicate checking;
• flags sort by canonical spelling and then effective platform scope;
• platforms sort by their closed enum order after inheritance/intersection;
• positional arguments retain authored order; and
• flattened commands sort by full canonical path and receive stable one-based identifiers, while flag, argument, and document identifiers follow that stable traversal.

Compilation uses an in-memory database with fixed page size and schema pragmas. One transaction writes the exact typed JSON snapshot and all normalized rows; no timestamp, random value, absolute build path, locale-dependent comparison, filesystem iteration order, or network result enters the image. Compiling the same typed catalog twice must yield identical bytes. application_id is QCNC (0x51434e43) and user_version is 2.

A reader validates the application identity, schema version, quick_check(1), deny-unknown typed snapshot, all semantic limits, and exact deterministic recompilation. The last check binds normalized rows and the snapshot: editing a row, adding an object, changing ordering, or using a different compiler image causes admission to fail.

Normalized and semantic projections

The database contains normalized tables for the catalog snapshot, provenance, commands, aliases, effective platforms, intent phrases, flags, flag platforms, arguments, semantic documents, and semantic-document platforms. Commands store canonical full paths and parent identifiers; flags and arguments retain summaries, descriptions, cardinality, value names, and closed actions.

Every command creates one semantic document whose body concatenates its full path, aliases, summary, description, and intents. Every flag document contains the command path, long and optional short name, summary, description, and value placeholder. Every argument document contains the command path, name, summary, and description. This projection is deterministic and carries no embeddings.

Native semantic lookup is bounded deterministic lexical matching: it lowercases whitespace-separated query terms, scores one point for each term contained in a lowercased document body, discards zero-score documents, then orders by descending score and lexical command path, target, and title. The host platform filters documents before scoring. ADR 0021's local intelligence cache may separately embed composed documents; embeddings are not part of the native KDL artifact or its source of truth.

Completion lookup provides prefix-filtered, stable-order projections for root or nested command names and aliases, flags, and ordered argument placeholders. It returns summaries, descriptions, and actions, never an instruction to execute a candidate.

Publication, artifacts, and cleanup

Publication compiles and validates in memory before creating a staging file. The destination parent must already be a directory and, on Unix, must not be group- or other-writable. Existing destinations must be admitted regular, unlinked, bounded, permission-safe databases; on Unix, group/other writable modes and link counts other than one are rejected. Path/handle identity, size, modification time, link count, type, and permissions are checked across reading.

The publisher creates a unique sibling with exclusive creation and Unix mode 0600, writes and content-syncs the complete image, rereads it, and compares exact bytes. It then verifies that an old destination is unchanged or that a previously absent destination has not appeared. Atomic rename is the publication point. An RAII guard removes staging files on every earlier failure or unwind. The parent directory is best-effort synced after rename because rolling back a visible valid image after a second failure could destroy the only good copy.

CI must format-check and strictly validate KDL, compile twice, compare bytes, open the result through the hardened reader, and publish the immutable SQLite artifact with a cryptographic checksum bound to the exact source revision. Release records name the QCNC schema version, artifact byte length, checksum, and source revision. Drafts are never release inputs.

Curated and Carapace-draft workflow

Carapace is only an untrusted, pinned, build-time draft source. It is never a runtime dependency or intermediary and cannot directly update curated KDL. A pinned revision update is reviewed as a supply-chain change:

1. Change the tooling-owned pin to one immutable upstream revision and verify the fetched/provided source identity.
2. Run the bounded import operation for explicitly selected commands into a separate draft area. The checked-in selection currently contains 90 root command files and 96 command paths, including common filesystem and shell commands, development tools, editors, network utilities, archive tools, Windows-specific commands, and the nested task completion <shell> tree. Explicit coverage floors in catalog/carapace-import.json prevent silent corpus shrinkage.
3. Run the canonical format operation on the draft, then inspect the complete KDL diff. Do not accept invented or silently dropped facts.
4. Review command shape, subcommands, aliases, flags, argument ordering, summaries, descriptions, intents, platforms, actions, and upstream version meaning against authoritative documentation or the named executable. For operating-system utilities, review every supported platform projection independently: a shared command name does not make GNU, BSD, macOS, and Windows flag spellings or meanings interchangeable.
5. Review author, license, revision, and source; preserve any additional notices required by the upstream license.
6. Copy only approved semantic changes into curated KDL, then run format check, strict schema check, deterministic build, hardened-reader validation, and the catalog tests.
7. Review the resulting checksum change. Unchanged canonical KDL must not produce a different database.

The initial reviewed runtime base contains 50 curated root commands. The wider 90-file Carapace corpus remains checked in beside it as update and promotion proposals; draft count is deliberately reported separately from runtime count. Catalog regression tests enumerate the deliberately flagless root commands and require every other root to retain at least one named option on every declared platform. They also assert that representative GNU-only flags cannot enter the macOS projection.

Run the implemented task-runner interface from the repository root:

cargo xtask catalog import-carapace --source <checkout> --revision <40-character-pin>
cargo xtask catalog promote --command <reviewed-command>
cargo xtask catalog fmt
cargo xtask catalog fmt --check
cargo xtask catalog check
cargo xtask catalog build

import-carapace requires the revision to match both catalog/carapace-import.json and the detached local checkout. It also asks local Git to prove that every manifest-listed file exists at that commit and has no worktree changes; Git is bounded to ten seconds and upstream code is never executed. The upstream and retained license files must both match the reviewed SHA-256 pin. Unsupported completion constructs are omitted rather than invented and are listed explicitly in carapace.import.json with their source file and Cobra variable for review. Each selected root is written to catalog/draft/<command>.kdl; provenance in the KDL and the exact path list in the import manifest identify ownership, so a redundant source prefix is unnecessary. Obsolete importer-owned aggregate or per-command drafts are removed only after every replacement has rendered and validated. The operation updates only catalog/draft; a draft may intentionally share a root name with curated KDL because that is how upstream updates remain reviewable. catalog promote is an explicit first-promotion helper: it accepts named commands only, requires matching reviewed provenance, and refuses to overwrite an existing curated file. Subsequent changes are applied by reviewing the semantic diff and editing curated KDL. fmt --check is non-mutating. check validates formatting, schema, provenance, license retention, curated/draft separation, generated database bytes, the checksum, and hardened-reader admission using the same embedded limits as runtime. build refreshes the generated SQLite image and checksum from curated KDL only.

The importer does not read whichever Zsh completion functions happen to be installed on the build host. Those files vary by operating system and package set and therefore cannot be a reproducible catalog input. A future Zsh-source import must pin an immutable source revision and license, retain its own per-command provenance, and pass through the same draft review boundary. Catalog development commands admit at most 256 total curated and draft KDL files, so expanding the source set remains an explicit bounded change.

Runtime lookup and fallback

The checked-in SQLite image is compiled into the Quirl executable. At first use, the composition root opens and validates a bounded in-memory copy through NativeCatalogReader, selects the compile target's effective platform, and normalizes admitted facts into the schema-v4 command graph. No runtime path reads KDL, invokes Carapace, downloads a replacement, or opens a native-catalog file.

Curated external facts are declared, high-confidence metadata. Exact Quirl builtin contracts and trusted plugin declarations therefore retain precedence when names collide; mutable local intelligence already admitted at equal or greater confidence also remains authoritative. The same composed records feed completion, help and docs, agent discovery, LSP, and the capability-limited MCP catalog. Closed completion actions remain inert provider identities; the catalog never contains or executes provider code.

The embedded artifact is a required build and release input. Corruption, incompatibility, projection mismatch, or a compiled resource-limit violation is an actionable loader diagnostic and a build/test failure. At runtime it remains unavailable knowledge rather than startup authority: composition omits the native projection and continues with builtins plus any independently admitted local intelligence. It never invokes Carapace or parses KDL as a fallback.

Composed semantic catalog schema v4

After composition, help, completion, LSP, agent context, plugin contributions, and local intelligence continue to consume schema-v4 CommandSpec records. The graph is deny-unknown and versioned; no completion, documentation, LSP, agent, or plugin projection maintains another command list.

Command records, quality, and migration

Each command has stable id, optional declaring version, display path, aliases, optional stable parent id, signature, summary, details, examples, typed streaming IO, effects, integer exit-code descriptions, and fact-level provenance. The JSON field is named arguments. Each argument records names, its positional, option, or flag kind, value type, required/repeatable state, optional static or dynamic completion source, conflicts, documentation, examples, and provenance.

Provenance contains source, confidence, trust, optional origin, optional fingerprint, and optional generated_at. Builtins are exact/builtin and validated plugin declarations are exact/trusted. Fish, Bash, and Zsh facts remain attributed declarations; help/man extraction remains heuristic. Wall-clock timestamps are omitted unless a producer supplies a deterministic source timestamp.

Catalog::quality_issues() rejects incomplete exact records: stable identity, declaring version, command and argument documentation, types, examples, IO, and exit-code descriptions are mandatory. It also validates parent ids, alias and argument-name uniqueness, resolvable conflicts, and nonempty static/dynamic completion sources. Imported external observations may retain Unknown IO, no version, no examples, and no exit-code map; Quirl does not promote them to exact facts.

Catalog::from_json() accepts v4 and migrates cache schemas 2 and 3. Migration preserves paths, prose, effects, confidence, origin, and fingerprints; converts legacy options into arguments; derives stable ids and parents; and marks new facts unknown or empty. The CLI merges migrated records beneath current compiled builtins, so an old cache cannot remove or overwrite an exact builtin. Unknown versions fail validation and require a cache rebuild.

Durable local intelligence cache

Interactive startup admits the CLI-owned cache automatically. One session-owned discovery worker starts immediately after catalog admission, does not wait for accepted input, and never runs on highlighting, editing, completion, or rendering paths. A full scan has a 30-second background deadline and the periodic interval is 60 seconds; accepted input may request one coalesced additional scan.

Discovery inspects executable PATH entries and reads admitted declarative Fish, Bash, Zsh, help, and man sources. It never invokes an executable, shell, man, or user startup file. The source-file ceiling is 4,096 within separate 8,192 directory-entry, 1 MiB retained-path, 16 MiB aggregate source-byte, and 16 MiB canonical-catalog bounds. At most 512 matching plain manual pages are retained; each remains within the 1 MiB documentation limit. Source admission, symlink handling, file type, permissions, hard links, deadlines, records, and diagnostics remain bounded.

The default cache is $XDG_CACHE_HOME/quirl/catalog.sqlite3 (or the equivalent under $HOME/.cache) and QUIRL_INDEX_PATH may select another path. It stores the exact composed catalog snapshot, normalized commands, aliases, arguments, names and values, conflicts, examples, effects, exits, provenance, semantic documents, Model2Vec embeddings, and versioned discovery state in transactions. Discovery state binds its sorted source inventory and fingerprints to the catalog fingerprint and schema version.

Cache application id QUIR and user_version 1 are distinct from native artifact identity QCNC. Cache reads are limited to 128 MiB, attached databases and SQLite worker threads are disabled, and discovery remains capped at 65,536 records. A missing, stale, corrupt, incompatible, or fingerprint-mismatched cache is a miss. A complete in-memory image is atomically published; a failed generation preserves the last valid database. If none exists, a builtin-only SQLite image supplies an indexable fallback. Concurrent shells coordinate writers through a dedicated persistent sibling lock; readers observe only complete generations.

Local semantic intelligence

After initial catalog admission, a session-owned worker validates the current release's downloaded command-model asset and builds embeddings automatically. The provider-neutral release manifest fixes the model bundle's version, size, SHA-256 digest, format, compatibility, and immutable URL. Quirl streams that single bounded tar payload, verifies it before admission, expands only its closed three-file model layout, and atomically publishes a private, content-addressed model directory. It never downloads model files from a mutable branch. An explicit QUIRL_MODEL_PATH is never replaced automatically. Redirects, waits, retained chunks, staging attempts, directory depth, file bytes, documents, tokens, batches, vector dimensions, serialized bytes, and finite floats are bounded; cancellation is checked between streamed chunks and 32-document batches.

Embeddings carry model id and source-document fingerprint and publish only when their request generation and exact source database remain current. A changed database schedules the newest generation rather than allowing stale embeddings to overwrite it. Missing or invalid model assets and missing/stale embeddings select bounded deterministic lexical ranking.

The explicit AI index operation remains a diagnostic/refresh path, not a setup requirement. AI search, related suggestions, and interactive AI mode read the same cache. Interactive selection inserts text into normal mode for review; subcommands display candidates. Neither executes a suggestion automatically.

Plugin commands enter composition only after manifest validation and retain their package version, typed IO, arguments, effects, numeric exit codes, source fingerprint, and trusted provenance. Builtin signatures remain the declaration for mechanically projected positional shapes, whose provenance is high/declared, not exact.

These composed-cache rules remain governed by ADR 0007 and ADR 0021. The native KDL compiler supplies attributable external facts without creating a parallel Quirl builtin contract or a second mutable cache.