commit - 9c156e43a6c19018ba9fa33b3031ab6bf197b0ba
commit + 4a93e0de0e87b979672e0c5ca321485d1ee4a16f
blob - f4993a3a63c5be2e2470480bcf0bfabba6d5dfb9
blob + bdbbd99776166f2f853207fbc6392a1648aeeede
--- docs/compaction.md
+++ docs/compaction.md
# Compaction
Compaction replaces the Lead's active Projection with model-written
-continuation notes. It is currently initiated explicitly with `/compact`.
-The completed notes are journaled as the Compaction record, injected into the
-rebuilt system block, and recovered on resume. The transcript and the command's
-visible behavior do not change.
+continuation notes. `/compact` initiates it explicitly. With `auto_compact yes`
+(the default), the canonical Lead Turn can also Compact before a prospective
+Generation crosses its configured reserve, or once after an exactly classified
+context-overflow rejection. A completed cut, its immutable System epoch, its
+structured file details, and any retained Projection suffix are one Journal
+record and are recovered together on resume.
+## Threshold and estimate
+
+Automatic threshold Compaction runs only when the Effective window is known.
+The existing window priority remains authoritative: `context_limit`, then
+Provider-reported model metadata, then the compiled model table. Before every
+Lead Generation, including continuations after Tool results, fugu conservatively
+estimates the actual prospective request from the selected System-baseline
+bytes, canonical Projection blocks, offered Tool definitions, and fixed
+Provider-wire structural allowances. It converts the resulting bytes with
+`ceil(bytes / 4)`; usage reported for an earlier request is never substituted
+for this prospective estimate.
+
+The strict trigger is:
+
+```text
+estimated tokens > Effective window - compact_reserve_tokens
+```
+
+The default reserve is 16,384 tokens. An unknown window skips this threshold
+check. `auto_compact no` disables both threshold cuts and overflow recovery but
+does not disable `/compact`.
+
+## Cut selection
+
+Automatic Compaction normally keeps complete recent Turns up to
+`compact_keep_tokens` (20,000 by default) and summarizes the older prefix. The
+retained Projection never starts with a Tool-result message and never separates
+an Assistant Tool call from its exact result set. Prompt-cache receipts become
+cold after every successful cut.
+
+When the current Turn alone exceeds the keep target, fugu may split its early
+prefix at a valid Assistant boundary. A Tool-using Assistant is a valid retained
+boundary only when the immediately following User message contains exactly one
+result for each call. The early portion is separately delimited as
+`<turn-context>` and merged into the replacement notes; the valid recent suffix
+stays canonical. If no such boundary exists, the complete Turn is retained.
+
+An automatic cut during an active Turn is provisional. Cancellation, failure,
+or a crash discards both the cut and that Turn's Pending Projection. The epoch
+and retained snapshot become replayable only with the matching successful Turn
+terminal, so already executed Tools are neither lost nor replayed.
+
## Isolated request
The Compaction Generation does not replay the active Projection in its live
</conversation>
```
+For an oversized active Turn, the summarized early prefix follows as a second
+quoted section:
+
+```text
+<turn-context>
+[User]: current Turn input
+
+[Assistant]: early work within this Turn
+</turn-context>
+```
+
Records are separated by one blank line and retain canonical message and block
order. Adjacent User text blocks concatenate. Adjacent Assistant text blocks
join with a newline. Adjacent Tool calls join with `; `. Empty text and Tool
and replaces malformed UTF-8 with U+FFFD.
The complete User document, including any previous-summary section and both
-wrappers, is capped at `FUGU_REQUEST_MAX`. Provider serialization is checked
-against the same cap afterward because JSON escaping can expand controls and
-malformed bytes. Exceeding either bound fails Compaction conspicuously without
-changing the Projection, Compaction notes, Journal, or Projection cursor.
+wrappers (and an optional turn-context wrapper), is capped at
+`FUGU_REQUEST_MAX`. Provider serialization is checked against the same cap
+afterward because JSON escaping can expand controls and malformed bytes.
+Exceeding either bound fails Compaction conspicuously without changing the
+Projection, System epoch, structured file state, Journal, or Projection cursor.
+## Structured file details
+
+After accepting model prose, fugu derives cumulative file details from
+structured `read`, `write`, `edit`, and `multi_edit` Tool-call arguments in the
+summarized prefix. It never guesses paths from shell text. Accepted calls are
+tracked even when their result reports failure: these lists describe
+model-visible structured work, not an audited filesystem log.
+
+The new details are merged with the prior epoch's structured sets. Modified
+paths win over read-only paths; both sets are bytewise sorted and deduplicated.
+Fugu, not the model, then appends exactly one pair of blocks after the prose:
+
+```text
+<read-files>
+src/read-only.c
+</read-files>
+
+<modified-files>
+src/changed.c
+</modified-files>
+```
+
+Paths must be bounded valid UTF-8 without controls or line separators, and
+`&`, `<`, and `>` are escaped in the rendered blocks. The structured arrays are
+stored separately in the System-epoch Journal record, so repeated Compaction
+and resume merge trusted details rather than parsing model prose. `/clear`
+empties them.
+
+## One-shot overflow recovery
+
+When automatic Compaction is enabled, a rejected Lead Generation may be
+recovered exactly once per Turn only if no Provider content was accepted and
+the structured error is unambiguous. Today that means an OpenAI-compatible HTTP
+400 response or stream error whose exact `error.code` is
+`context_length_exceeded`. English text, HTTP status alone, Anthropic's broad
+`invalid_request_error`, Claude Subscription errors, and HTTP 413 byte limits
+never qualify.
+
+Recovery Compacts the current Projection and retries only the rejected
+Generation. It does not rerun a Tool batch that has already produced results.
+A second overflow, any post-content failure, cancellation, failed Compaction,
+or overflow from the summarization request terminates normally through the
+existing error path without another retry. Usage from the summarization and
+retried request remains additive.
+
## Completion and failure
A Compaction result is accepted only when the Provider completes one normal,
non-empty Assistant text response without Tool calls. Cancellation, a Provider
-error, a length stop, malformed output, an attempted Tool call, or either size
-bound leaves the existing state intact. On success, the Turn transaction
-journals the notes and resets the Projection through its existing Compaction
-operation; `/clear` later voids those notes as before.
+error, a length stop, malformed output, an attempted Tool call, or any size
+bound leaves the existing epoch, Projection, structured files, and cursor
+intact. On success, one complete replacement System epoch is appended and
+synchronized before the live prefix is removed. `/clear` later installs a new
+no-summary epoch and voids the notes and file details.
blob - 558868ffb6292c86f70e4ad0fa0c6184be41d09c
blob + e560d830dc67656f1004feabeb8ea26b488f9cdb
--- handoff/behavior.md
+++ handoff/behavior.md
- **/clear** — reset the active context: the next message starts a
fresh conversation. The on-screen transcript stays as scrollback;
- the journal is preserved and records the reset so a resume also
- starts fresh (§7).
+ the journal is preserved and synchronizes one replacement System epoch so a
+ resume also starts fresh (§7). It adopts the invocation's current base,
+ reuses the Session-cached personal bytes, re-reads project context under
+ current policy, and drops Compaction notes and file details.
- **/compact** — replace the active context with a model-written
summary of the conversation (one extra API call, no tools offered to
it). Serialize any prior Compaction notes and the active Projection as one
preserve goals, constraints, decisions and rationale, work state, key
paths and identifiers, and open questions, addressed to itself as
continuation notes. Prior notes remain quoted User data, never privileged
- instructions. The journal records the summary; resume carries the compacted
- state into later Compactions (§7). The UI shows a distinct busy state.
+ instructions. Fugu appends deterministic cumulative `<read-files>` and
+ `<modified-files>` blocks derived only from supported structured Tool calls;
+ modified paths win and the Journal stores the sets separately from model
+ prose. The complete replacement System epoch and Projection reset are
+ synchronized together; resume carries them into later Compactions (§7). The
+ UI shows a distinct busy state. Automatic threshold cuts and one-shot
+ overflow recovery use this same operation: a conservative prospective
+ request estimate is compared strictly with Effective window minus
+ `compact_reserve_tokens`, and whole recent Turns are retained up to
+ `compact_keep_tokens`. An oversized current Turn may split only at a
+ Provider-valid boundary with every Tool call/result pair intact; its early
+ prefix is separately delimited as `<turn-context>`.
- **/model** — the picker (§2.1).
- **/context** — print token *estimates* for the current projection —
- system prompt, project context, conversation messages, tool
- results, compaction summary if present, and the largest single
+ exact selected System baseline, conversation messages, tool
+ results, and the largest single
items (reference: five; count free) — labeled as estimates
(byte-derived heuristic; exactness not promised) against the
effective window. The effective window
### Project context files
-At startup fugu reads the first of `FUGU.md` then `AGENTS.md` in the
+For a new Session, fugu reads the first of `FUGU.md` then `AGENTS.md` in the
working directory, plus the personal `~/.fugu/FUGU.md`, each capped
at 64 KiB — an oversize, NUL-bearing, or `protect`-listed file is
skipped with a one-line notice, like skills. The contents are
the tool-executing role over the existing IPC (so no role widens,
I9); the personal file's bytes are cached at startup and reused —
it is never re-read mid-session. The injected bytes are recorded in
-the journal as a context event so a resumed session replays them
-verbatim (invariants.md I14) even if the files have since changed,
+the journal inside a complete System epoch so a resumed session replays the
+exact stored baseline variant and snapshots (invariants.md I14) even if the
+files, configured `system`, or composition code have since changed,
and the injection position must not create adjacent same-role
messages. The files are read by roles that already hold the needed
filesystem access — never the credential custodian (I1); the `#`
-capture line (§3) is the write-side companion. `project_context no`
-(§6) disables the mechanism entirely. The manual's CAVEATS must state
+capture line (§3) is the write-side companion. On resume, current
+`project_context yes` selects the stored full baseline; `no` selects the stored
+base-plus-Compaction variant and suppresses stored external bytes from
+Subagents and capture without deleting them. A later resume may select the
+other stored variant. External edits and capture do not mutate the live epoch;
+only `/clear` or a new Session admits them. The manual's CAVEATS must state
that starting fugu in an untrusted tree injects that tree's
instructions into the model's context (invariants.md §4).
| `provider` *string* | default model route: experimental pure-C subscription compatibility `claude`, direct `anthropic` [default], or direct `openai` (the `/v1/chat/completions` dialect that third-party and local servers emulate, Bearer auth) |
| `system` *string* | replaces the built-in system prompt entirely (single quoted line) |
| `max_tokens` *number* | per-turn generation ceiling [4096] |
-| `context_limit` *number* | advisory token budget for the projection; when set, also the effective window for /context and the status gauge (§3) |
+| `context_limit` *number* | first-priority Effective-window override for /context, the status gauge, and automatic Compaction (§3) [unset] |
+| `auto_compact` yes\|no | enable prospective threshold Compaction and one-shot typed overflow recovery [yes] |
+| `compact_reserve_tokens` *number* | tokens reserved below the Effective window before automatic Compaction [16384] |
+| `compact_keep_tokens` *number* | target size of the valid recent Projection suffix retained by Compaction [20000] |
| `allow_write` yes\|no | offer mutating file tools [yes] |
| `allow_subprocess_net` yes\|no | widen subprocess exec ceiling with network [no] |
| `web_search` yes\|no | offer web_search + web_fetch [yes] |
- Append-only; one record per event; a crashed session's journal is
still replayable up to the last complete record.
-- The journal captures everything needed to reproduce conversation
- state: user and assistant messages (including tool exchanges), and
- the context events — clear, compact (with its summary), skill
- invocation, project-context injection (§4, the injected bytes
- verbatim), and turn failure/abandonment.
+- The journal captures everything needed to reproduce conversation state: user
+ and assistant messages (including tool exchanges), plus one versioned,
+ binary-safe System-epoch record at fresh start, clear, successful Compaction,
+ and legacy migration. An epoch owns exact base/personal/project/summary
+ components, authoritative full and restricted Lead baselines, structured file
+ sets, and an explicit retained Projection snapshot. New reset writers never
+ split that state across independently replayable records.
- Undispatched Queue entries (§2.1) are process-local interface state
and are never Journaled. `turn_txn_accept` provisionally appends the
records for messages accepted into the Pending Projection, including
Steer replays at its exact delivery position. An ordinary Queue may
remain after an error only while the same front end survives. Quit,
crash, and session-switch re-exec do not preserve it.
+- An automatic cut inside an active Turn carries that Turn's number and remains
+ provisional. Replay applies its epoch and retained snapshot only with the
+ matching synchronized `ok` or `cap` terminal, then appends later messages
+ from the same Turn. Abandonment or an incomplete tail retains the previous
+ epoch and Projection byte-for-byte.
- Print-mode turns (`-p`) append under the same rules; `fugu -r id
-p "..."` continues a saved session headlessly.
- A turn ended by the Tool-batch cap (§5) is a *completed* turn: its
abandoned turns contributing nothing. In particular the projection
sent to the provider after a resume must be one the provider
accepts (no orphaned tool calls, no adjacent same-role messages).
+- Resume selects an already stored exact baseline with current
+ `project_context` policy; Provider/model routes, credentials, Tools, skills,
+ endpoints, protection, write, and network authority remain live. Legacy
+ journals are never rewritten: the first resume reconstructs under old rules
+ and appends one synchronized migration epoch before any Provider request.
- `/clear` and `/compact` never truncate the file; they append.
- Full journals remain the Owner's plain files; the manual documents
that running fugu from `$HOME` lets the tools read them (CAVEATS).
(§7). The policy is pinned; no
configuration keyword.
+Separately, with `auto_compact yes`, the canonical Lead Turn may recover one
+rejected Generation before any Provider content was accepted. The only current
+typed discriminator is an OpenAI-compatible HTTP 400 or stream error whose
+structured `error.code` is exactly `context_length_exceeded`. Recovery
+Compacts the current Projection and retries that Generation once; it never
+reruns an already completed Tool batch, recurses through the summarizer, or
+retries a second overflow. Status alone, English prose, HTTP 413,
+Anthropic `invalid_request_error`, and Claude Subscription errors are not
+classified. Unknown or post-content errors retain the terminal behavior above.
+
The built-in system prompt (sent when `system` is unset) must be
concise and cover: fugu's identity as a terminal coding agent on a
Unix/OpenBSD system; terse, no-preamble answer style with markdown and
blob - d99806e29c16f1832f8c21dee5c341e3d117c21b
blob + e8dc8af05dd5e68f18edd1cb227218ba2d2000be
--- handoff/invariants.md
+++ handoff/invariants.md
that fails mid-flight (provider error, dropped stream, cancellation)
leaves no partial records that a later resume would replay. Resuming
a session yields precisely the conversation state the Owner last saw
- after the most recent successful turn. (This is also a behavioral
- requirement; see behavior.md §7.)
+ after the most recent successful turn. Every reset boundary is one
+ versioned, binary-safe System epoch containing exact authoritative baseline
+ variants, source snapshots, Compaction/file state, and any retained
+ Projection. A cut inside an active Turn is replayable only with that Turn's
+ matching successful terminal; a crash or abandonment preserves the prior
+ epoch and Projection. Resume selects stored privileged bytes rather than
+ rebuilding them from current configuration, files, or code. Legacy migration
+ is additive and synchronized before the first resumed Provider request.
+ (This is also a behavioral requirement; see behavior.md §7.)
### Subagents (new in this version)
blob - 5f9df1209a5f60dd4c83b60661ca45f545ced8d7
blob + 11d176d1a06ecbfebfc9c7962c7f99e5ff6281bc
--- handoff/verification.md
+++ handoff/verification.md
the file changes on disk mid-session; oversize and NUL-bearing
files skipped; `#` capture appends atomically and is refused under
`allow_write no`.
+- **System epochs (I14):** binary-safe round trips cover every component,
+ authoritative full/restricted variant, reason, structured file array, and
+ retained Projection block. Fresh start, clear, successful Compaction, and
+ legacy migration synchronize before request/publication. Unsupported,
+ missing, duplicate, inconsistent-derived, over-bound, and malformed fields
+ hard-fail; a partial final epoch is ignored and retried. Cross-process
+ fixtures change base configuration, context files, composition code,
+ Provider/model, Tools, and authority policy, then prove current
+ `project_context` selects exact stored bytes for all Provider codecs and
+ Subagents without deleting the other variant. Every append/sync/live-publish
+ crash boundary and active-Turn ok/cap/abandon path is exercised.
- **Queue and Steer admission:** `regress/turn` uses staged curses PTY
fixtures to distinguish transcript/Queue admission, Queue removal or
Steer selection, Pending Projection and provisional Journal
- **Background jobs:** `max_bg_jobs` honored; jobs die on turn
failure, cancellation, session end, and drawer switch; unknown and
prior-session ids error cleanly; ring-buffer overflow marked.
-- **Retry policy:** no retry after the first content delta;
+- **Retry policy:** transport retries never occur after the first content delta;
`Retry-After` clamped to [1, 60] and parsed as untrusted (I13);
- cancellation during backoff leaves the journal clean.
+ cancellation during backoff leaves the journal clean. Separately, exact
+ OpenAI `context_length_exceeded` body/stream fixtures and near misses prove
+ one no-content Compaction recovery, a single retry, no recursion, and generic
+ handling for Anthropic/Claude, 413, status-only, prose-only, post-content,
+ and second-overflow errors. A recovery after a completed Tool-result batch
+ proves the Tool runs once while usage remains additive.
- **Paste and compose:** paste cap enforced, NUL rejected, ESC
stripped; bracketed-paste mode restored on every teardown path;
editor compose leaves input untouched on nonzero exit.
across repeated and resumed Compactions, and exercise post-JSON expansion
past the request cap without changing the Projection. Rejected Provider
Generations separately prove the Projection and cache cursor stay paired.
+ Prefix planning covers strict threshold boundaries, unknown windows,
+ prospective System/Projection/Tool estimates, whole recent Turns, giant-Turn
+ safe splits, complete multi-Tool result sets, and no-safe-cut fallback.
+ Structured file tests cover cumulative resume/recompact, read-to-modified
+ promotion, sorting/deduplication, supported-call-only collection, forged
+ paths, empty blocks, failed Tool results, and failure/cancellation rollback.
- **Context meter:** the effective-window resolution order
exercised (`context_limit`, provider-reported, compiled table,
unknown); `/context` leaves the journal byte-identical; the status count
blob - e49cacbcde9a144630ae9fdc3c81335d4efbdb3d
blob + 80eced6a1f837288c640a3b7eada48b2e5c68fba
--- src/fugu/fugu.1
+++ src/fugu/fugu.1
.It Cm /clear
Reset the active model context while retaining the displayed transcript and
append-only journal.
-The in-tree project context file is read again.
+The in-tree project context file is read again and a new immutable System
+epoch adopts the current configured or built-in base prompt.
.It Cm /compact
Ask the model to replace the active conversation with continuation notes
that preserve goals, constraints, decisions, work state, important paths,
and open questions.
-The summary is journaled and becomes the resumable context.
+The summary and deterministic read/modified file lists are journaled in a new
+System epoch and become resumable context.
+Automatic Compaction uses the same operation; see
+.Xr fugu.conf 5 .
.It Cm /model
Open the curses model picker.
.It Cm /context
Prompts are limited to 4 MiB and final reports to 1 MiB.
Labels are at most 63 bytes and contain only ASCII letters, digits,
underscores, hyphens, and dots.
-The subagent receives the configured system prompt and current personal and
-project context, but none of the lead conversation or compaction summary.
+The subagent receives the active System epoch's pinned base prompt and, when
+current policy permits it, its stored personal and project context, but none
+of the lead conversation or compaction summary.
Only its final report is returned to the lead.
.
.Pp
Other HTTP 4xx responses and failures after streaming begins end the turn.
Provider and TLS error details are shown to the Owner without exposing
credentials.
+When automatic Compaction is enabled, one OpenAI-compatible error whose exact
+structured code is
+.Li context_length_exceeded
+may Compact and retry the rejected Generation once, but only before any
+Provider content was accepted.
+HTTP status or English prose alone, direct Anthropic errors, Claude
+Subscription errors, and HTTP 413 byte limits do not trigger recovery.
.
.Sh SESSIONS
Every session has an append-only, mode-0600 journal in
.Pa ~/.fugu/sessions .
Successful turns, tool exchanges, context changes, skills, delivered steering
input, and compaction summaries are recorded.
+A complete versioned System epoch is synchronized before the first Provider
+request and at every clear or successful Compaction boundary.
+It preserves exact full and no-project-context Lead baselines, source
+snapshots, structured file details, and any retained Projection suffix.
+A resumed invocation selects one stored baseline using current
+.Cm project_context
+policy; it does not rebuild the baseline from current files, configuration,
+or binary composition code.
+A legacy journal gains one additive migration epoch before its first resumed
+request and is never rewritten.
A failed, cancelled, or abandoned turn contributes no partial conversation
state; a turn stopped by the 50-iteration cap is complete and remains
resumable.
