system/devops-cycle-design.md
Status: approved model, landing incrementally. The two-workflow task
status model is now live —Taskstages are
designed → building → submitted(Build) and
submitted → reviewed → assembled → shipped → archived(Deploy — the
shipped → archivedarchive loop closes it, §1.4) — meeting at the
submittedseam — plusblocked(side) andarchived(terminal).
bin/task,bin/dor-check, and the board speak it.Landed since: the
Releasesingleton model; the persistent-release
branch CLI —bin/release init|merge|prepare|ship(§1.1); and
bin/agent-worktree's release-aware base default —newcuts the feature
branch fromorigin/release(falling back toorigin/mainwhere norelease
branch exists) andfinish --propens the PR with--base release.Still to land (each its own task): the Discord progress webhook (§5). Where
this doc describes it, it is the spec for the follow-up.Since landed (do NOT rebuild): multi-repo
ship—bin/release shipis
now a full producer-first, hub-before-satellites pipeline (gems publish → auto
re-pin consumers → hub app deploy → satellites), with the per-repotest_cmd
gate and partial-ship recovery (Release::ShipSequence,bin/release.rb); it is
not hub-only. The heartbeat plannerbin/devops-cyclewas also migrated off
the legacy stage names (it speakssubmitted/reviewed/assembled).Deploy-flow redesign (decided, 2026-06-22). The
submitted → shippedhalf
was re-homed by role — review is delegated by Avi to two seniors (no longer
his solo gate),assembledis owned by Steffon (now titled Platform
Engineer), andshippedis owned by Avi (full e2e on the frozen ship
SHA) plus explicit ship authority: the default operator gate or the
Merge, Assemble, Deployautonomous kickoff. §1.2 is the rewritten spec.
It lands via three build tasks:deploy-flow-heartbeat-tooling(planner/tooling + the
prepareretry/wait-for-boot fix),stages-page-step-outlines(the per-step
/stagesoutlines), andseed-souls-prod-qa(the reviewer souls, incl. a new
Alex Documentation reviewer persona distinct from the orchestrator seat).Operator companion: the board stage guide at
/stages, and the rendered SOP
infographic — the cycle drawn as accountability swimlanes, one row per owner —
at/stages/sop. Both render from
config/devops_vocabulary.yml(read viaDevops::Vocabulary), the single
source of truth for the SOP vocabulary: rename a term there and it flows to the
UI in one edit, so the page and these docs cannot drift. This document remains
the canonical full SOP.
This design answers seven goals:
| Capability | Where | Reuse as |
|---|---|---|
Task state machine — Build designed→building→submitted→reviewed, Deploy reviewed→assembled→shipped, plus blocked/archived |
Task model, devops-task-board.md |
The spine. Everything routes through the task. |
kind (feature/bug/chore/qa/release/cleanup), metadata["devops"] contract |
devops-task-board.md |
SOP routing key + handoff record. |
Activity log: comment / clarification / qa_feedback / handoff + scout reports |
Activity, task-board API |
The durable QA↔feature-agent channel. |
Sealed-bid sizing, backend_migration advisory-lock lane, release_slug lane |
sizing-rubric.md, exclusive-lanes.md |
Order-of-operations machinery. |
Test lanes (pr_review_gate / local_proof / qa_acceptance / production_smoke / nightly_deep / quarantine) + config/devops_test_suites.yml + bin/devops-tests |
testing.md |
The when/where axis of the pyramid. |
bin/qa-intake, bin/devops-cycle (scout packets/decisions/readiness), bin/agent-worktree, bin/qa-server, bin/deploy |
parallel-agent-devops.md |
The conductor toolchain the heartbeat agent drives. |
Discord POST /api/v1/release_notes (dry-run, grouped-by-app, standardized) |
release notes service | The standardized visibility primitive. |
| "Future Heartbeats" lease-model spec | devops-task-board.md |
The literal blueprint for the airgapped agent. |
The job is formalize + close gaps, not greenfield.
The flow is two workflows, matching how the work actually splits and who
owns each. Building a change (the feature agent) and shipping a release
(DevOps) are different jobs at different cadences, so they are different
lifecycles that meet at one seam — submitted.
designed → building →
submitted. A task is specced (designed), an agent claims and builds it
(building), and opens a PR (submitted) — where the feature agent's part
ends. A wall, bounced PR, or unready dependency parks it at blocked.submitted → reviewed →
assembled → shipped. The souls bookend it (2026-07-03): Avi supervises
review and ships; Steffon owns the whole middle. Review runs a 3-level
hierarchy: Avi is the SUPERVISOR (a thin gate that never reviews the
code) — he confirms product-acceptance and picks the primary + light
pair (bin/reviewer-select), then spawns both reviewers in parallel as his
own sibling children. The PRIMARY does the deep review (acceptance, base
tests, standards, smell, scalability); the LIGHT gives a focused second
read. On two approvals with no blocker the supervisor drives the task
to reviewed — and STOPS (review-only). A blocker lands it at blocked (rework, with a
qa_feedback note). Steffon (Platform Engineer) then runs the
self-healing qa-release (bin/release prepare): it SWEEPS every
reviewed task (+ any assembled straggler) onto the single release
candidate (RC) — merging each PR into the persistent release branch and
stamping merged: "release", stages unmoved — gates origin/release on the
next tier (integration + an e2e smoke), deploys it to QA, and flips members
reviewed → assembled only on QA-green. At ship Avi runs the full e2e
on the frozen ship SHA and, with explicit ship authority, the conductor
fast-forwards release → main (stamping merged: "main", then shipped).
submitted is the seam — the feature agent hands the PR to DevOps there.QA and production are properties of the release, not the individual task —
so there is no per-task QA stage; ship authority is a single decision on the RC
after the frozen-SHA gate.
WORKFLOW 1 · Build (feature agent) WORKFLOW 2 · Deploy (DevOps · Release model)
designed → building → submitted ─────────► submitted → reviewed → assembled → shipped → archived
▲ │ (review) (approved) (merged RC, ("run the (archive
└ blocked ┘ e2e green, deployment" loop,
(rework / env / dep) QA-deployed) → prod) §1.4)
blocked is the single "not in the pipeline's court" state — an agent hit a
wall, QA bounced the PR, or a dependency isn't ready. It records blocked_from
(captured automatically) + block_kind (environment / rework / dependency) so a
heartbeat agent routes it without re-reading the thread. archived is terminal.
The RC is a Release singleton (only one assembles at a time). Member tasks
carry its release_slug; it carries them through QA→prod and flips them to
shipped when it ships. The airgapped agent runs both workflows and crosses the
ship gate only when the session carries explicit production authority (Merge,, direct ship approval, or an already-approved rollout prompt).
Assemble, Deploy
The task API is on production, so the airgapped box only needs an internet
connection — no separate pull/sync layer.
Open every cycle by assessing the board BY STAGE — bin/task list --stage, never the default flat
reviewed|assembled|shipped|submitted|buildingbin/task (it caps at the 20 newest tasks across all stages with no truncation
list
warning, so older actionable work silently falls off). See
parallel-agent-devops.md → Step 0.
release branchRelease (singleton model) coordinates one candidate from assembly through
ship. Consolidation (decided): the legacy release_train field has become
release_slug — one concept, one name.
The integration branch is PERSISTENT. Every repo keeps a single release
branch — the same name in every repo (Release::BRANCH = "release"). Feature
PRs target release, not main; main is always an ancestor of release.
bin/release init creates the branch (= main) on every gem + app repo, once
(idempotent). Membership records at the sweep: Steffon's bin/release (or the per-task
preparebin/release merge <task> primitive it loops) merges
an approved PR into release and attaches the task to the active candidate —
release_slug + merged: "release", stage still reviewed. The member flips
reviewed → assembled only on QA-green (Release::Conductor.qa_green!,
after prepare deploys origin/release to QA and it smokes green). ship
fast-forwards each repo's release → main (stamping members merged: "main")
and deploys prod. After a ship, release collapses to main and re-accumulates
the next candidate.
| Field | Meaning |
|---|---|
slug |
Canonical id, e.g. 2026-06-20-s3-uploads. |
state |
assembling → assembled → shipped (+ abandoned). assembled = the QA candidate is built (members merged into release) and its suite checks out. |
branch |
The persistent integration branch release (same name in every repo); feature PRs merge into it, QA deploys from it, and ship fast-forwards it into main. |
confirmed_at / confirmed_by |
The ship authorization at assembled → shipped — operator approval for the QA workflow, or the autonomous production kickoff. |
qa_url / production_url / deployed_sha / release_notes_sent_at |
Deploy + notes record. |
| stage timestamps | The fine-grained stage timeline under state — see below. |
has_many tasks |
via tasks.release_slug. |
The stage timeline (Release::STAGES). Under the coarse state machine the
release carries an ordered set of timestamp columns, each a time-and-boolean
(stamped = the stage started/landed; blank = not yet). They are the single input
the /deployments pizza-tracker reads, they stamp first-write-wins (a replay
never rewrites history), and the release's current stage is the latest
stamped one (monotonic — a late upstream write never winds the tracker back):
| Stage | Stamp | Tracker reads |
|---|---|---|
testing |
testing_started_at |
node 1 Testing yellow |
tested |
tested_at |
(not a tracker node) — the /deployments Tested column's end stamp |
assembling |
assembling_started_at |
node 1 green · node 2 Assembling yellow |
assembled |
assembled_at |
node 2 green |
qa_deploying |
qa_deploy_started_at |
node 3 Deploying QA yellow |
qa_deployed |
qa_deployed_at |
node 3 green "Live on QA" — node 4 stays dark |
confirming |
confirming_started_at |
node 4 Confirming yellow |
confirmed |
confirmed_at |
node 4 green |
prod_deploying |
prod_deploy_started_at |
node 5 Deploying yellow |
shipped |
shipped_at |
node 5 green |
Stamps flow from the release event trail: every record_event! write — the
conductor's bin/release prepare/ship checkpoints AND the agent-facing
POST /api/v1/releases/:slug|current/events/:step/(start|complete) API — maps
(step, status) to its stage stamp (Release::EVENT_STAGE_STAMPS). prepare
brackets its pre_qa_gate (the integration/e2e-smoke test run) with
review_tests started/completed, which stamp testing_started_at /
tested_at — the /deployments Tested column (testing_started_at → tested_at).
tested is a duration stamp only, NOT a sixth tracker node (node 1 still greens
on assembling). A node
lights yellow ONLY on its own start stamp, so a finished stage leaves the next
node dark until its owner posts their start. That gap is the explicit
Steffon → Avi handoff: Steffon's qa-release finishes at qa_deployed (three
greens, Confirming dark, "Live on QA"); stage 4 lights only when Avi posts
confirming/start as he picks the candidate up (production-deploy SOP).
reopen! (a late sweep onto an assembled RC) clears the assembled →
confirmed stamps so the re-assembly + re-QA re-stamp fresh; shipped is only
ever stamped by ship!, never by an API post. Full API contract + the
post-by-post table: docs/agents/modules/task-board-api.md ("Release stage
timeline").
Task gains three links:
| Field | Meaning |
|---|---|
release_slug |
The Release this task rides (null until the sweep attaches it; the stage flips assembled on QA-green, not at merge). |
merged |
WHERE the code physically is — the crash-recovery git-location, orthogonal to stage: nil = not merged anywhere · "release" = merged onto the release branch (QA in flight) · "main" = ff'd into main (prod deploy in flight). Matrix: reviewed+nil = not swept · reviewed+release = swept/QA-in-flight · assembled+release = QA-green awaiting Avi · assembled+main = ff'd/prod-in-flight · shipped+main = done. An interrupted Steffon skips re-merging release; an interrupted Avi skips re-ff'ing main. |
dependencies |
Array of task slugs this one needs shipped first. Now enforced by the conductor (Release::Ordering) — a member sorts after every task listed here — composed under the producer-first rule (e.g. an engine gem before the apps that consume it). |
dependencies (task→task) and the exclusive lanes (resource-level:
migration, release, vault single-writer) compose: dependencies say "B needs A's
output"; lanes say "only one of these at a time." As of the gems-first
release work, dependencies is no longer spec-only: Release#ordered_members
honors it (a stable topological sort that falls back to position), so the
conductor sequences members producer-first and respects explicit
task-to-task edges. See "Gem members & producer-first ordering" below.
Membership records at the sweep; the stage flips on QA-green. Feature PRs
already target release, so there is no branch-cut and no PR-base retarget.
Steffon's bin/release prepare (the self-healing qa-release) DETECTS the
work — every reviewed task plus any assembled straggler not riding the
current RC — ensures a candidate exists (Release.current_or_open!), and SWEEPS
each detected task: gh pr merge its PR into its repo's release (SKIPPED for
a task already merged: release/main — the crash-recovery signal an interrupted
run leaves behind), then Release::Conductor.sweep! records membership +
merged: "release" in one heroku run for the whole batch. Stages do NOT
move at the sweep. The per-task primitive is still bin/release merge <task> (batched; with ≥2 slugs it prints an overlap planner —
[<task> …]
colliding files + suggested order + likely rebases, warning-only). A merge
conflict surfaces at this PR-merge step (resolve on GitHub, or block the
task for rework — prepare sweeps past it and keeps the rest) — release never
force-pushes. Next comes Steffon's pre-QA gate — the registry qa_test_cmd
tier (integration + e2e-smoke) on origin/release BEFORE anything deploys; a
regression ejects the offender (bin/release eject <task> → blocked +
detached + merged cleared, pair with the merge-commit revert) and the REST of
the RC rides the re-run. Gate green → prepare deploys origin/release to
QA + records the QA URL, waits for boot, smokes /up, and on QA-green
flips the swept members reviewed → assembled (Release::Conductor.qa_green!;
merged stays "release") → the release is assembled (the QA candidate).
A QA failure flips NOTHING — members stay reviewed and the next self-healing
run picks them back up, skipping the already-done merges. A late PR sweeping in
after the flip reopens the RC (Release#reopen!) so it re-QAs before
shipping. At ship, Avi first runs the full e2e + highest-tier suite on the
frozen ship SHA (the exact prod code — closing the merge-forward "shipped ≠
tested" gap); on green the ending depends on the trigger. A QA-only run
(pr-review → qa-release) stops for the operator, while full-cycle
continues with the already-authorized ship. The ship action (surfaced as the current release on
/deployments, not a passive status): bin/release ship fast-forwards each
repo's main up to release (so release collapses into main), stamps that
repo's members merged: "main" as each ff lands, deploys prod, and flips
members to shipped. Ship authority always lands after Avi's test
confirmation, before the deploy.
Release progress is also recorded as explicit ReleaseEvent checkpoints
(review_tests, assemble_release, deploy_qa, qa_smoke, ship_gate,
ship_authorized, deploy_prod, prod_smoke, release_notes,
archive_tasks). The /deployments tracker reads those events before falling
back to legacy Release fields, so ship_gate:completed can visibly finish the
Confirming step before deploy_prod:started begins production work. Steps that
take measurable work should be bookended with started and completed; older
completed-only checkpoints render as instant duration rows so their recorded
timestamp is still visible in release analytics.
Release analytics are cached on the Release record. Release::DurationCache
stores duration_metrics (versioned JSON), duration_metrics_cached_at, and
duration_cache_version; the /deployments dashboard shows last-three-release
averages, /deployments/all lists release timestamp rows, and
/deployments/:slug shows the per-release read view. Task/release event writes
refresh the owning release best-effort, and production follow-up tasks should use
the idempotent post-deploy hook bin/rails releases:refresh_duration_metrics to
backfill the last three shipped releases after a deploy.
Gem members & producer-first ordering. A release is not apps-only — it can
carry gem tasks (studio-engine, solana-studio) as first-class members
alongside apps. The classification lives in config/release_repos.yml (read by
Release::Repos): every member is a :gem (producer) or an :app (consumer).
Gems and apps are handled differently at both ends of the Deploy workflow:
release branch like any other, but there is no app artifact
to deploy — so at prepare the conductor skips gem members in the QA-deploy
loop. They ride the release as a record, and are QA'd indirectly through a
consuming app (the consumer's branch is what gets deployed + tested).Release#ordered_members
returns members gems-first (then apps), honoring dependencies within
that. bin/release ship publishes every gem member to RubyGems first
(approval-gated gem push, version from the gem's version_file), and only
then deploys the apps. So a consuming app always builds + deploys against the
just-published gem version — never ahead of it. If a gem fails to publish,
the ship aborts before any app deploys.lib/studio/version.rb
for studio-engine, the .gemspec for solana-studio) is part of the gem task's
own PR; member_plan reads it for the publish + the board's 💎 gem badge.
Post-publish, consuming apps re-pin ~> x.y and deploy as their own members
(or follow-up tasks). See docs/agents/modules/deployment.md →
"Releasing a gem (producer-first)" for the operator runbook.This is the ordered release_slug from §4.2 ("gem publish → consumer lockfile
bump → app deploy"), now expressed as first-class release membership rather than
a separate lane: the gem and its consumers can be members of the same release,
sequenced by kind + dependencies.
Abandon (revert, never force-push). release is permanent and shared, so a
stuck RC is not thrown away by deleting a branch — it is unwound by reverting.
Release#abandon! drops board membership (members fall back to reviewed,
release → abandoned, the singleton frees up). The git-side remediation is owned
by the conductor/CLI as a documented step:
release (git revert -m 1 <merge-sha>, push) — never a force-push, since
release is the shared persistent branch.reviewed; the e2e culprit goes to blocked.release (a new candidate
accumulates) and re-prepare. main never moved.Distinguish the accountable role (the soul whose rubric governs the stage)
from the executor (who moves it). The heartbeat agent executes by wearing
each lane's hat; accountability maps to a soul; there is exactly one operator
gate — the ship.
Redesigned submitted → shipped (decided, 2026-06-22; review lane re-homed
2026-06-26; assembly moved to Steffon 2026-07-03). The Deploy half is re-homed
by role — the souls bookend: Avi supervises review and ships; Steffon owns the
whole middle (sweep + merge + QA). Review runs a 3-level hierarchy, not a
serial hand-off: Avi is the SUPERVISOR (a thin gate that never reviews) — he
confirms product-acceptance and picks the primary + light pair
(bin/reviewer-select), then spawns both reviewers in parallel as sibling
children (the PRIMARY does the deep technical review, the LIGHT a focused second
read) — review-only: on two
approvals the task stops at reviewed. The merge and assembled are owned
by Steffon (Platform Engineer) via the self-healing qa-release sweep; and
shipped is owned by Avi (who runs the full e2e on the frozen ship SHA)
ahead of explicit ship authority. The senior reviewer pool is {Shannon = UI ·
Carl = backend · Jasper = Web3 · Steffon = DevOps/Platform · Alex = Documentation}
(Alex is both the orchestrator and the pool's launchable Documentation review
seat — one identity). Avi picks the pair with bin/reviewer-select <task>
(wraps ReviewerSelector).
Merge timing (decided 2026-07-03 — reverses the 2026-07-02 fold): a
reviewed task is merged into the persistent release by Steffon's
self-healing qa-release sweep (bin/release prepare, looping the per-task
bin/release merge primitive) — which stamps merged: "release" but moves NO
stage; the reviewed → assembled flip lands only on QA-green. The release
branch is the live RC; main only moves when it ships. The PRIMARY reviewer
does NOT merge — on two approvals (primary + light) with no blocker it
drives the task to reviewed and stops (this is safe because Steffon assembles
promptly; the sweep is idempotent + crash-recoverable via merged). Bias to
action: green tests = go, because release is recoverable by revert, so we
don't fear sweeping there.
| Stage (entity) | Accountable | Progressed by | Action | Gate |
|---|---|---|---|---|
| → submitted (task, entry) | Feature agent | Feature agent | bin/full-suite-check (certify FULL suite + rubocop) → pass bin/dor-check, record checks_run, open PR (base release), move in |
self-gate |
| submitted (task) — REVIEW | Avi (SUPERVISOR) + PRIMARY/LIGHT experts | Avi (SUPERVISOR, never reviews) → spawns PRIMARY + LIGHT in parallel | Avi confirms product-acceptance, then picks 2 reviewers from {Shannon=UI · Carl=backend · Jasper=Web3 · Steffon=DevOps/Platform · Alex=Documentation} by domain fit + a logged, seeded-per-task tiebreak via bin/reviewer-select <task>, assigning 1 PRIMARY (deep) + 1 LIGHT (ReviewerSelector, excluding the QA owner so a reviewer never QAs their own change, the task's builder so a soul never reviews their own work, and busy souls so review never lands on an agent mid-build/review elsewhere — the builder is read from devops.built_by, auto-stamped on the move to building from the soul build-claim actor (--actor <soul>) OR the task's assigned agent_slug so a bare bin/task move <slug> building records the builder with no manual flag, falling back to the → building event actor; busy souls come from bin/reviewer-select --busy a,b,c and/or --busy-auto (a board query of agents on stage=building tasks); KEEP fallback: when the builder + QA-owner + busy exclusions would leave fewer than two candidates, the least-bad ones are kept so a PRIMARY+LIGHT pair is always returned (the decision/log flags it), and a non-soul/non-pool builder is never reported excluded; the pair + primary/light is recorded on the submitted→reviewed TaskEvent.metadata["reviewers"] for the avatars UI). Avi then spawns both the PRIMARY and LIGHT in parallel as sibling children; the PRIMARY runs the deep review, the LIGHT a focused second read; each confirms DoR base tests green, code standards, code smell, scalability, and acceptance. No blocker on either → the supervisor drives the task to reviewed ✅ and STOPS — review-only; the sweep (next row) is Steffon's; a blocker → blocked (rework, with qa_feedback) |
2 senior approvals (PRIMARY = Opus on migration/payment/solana/auth); ⛔ one complete qa_feedback on fail |
| reviewed ✅ — SWEEP (task) | Steffon (Platform Engineer) | DevOps agent as Steffon (qa-release) |
bin/release prepare DETECTS every reviewed task + any assembled straggler off the current RC, ensures a candidate (Release.current_or_open!), and SWEEPS each: gh pr merge its PR into release (SKIPPED when merged: release/main — interrupted-run recovery), record membership + merged: "release" (Release::Conductor.sweep!) — stage stays reviewed. Honors dependencies + producer-first. Nothing detected + nothing active → idempotent no-op. Bias to action: green tests = go (release reverts cleanly) |
deterministic sweep (conflicts surface at PR-merge; a conflicted PR is swept PAST — block-and-move); review gate: only reviewed/assembled tasks sweep (--override = audited review_bypassed) |
| assembled (release) — QA | Steffon (Platform Engineer) | DevOps agent as Steffon (qa-release, same run) |
After the sweep, the pre-QA gate runs the next tier — integration + an e2e smoke (registry qa_test_cmd) on origin/release BEFORE deploying; green → prepare deploys it to QA → Discord QA-deployment note → on QA-green Release::Conductor.qa_green! flips swept members reviewed → assembled (merged stays release) + release assembled |
deterministic suite; ⛔ regression → eject the offender (bin/release eject <task> = detach + block + merged cleared; revert its merge commit) — the REST rides the re-run. prepare waits-for-boot (/up-smoke race) and defers the flip until QA returns 200 — a failure leaves members reviewed for the next self-healing run |
| → shipped (release) | Avi, then ship authority | Avi tests; operator or autonomous kickoff authorizes; conductor deploys | Avi runs the full e2e + highest-tier suite on the FROZEN ship SHA (the exact prod code — fixes "shipped ≠ tested"). A QA-only run (pr-review → Steffon's qa-release) stops here for the operator; Avi's production-deploy act ships a QA-green release, and Alex's full-cycle continues with bin/conductor ship --run. On ship authority: bin/release ship ff's release → main per repo (stamping members merged: "main" as each ff lands — the interrupted-Avi skip signal), deploys → production_smoke → Discord release notes → members shipped (merged stays main) |
🔒 explicit ship authority — after Avi's test confirmation, before deploy; rollback on smoke fail |
Clarifications:
Release::STEP_TEST_TIERS (ownership is disjoint by construction — a tier maps
to exactly one step); ship runs Avi's full-e2e gate on the frozen SHA before
ship authorization (bin/release ship → avi_ship_gate, then confirm, unless
the authorized autonomous workflow passes --yes).assembled now means ONE thing at both scopes: QA-green. A task flips
assembled only when the QA deploy it rides smokes green
(Release::Conductor.qa_green!) — being merged into release alone leaves it
reviewed + merged: "release". The release assembles in the same flip.
Production authority is at ship (after Avi's full-suite run on the frozen
SHA), not here — at this scope assembled is a state the conductor flips,
not a human approval.reviewed and stops — review-only; the merge belongs to
Steffon's sweep. Avi never reviews the code himself.
The selection tiebreak is seeded per task (reproducible, not
process-random) and logged (auditable), so reviews spread across the pool
instead of always landing on the obvious domain owner. Because the seed is
derived from the task identity (+ its exclusions), bin/reviewer-select's
preview matches the pair recorded on the submitted→reviewed TaskEvent —
the CLI and the in-app recorder roll identically and never disagree. (That
match holds for the default QA owner; passing a custom --qa-owner changes
the candidate pool + seed, so the preview is then advisory.)assembled step — one soul cannot both review and
QA the same change. (Steffon remains a valid reviewer for other PRs,
especially DevOps/Platform ones.)alex seat is the orchestrator who
also holds the Documentation domain review seat — one identity (seeded in
db/seeds/02_agents.rb, a launchable review agent). ReviewerSelector /
bin/reviewer-select pick alex for docs-shaped PRs (the QA-owner and
builder exclusions still apply, so Alex never reviews a change he built).release
once it carries the two approvals (primary + light) — the consequence of
"Review + QA, gate prod". Risk raises scrutiny (the PRIMARY review goes to
Opus + full integration/security suite), not a second human.
config/release_builder.yml gates only QA assembly autonomy; adding a separate
human pre-sweep gate for payment/solana would require a code/config
change, not a doc-only knob.conductor-review and
routes to a human Avi/Steffon session instead of approving the task into the
sweep queue.Resolved: release_train → release_slug (one field/model); feature PRs
merge into a persistent per-repo release branch, membership recording at
Steffon's sweep with the assembled flip on QA-green (2026-07-03);
no per-task QA stage; Release is its own singleton model — states assembling →, where explicit ship authority Makes the release from
assembled → shipped
the assembled RC. Decided 2026-06-22 (§1.2): review is delegated by Avi to
two seniors (not his solo gate); the merge + assembled are owned by
Steffon (titled Platform Engineer); and at shipped Avi runs the
full e2e + highest tier on the frozen ship SHA before ship authority is
exercised — so the deploy gate sits after test confirmation, before the
deploy.
RC assembly autonomy is the one evolving policy — so it lives in one
tunable config file, config/release_builder.yml, read by
Release::BuilderPolicy. Current policy:
migration/payment/solana risk tag.production_ship.operator_gated is true) for a QA-only run (pr-review →
Steffon's qa-release). Alex's full-cycle launcher is the explicit
autonomous production authorization; it uses the same frozen-SHA/test/smoke
gates, then passes --yes to the production ship command.Change thresholds in config/release_builder.yml, then run
bin/rails test test/models/release/builder_policy_test.rb. This policy only
decides QA assembly autonomy plus names the autonomous production kickoff;
ordinary bin/release ship remains separately gated unless that kickoff or
another explicit production rollout prompt grants ship authority.
The /deployments board and /stages page surface a short copy-paste command
per DevOps stage (source of truth: ApplicationHelper#devops_kickoffs). Pasted
into an agent session run from /Users/alex/projects, each kicks off that
stage's workflow. The feature-agent lane (designed → building → blocked →) has none — the operator drives those hands-on. The DevOps lane maps
submitted
each command to a deterministic runbook. The release-wide launchers are the
soul heartbeat acts on the /deployments Heartbeats card
(ApplicationHelper#heartbeat_launchers; every row is a recognized launcher —
see "The three soul heartbeat launchers" below):
pr-review / pr-review-slow (Avi Heartbeat acts) — review ALL
submitted PRs, in waves of ≤5 or serialized one PR at a time.
Review-only: approved work stops at reviewed — the merge belongs to
Steffon's sweep.qa-release (Steffon Heartbeat act) — the self-healing sweep: merge
reviewed tasks + assembled stragglers onto release, deploy QA, and flip
members assembled only on QA-green.production-deploy (Avi Heartbeat act; ship authority) — ship a
QA-green release: fast-forward release → main and deploy prod; idempotent
no-op when nothing is ready.full-cycle (Alex Heartbeat act; full ship authority) — the whole
release, review → assemble → QA → prod ship.deploy-with-task (Avi Heartbeat act; ship authority for ONE task) —
expedite ONE task to prod. Guarded on a clean release (release == main); on
a dirty release it refuses and points at the full release pipeline
(full-cycle) instead. Launched bare it asks "What task?".The launchers above are not monoliths — they are a small set of atoms plus
compositions of them, so the same building blocks recombine instead of each
flow carrying its own copy of the runbook. Learn the atoms once; every launcher
is a sequence of them.
Atoms (the indivisible steps — each maps onto an existing bin/release verb
or the review-one SOP; none is a new command to build):
| Atom | Is | Command / SOP |
|---|---|---|
review-one <task> |
the PRIMITIVE — the Modular PR-Review SOP on ONE PR (Avi reviewer-select → spawns PRIMARY + LIGHT in parallel → all-clear = the supervisor drives reviewed and STOPS — review-only; else block) |
pr-review-sop.md |
pr-review |
review-one fanned across all submitted PRs, in waves of ≤5 (review-only — Steffon's sweep merges) |
agents/avi/sops/pr-review.md |
pr-review-slow |
the same, serialized — one PR at a time | agents/avi/sops/pr-review-slow.md |
qa-release |
the SELF-HEALING sweep: detect reviewed + stragglers → merge PRs into release (skip merged: ones) → pre-QA gate → deploy QA → members assembled on QA-green |
agents/steffon/sops/qa-release.md / bin/release prepare --yes |
archive-shipped |
archive shipped work and reclaim completed worktrees from prior cycles | agents/steffon/sops/archive-shipped.md / bin/release archive --yes |
production-deploy |
ff each repo release → main (members stamp merged: "main"), deploy prod, smoke, release notes (members → shipped), post-ship agent-docs sync — ship-authority gated |
agents/avi/sops/production-deploy.md / bin/release ship |
Compositions (the operator-facing launcher phrases = a sequence of atoms):
| Composition | Expands to |
|---|---|
full-cycle (Alex Heartbeat act; full ship authority) |
agents/alex/sops/full-cycle.md: pr-review → qa-release → production-deploy — the whole release, review to prod. Formerly the retired Merge, Assemble, Deploy chip; named full-cycle to avoid colliding with the read-only bin/devops-cycle snapshot tool. |
deploy-with-task (Avi act; ship authority for ONE task) |
agents/avi/sops/deploy-with-task.md: GUARD release == main → review-one <task> (→ reviewed) → qa-release (sweeps + merges it) → production-deploy. Interactive — launched bare it asks "What task?". Formerly the Deploy with Task <task> write-up here. |
Retired chips (2026-07-02). The four legacy release-card chips —
Avi Heartbeat,
SlowAvi Heartbeat Fast,Build and Deploy QA Release,Merge, Assemble,— were removed from the UI and relocated into the soul heartbeat acts:
Deploy
serialized review is now Avi'spr-review-slowact, QA-only ispr-review→
Steffon'sqa-release, and the full autonomous run is Alex'sfull-cycle
act. (2026-07-03: the acts went back to review-only — the merge now lives in
Steffon's self-healingqa-releasesweep.) Thebin/pr-reviewreview-only
loop still exists but is no longer a card chip.
The only NEW code this set required is the clean-release GUARD that
deploy-with-task runs first (bin/release status --clean-only, backed by the
unit-tested Release::CleanCheck); everything else is the atoms recombined.
The standalone Heartbeats card (tasks/_heartbeats_card on /deployments,
sized to match the Next Release card) renders the three soul heartbeat launchers
(ApplicationHelper#heartbeat_launchers, one tasks/_heartbeat_launcher per soul)
in a 3-up grid. Each launcher is a soul face (linking to /agents/<slug>) over
a prompt-like row 1 (Avi Heartbeat / Steffon Heartbeat / Alex Heartbeat)
plus one or more copyable action rows, each with a leading icon (❤️ on the
heartbeat row; 1️⃣–4️⃣ on the four ordered release actions, a themed glyph on the
rest). Any row, pasted into a fresh session, is a recognized launcher. The
5-stage release tracker stays in the Next Release card. Cross-soul launcher
map: heartbeats.md. Per-soul heartbeat launchers
live with the souls:
Avi,
Steffon, and
Alex.
Sticky attribution. The FIRST action of a
<Soul> Heartbeatis
bin/agent-activity heartbeat <soul>— it sets a session-sticky acting-agent so
every activity self-attributes to that soul (stacked over the base mascot) without
re-passing--agent; an explicit--agentstill wins, and it clears at session
end (close-open) orheartbeat --clear.
| Soul (row 1) | Acts | Does | Exit seam |
|---|---|---|---|
Avi (Avi Heartbeat) |
production-deploy · pr-review · pr-review-slow · deploy-with-task (direct-invoke only) |
downstream-first: ship a QA-green release (bin/release ship --yes, stages 4–5, stamping merged: "main" at each ff) if one is ready; then review submitted PRs — review-only (waves ≤5, or serialized via pr-review-slow) |
the ready release shipped (or no-op); then each PR reviewed/blocked |
Steffon (Steffon Heartbeat) |
archive-shipped · qa-release |
downstream-first: archive shipped tasks (bin/release archive --yes) from the prior cycle; then the self-healing sweep — merge the reviewed queue onto release, pre-QA gate, deploy QA, flip members assembled on QA-green (bin/release prepare --yes, stages 1–3) |
prior cycle archived (or no-op); then RC deployed to QA, members assembled |
Alex (Alex Heartbeat) |
grade-events · share-insights · full-cycle |
grade the 10 most recent resolved activities at /alex/heartbeat; share the mcr-confirmed insights out (regenerate the lessons doc + distribute); OR run the whole cycle review→assemble→QA→prod ship (full-cycle, full ship authority) |
10 graded + insights banked; confirmed insights shared out; or the whole release shipped |
The release handoff seam. The pizza-tracker (RELEASE_TRACKER_STAGES) is five
stages: 1 Testing, 2 Assembling, 3 Deploying QA, 4 Confirming, 5
Deploying (prod). Steffon owns stages 1–3 (qa-release = bin/release) and stops at Live on QA; Avi owns stages 4–5 (
prepareproduction-deploy
= bin/release ship) and finishes at Deployed. The seam between them —
"deployed to QA." — is the Steffon → Avi handoff: Steffon's qa-release
ends and reports there; Avi's production-deploy starts only once it is true.
These are operator-launched (copy-paste) today, schedule-ready tomorrow — each
act is idempotent, with an explicit precondition + a named exit seam, so a
scheduler can fire it later without rework (see heartbeats.md).
Note pr-review is review-only (stops at reviewed) — Steffon's qa-release
sweep owns the merge and flips members assembled on QA-green.
Per-act procedures live in the registered SOP files, not here. Each act in
the atom and composition tables above links its owning
docs/agents/agents/<agent>/sops/<sop>.md, and every SOP is executable
standalone. This section keeps only the architecture: the atom/composition map
and the per-stage building blocks the deploy CLI implements. (The retired
composition write-ups that lived here — Avi Heartbeat Slow/Fast,
Build and Deploy QA Release, Merge, Assemble, Deploy — were absorbed into
pr-review-slow, pr-review, qa-release, full-cycle, and
deploy-with-task.)
A non-interactive agent MUST pass
--yesonly for approved confirms. An
agent's shell has no TTY — stdin is EOF, which a confirm prompt reads as
"no". The consequence differs per release verb:
-prepareaborts without confirmation in a non-interactive shell. Always
runbin/release prepare --yesfor the approved QA deploy step.
-shipaborts loudly without confirmation — that is intentional. Do not
pass--yesunless the session launched a ship-authority SOP
(production-deploy,full-cycle,deploy-with-task) or Mr. McRitchie
explicitly gives the production ship go in this session.
-statusis a read-only report (no confirm);--clean-onlymakes it a
GATE that exits non-zero on a dirty release. It never deploys, so it needs no
ship authority.
-archivecan use--yesafter the shipped release is verified and the
operator has approved cleanup.
-mergedoes not prompt today;--yesis harmless future-proofing.
--yesbypasses the human confirm only — it never skips a test gate
(avi_ship_gateruns and can still abort the ship).--prodis already the
default (the board is prod) — don't add it redundantly.
The per-stage commands below are the building blocks the compositions above
sequence:
One-time setup (per machine/clone). Run bin/release init once: it
creates the persistent release branch (= origin/main) on every gem + app repo
in config/release_repos.yml that doesn't already have one. Idempotent — a repo
that already has origin/release is skipped.
Review submitted PRs (submitted → reviewed — review-only)
Review is a 3-level hierarchy (§1.2): Avi is the SUPERVISOR — a thin gate
that never reviews — who spawns the PRIMARY and LIGHT reviewers in parallel as
sibling children and collects both verdicts (the merge is Steffon's sweep, not the
reviewer's). Not a solo avi pass, and not a serial primary→light hand-off. The
formalized, agent-role how-to for
this cascade — Avi assigns the pair, each reviewer narrates its review as its
soul (--agent) into the heartbeat's Agent column, any reviewer can block — is
the reusable PR Review SOP module; this section
is its release-context anchor. For each submitted task (bin/task list or the
board):
release) meet the task's acceptance criteria? — and
picks the pair (step 2). He never runs the deep technical review.bin/reviewer-select <task> — it loads
the app and scores the pool {shannon, carl, jasper, steffon, alex} by
domain fit (the task's shape + repositories + risk tags vs each soul's
domains) with a logged, seeded-per-task tiebreak, returns 1 PRIMARY + 1 LIGHT, and
excludes the QA owner (Steffon, who QAs the assembled RC — no self-gating),
the builder (read from devops.built_by, auto-stamped on the build move
from the assigned agent_slug — so a soul never reviews their own work with
no manual flag), and any busy souls you name. alex is the
orchestrator who also holds the launchable Documentation review seat — one
identity. (--qa-owner SLUG excludes a different soul when Steffon isn't the
one QAing this task; --builder SLUG overrides the recorded built_by;
--busy a,b,c and/or --busy-auto (a board query of agents on
stage=building tasks) drop agents mid-build/review elsewhere — the pool is
never starved below a pair, the least-bad are kept back; --json for a
machine-readable pick; --record writes the picked pair onto the task as a
review intent so /deployments + the task timeline show the two seniors
reviewing live — a green ticking timer — the moment Avi kicks off review,
before →reviewed lands. With a custom --qa-owner or a non-empty --busy
set the preview is advisory — --record it so the busy-aware pair is the
one the timeline shows.)migration/payment/solana/auth)
— diff-vs-acceptance + code standards + code smell + scalability, plus
confirming the shape's base tiers are green — while the LIGHT gives a
focused second read. Both experts are the supervisor's children (siblings, not
a serial hand-off), running concurrently. Honor the ≤5-concurrent cap (operating model): across all PRs in
flight, keep at most 5 review agents running at once — a queue wider than ~2 PRs
reviews in waves of ≤5, never the whole batch at once.reviewed (bin/task move
<task> reviewed --actor avi) and STOPS — review-only; Steffon's
qa-release sweep merges the PR onto release and flips the member
assembled on QA-green. Any reviewer
blocks → bin/task block <task> --kind rework --feedback "…" (one complete
send-back). Bias to action: two green approvals = go (release reverts
cleanly, and the sweep follows promptly).Agentic intent — the live "who's on it now". Each event carries the agent
that STARTED it, not only the one that completed it, so /deployments and the
task's consolidated Stage Timeline show who's working right now with a
green ticking timer — the Deploy mirror of the build lane's live counter. These
are append-only TaskEvents of kind: intent (completed transitions stay
kind: transition; named non-moving lifecycle completions such as heavy/light
review verdicts use kind: checkpoint; and an intent never enters the duration spine); an intent is
"open" only inside the source-stage cycle where it was recorded. A later
transition into its target stage supersedes it, and leaving the source stage
(for example submitted → blocked) closes it even if the target never landed.
If QA blocks a PR for rework and the feature agent rebuilds/resubmits it, Avi can
record a fresh →reviewed intent for the second review round. The build lane is
the special viewer case: building is both the stage-change conclusion and the
"agent started working" signal, so the task detail timeline marks the existing
Designed → Building card live instead of rendering a duplicate live
Building card. The Created → Designed genesis row stays deterministic and
usage-free; design accounting belongs on the Designed → Building transition
because bin/task create seeds the usage baseline only after the task slug
exists. The build-lane face is the task's
Pokémon mascot (assigned at create). The
review pair is recorded by bin/reviewer-select <task> (step 2 — recording
is the DEFAULT now; pass --no-record/--dry for an advisory-only preview);
Steffon's QA and Avi's ship intents are auto-recorded by the deploy CLI —
bin/release prepare fires the assembled intent (actor: steffon) and
bin/release ship the shipped intent (actor: avi), both via
Release::Conductor.record_deploy_intents! over every release member (the Deploy
mirror of bin/reviewer-select's default review-intent write). So the conductor
no longer hand-runs them — the 2026-06-25 unfilled-ship-slot incident (a missed
manual bin/task intent --to shipped --actor avi left the ship crew slot blank
mid-deploy). The manual bin/task intent <task> --to assembled --actor /
steffon--to shipped --actor avi (or POST) stays as the fallback / one-off path. All of them
/api/v1/tasks/<slug>/intent
are append-only + idempotent — an identical open intent in the current
source-stage cycle is reused, and the call is a no-op once the target stage has
landed in that cycle. Actor-less
conductor moves on assembled/shipped still attribute to their role owners
(Steffon QAs assembled, Avi ships) so the Deploy crew never goes blank.
Prepare release (reviewed → assembled — Steffon's SELF-HEALING qa-release)
ONE deterministic verb — bin/release prepare --yes [--task SLUG ...] — owns the whole middle:
[--slug rel-…] [--prod]
reviewed task + any assembled straggler not riding the
current RC (Release::Conductor.sweep_candidates). Nothing detected and no
active release → idempotent no-op (report + exit 0). --task narrows the
sweep to the named slugs (operator curation).Release.current_or_open!; --slug names a fresh one).release, gh pr merge it — SKIPPED when merged: release/main (an
interrupted prior run already landed it; a failed gh merge also falls back to
the PR's real state) — then record ALL memberships in one heroku run
(Release::Conductor.sweep!: release_slug + merged: "release", stage
stays reviewed). The record write rides an ensure, so every PR that DID
merge is recorded even if a later merge aborts. A merge conflict is
block-and-move: that task is left reviewed (resolve on GitHub or block it)
and the REST of the sweep proceeds; release is never force-pushed. Gem PRs
merge into their own repo's release like any other. The overlap planner
(warning only, ≥2 PRs) prints pairwise file collisions + a suggested order +
likely rebases (Release::MergePlan.compute). The per-task primitive stays
available as bin/release merge <slug> [<slug> …] --yes (same sweep
semantics; --override = the audited review_bypassed bypass). The sweep
records the reviewed→assembled intent, so assembly duration caches measure
from the sweep to the QA-green flip.qa_test_cmd (the integration +
e2e-smoke tier prepare owns — Release::STEP_TEST_TIERS) runs on
origin/release BEFORE anything deploys. A regression → eject the
offender (bin/release eject <task> --feedback "…" — detach + block +
merged cleared — then revert its merge commit on release) and re-run: the
sweep self-heals and the REST of the RC rides on. Unset qa_test_cmd = the
repo self-gates (skip).Release::Conductor.record_deploy_intents!, append-only + idempotent) so
/deployments shows Steffon QA-ing live; runs the per-app merge-forward
guard (keeps each repo's release ahead of main); then bin/qa-server
deploy <qa_app> origin/release per app member — gem members are
skipped (no app artifact; they ride the record and are QA'd via a consuming
app). Records release.qa_url + per-repo QA SHAs, waits for boot
(wait_for_boot polls /up), then runs each member's declared
devops.post_deploy_cmd on its QA heroku app (heroku run, records the
[post-deploy] outcome, aborts on non-zero — the {task, app, cmd} plan
is the unit-tested Release::PostDeploy.plan).Release::Conductor.qa_green! flips the swept members reviewed →
assembled (merged stays "release") and the RC assembling→assembled. A
QA failure flips NOTHING — members stay reviewed, the RC stays
assembling, and the next self-healing run picks everything back up
(skipping the already-done merges).Record ops run on the prod board by default (the board IS production) via
heroku run; --local opts into the stale local DB.
Run Deployment (assembled → shipped — promote the QA'd RC to prod)
Run bin/release ship [--by NAME] --prod. Without --yes it confirms
before deploying; under the full-cycle launcher (or another
explicit production rollout prompt), use
bin/release ship --by conductor --yes. --yes skips only the confirm prompt.
Preflight FIRST (before any fast-forward): ship asserts every app checkout
is on a clean main and aborts loudly — naming the offending branch / dirty
files — if any isn't. ship ff's each repo's main up to the QA-frozen SHA, so a
checkout a review agent left on a leftover pr-NNN branch or with an uncommitted
stale schema.rb would otherwise break the ff mid-ship (after gems published +
the ship authorization — the worst time). The preflight catches it up front,
before anything irreversible. Pure decision:
Release::ShipSequence.preflight_offenders / .preflight_message. Live ship
crew: right after ship authorization (so a declined gated ship never shows it),
ship auto-records the Avi → shipped intent for every member
(Release::Conductor.record_deploy_intents!(r, to_stage: "shipped", actor:) so /deployments shows Avi shipping live — a green ticking timer — through
"avi")
the whole deploy instead of an empty dashed ship slot until ship! lands (the
2026-06-25 incident). Append-only + idempotent (ship! supersedes it; a
partial-ship abort leaves it open — correct, Avi is still shipping — and a re-run
reuses it). Producer-first: before any app deploy, it publishes every
gem member to RubyGems in order — for each it prints the gem + target
version and asks Publish <repo> <version> to RubyGems? (approval-gated; honors
--yes / --dry-run), runs the gem's build (studio-engine: bin/release-check; otherwise
--buildgem build <gemspec>), gem pushes the artifact, and tags
v<version> in the gem repo. A build/push failure aborts the ship before any
app deploys, so apps never deploy against an unpublished gem. Then for the apps
it fast-forwards each repo's main up to release (so release collapses into
main), pushes origin — stamping that repo's members merged: "main" as
each ff lands (best-effort; the interrupted-run skip signal — a re-run's ffs
no-op and ship! re-stamps it regardless) — deploys (git push heroku main;
release phase runs migrations), and smokes /up. After every app deploys + smokes (and before the
shipped record), the post-deploy hook runs each member's
devops.post_deploy_cmd on its production app via heroku run, records the
[post-deploy] outcome, and aborts ship on a non-zero exit — the abort
lands before ship!, so the release stays assembled (recoverable) and a re-run
resumes (the command is expected idempotent). On success it stamps deployed_sha,
flips the RC + its members to shipped (Release::Conductor.ship!), and
auto-posts release notes
(Release::Conductor.post_release_notes → the same Formatter/Discord path as
POST /api/v1/release_notes; non-fatal if the webhook is unset). After a ship,
each repo's release equals main and re-accumulates the next candidate. Run
ship from a primary checkout (not a worktree): the gem repos are resolved
as siblings at the projects root.
Post-ship agent-docs sync (the OWNED installer run). After the primaries are
restored to the freshly shipped main, ship auto-runs the hub primary's
bin/install-agent-docs (sync_agent_docs, ship step 7b) — the owned
pipeline step that keeps the installed docs (~/.claude + ~/.codex skills,
the projects-root AGENTS.md/CLAUDE.md) in sync with what shipped, so an
adapter/skill/SOP merge no longer drifts until someone happens to run the
installer by hand. It is post-SHIP by design — the installer reads the LOCAL hub
checkout's docs, and only after the ff release → main + restore does the
primary's main hold the merged docs (a qa-release-time / prepare-time run
would install main's stale docs) — and NON-FATAL by construction (rescue-and-warn; a docs
sync never aborts a completed ship). Owner: Steffon (infra) owns the step and
its mechanism; it runs inside whichever act drives bin/release ship
(production-deploy / full-cycle). If the step warns, the fix is running
bin/install-agent-docs from the hub primary by hand.
Archive completed tasks (shipped → archived — the Deploy loop's conclusion)
Run bin/release archive [--dry-run] [--yes] [--prod] to close the loop. It
archives every shipped task that is not a member of Release.last_shipped
(shipped → archived), so the most recently shipped release stays on the board
as the read-only Last Release while older, superseded completed work is filed
away. The pure, unit-tested rule lives in
Release::Conductor.archive_completed! / .archivable_completed_slugs; the CLI
owns the board write plus the worktree teardown around it. After archiving it
reclaims the merged/shipped feature worktrees (bin/agent-worktree cleanup).
--reclaim --yes--dry-run previews the plan (archivable + kept) and the
reclaim list without mutating anything; --yes runs it hands-off (skips the
single confirm). Idempotent — a re-run finds nothing new to archive. Archiving
only flips a task's stage, never its release_slug, so the board's Last Release
section keeps linking to its members even after they're later archived,
preserving the release history. shipped is therefore no longer terminal —
the Deploy loop now closes at archived. The ledger commits itself: after the
reclaim appends to delete-later.md, archive commits that update to release
(best-effort, only when the ledger is the sole uncommitted change — pure guard
Release::ArtifactCommit), so it ships next round instead of piling up as
ship-preflight stash dirt the conductor has to park.
Release retro (post-ship "review & learn" — completely NON-BLOCKING)
Run bin/release retro [release-slug] [--worked "…"] [--friction "…"] [--followup after a ship to capture what the release
"…"] [--file-tasks] [--yes] [--dry-run]
taught us. It defaults to the current / most-recently-shipped release, auto-gathers
the release record (member tasks + kinds, per-member submitted → shipped cycle
timing from TaskEvents, rework rounds = bounces into blocked, reviewers, and
recorded checks_run), prompts a few judgment questions (what worked / what caused
friction / follow-ups — --worked/--friction/--followup supply them from args,
--yes runs fully non-interactive), and writes a durable doc at
docs/agents/audits/retro-<slug>.md, then commits that doc to release
(best-effort, non-fatal, only when the doc is the sole uncommitted change —
Release::ArtifactCommit) so the generated retro ships next round rather than
becoming ship-preflight stash dirt. --file-tasks opens each follow-up via
bin/task create. The gather + render rule is the pure, unit-tested
Release::Retro (.gather / .render / .write_doc); the CLI reaches it through
the same read-only conductor runner and writes the returned markdown to the local
tree. It writes no agent-memory store — the doc (+ any filed tasks) is the only
record. Retro is decoupled from the pipeline by design: archive does not
depend on, trigger, or wait for it, so the loop closes whether or not a retro was
run. Unlike ship/archive, retro never deploys or mutates the board, so it does
not gate on --yes.
Both ride the same stage machine. They differ at entry and in test emphasis.
Routing lives in AGENTS.md (see §6) so an agent self-loads the right one.
test_plan = the shape's required tiers.bin/dor-check) — --gate
build before you start coding, --gate merge before handoff (§3.3).checks_run, hand off with a handoff note, move to submitted.Every
bin/task moveleaves a paper trail — for free. Each stage change
appends aTaskEventcapturingfrom → to, the timestamp, and the time spent
in the prior stage (the deterministic spine; it renders as the Stage Timeline
on the task page). You do nothing to get it. To also attribute model cost to a
transition, add the optional per-transition usage on the move:
bin/task move <task> submitted --model claude-opus-4-8 --tokens-in N --tokens-out N --cost D.
Usage is best-effort and opt-in; the spine is recorded regardless (and for
non-agent moves too). Details:
task-board-api.md.
hotfix (production broken / funds at risk) vs normal.hotfix may go straight to building and use an expedited review, but
never skips the regression test or the operator ship gate.Why regression-test-first for bugs: it both proves the fix and permanently
pushes that class of bug down the pyramid, shrinking future PR-stage churn.
Not every app is a managed satellite. A standalone / client app uses the
studio's process — the task board, worktrees, the multi-agent merge patterns,
and the evergreen build conventions — but owns its own runtime and may be
handed off to a client. It rides the same Build workflow
(designed → building → submitted). Its Deploy half has two modes:
Release-managed standalone (Rolio, as of 2026-06-27):
release once bin/release init has created the branch.origin/release, and
bin/release prepare deploys origin/release to the app's QA Heroku target
from config/qa_environments.yml.bin/release ship --by conductor fast-forwards
release → main, pushes production, and smokes the app's
prod_deploy.smoke_url.studio-engine or hub SSO requirement.App-owned standalone:
main, not release — there is no persistent release branch
and no release slug. bin/agent-worktree already falls back to origin/main
as the base for any repo without a release branch.main by the
app's owner; it is not assembled into a studio RC.bin/dor-check ceremony. (Robust
error/API-failure logging is evergreen for both tiers — managed apps via
studio-engine's rescue_and_log/ErrorLog, standalone apps via plain
Rails.logger and/or their own tracker.)Full tier decision + phased checklist:
new-app-onboarding-sop.md. Multi-agent build/merge
patterns (several agents scaffolding one app in parallel):
../modules/worktrees.md → Multi-Agent Safety &
Merge Patterns.
Shapes are deployment-agnostic.
config/feature_shapes.ymland the
shape→tier contract (§3) classify the kind of change (ui-only, backend,
library, …), not the deploy tier — so they apply unchanged to a standalone app.
The shape still selects which tiers you write; the standalone tier only changes
where the PR lands and who ships it. Nofeature_shapes.ymlchange is needed.
Your insight: the pyramid must adapt to the nature of the feature, from one
general strategy, across all five repos. Three pieces: tier definitions (the
what), the shape→contract matrix (the adaptation), and the DoR gate (the
enforcement).
| Tier | General definition | Rails apps | studio-engine | solana-studio | turf-vault |
|---|---|---|---|---|---|
| Unit | Pure logic, no I/O | model/service/PORO/decoder specs | pure lib (ColorScale, Email…) |
Borsh/keypair/tx builders | single instruction handler logic |
| Component | One behavior + its immediate collaborators, no full stack | request/controller specs + rendered partial + Alpine factory | UI primitive via a host harness | client method w/ stubbed RPC | instruction + its account constraints |
| Integration | Multiple objects across a boundary | request→DB→job, RPC-mocked Solana (FakeVault) |
consumer-CI against both apps | client against test-validator | multi-instruction lifecycle (create→enter→settle) |
| E2E | Real browser / real chain | Playwright | (via consumers) | (via consumers) | devnet on-chain spec |
| Manual | Operator visual/UX acceptance | the release QA stop (eyeball the assembled RC, then Make the release) |
— | — | contract transparency / /contract review |
Tiers are the what; the existing test lanes are the when/where.
Mapping: Unit+Component+Integration → pr_review_gate/local_proof (block
merge); E2E happy-path → local_proof, full E2E → nightly_deep; Manual →
qa_acceptance; post-deploy → production_smoke.
A feature's shape is recorded in devops.shape. It selects the minimum
tiers that must be green by the time the task is submitted for review:
| Shape | Example | Required tiers (DoR contract) |
|---|---|---|
| ui-only | "make the button blue" | Component (rendered partial / Alpine) + Manual at QA. Unit only if it adds logic. |
| ui+db | new form that persists | Unit (model/validation) + Component (request+view) + Integration (request→DB) + E2E happy path |
| backend | new job/service | Unit (service/PORO) + Integration (job + mocked I/O) |
| library | studio-engine change | Unit in engine + consumer-CI (component/integration in both apps) |
| onchain | new turf-vault instruction | Anchor unit + Anchor integration (lifecycle) + Ruby decoder unit + devnet E2E (nightly) |
| onchain-vertical | new workflow w/ wallet + DB + UI + program | all tiers + devnet E2E; almost always its own release |
The matrix is the single source of "how much testing is enough" — it removes
the per-task judgment call that currently lets thin PRs through.
A task may not advance submitted → reviewed unless, for its shape:
checks_run;rubocop are certified green against the
exact code being shipped — not the touched-file subset. The shape's tier tags
prove the agent wrote unit/integration; they do not prove nothing else
broke. bin/full-suite-check <task> runs bin/rails test and bin/rubocop in
full and stamps fingerprint-bound [full-suite@<fp>] / [rubocop@<fp>]
checks_run lines; bin/dor-check re-grades them against the current code
fingerprint (a git tree hash — content-addressed, so it is stable across the
pre-commit→commit boundary and identical in a reviewer's fresh checkout of the
same tree), so a stale (edited-since) or partial (one-lane / touched-files)
record is refused. Both gates root the CODE they run + fingerprint at the
current worktree (the cwd's git toplevel), so a satellite task (turf-monster,
rolio) certifies its OWN repo even though it runs the hub's gate script — while the
shape config (feature_shapes.yml) stays resolved from the studio. Run
bin/full-suite-check and bin/dor-check from the worktree (the FULL_SUITE_ROOT
/ DOR_CHECK_DIFF_ROOT envs override the root; they are a CI/test seam, not for
routine use). Escape hatch — a record, exactly like post_deploy_cmd: none: a
reasoned [full-suite-bypass] <why> checks_run line passes the gate but is
flagged loudly in the verdict (use it for a pre-existing, unrelated red
tracked elsewhere — never to wave through your own break);metadata["devops"] fields are populated (existing contract);db/seeds,
db/migrate/), the task declares devops.post_deploy_cmd — the command
bin/release runs on the deployed app (QA on prepare, prod on ship) so a
seed/backfill isn't run by hand post-ship. Heroku's release phase auto-runs
db:migrate but not db:seed or a backfill rake; set post_deploy_cmd to
none to acknowledge a schema-only migration that needs no command.post_deploy_cmd safety rule (both gates): bin/release runs the command
verbatim against PRODUCTION, so it must be narrow, prod-safe, and
idempotent. A declared post_deploy_cmd is rejected when it is a bare
full-suite seed — bin/rails db:seed, rails db:seed, bundle exec rails
db:seed, db:seed:replant, or rake db:seed. db/seeds.rb loads every
db/seeds/*.rb, so a bare seed would inject demo News/Content/Tasks into prod
and abort the release on the first non-idempotent seed file. Declare a
narrow command instead: a scoped single-file runner —
rails runner 'load Rails.root.join("db/seeds/NN_x.rb").to_s' — or a
dedicated idempotent rake task (e.g. bin/rails pokemon:seed). This is the
fix for a real near-miss: merge-docs-reviewer-into-alex shipped
post_deploy_cmd='bin/rails db:seed' and was caught only when QA aborted,
because reviewers read the code diff, not the deploy metadata.This is deterministic — a bin/ gate (bin/dor-check <task>, default
--gate merge), not a judgment call. There is also a lighter --gate build
(spec-complete, no tiers) for the designed → building entry. The feature agent
runs it before handoff; the heartbeat agent re-runs --gate merge as gate zero
of review (the fingerprint-bound full-suite evidence is checkout-independent, so
gate-zero credits the same evidence the feature agent recorded). A failed DoR is
an immediate, cheap send-back that never consumes review-judgment tokens. This
is the structural fix for the review ping-pong: most "PR not ready" churn becomes
a pre-PR mechanical check.
On the merge gate, dor-check also verifies the PR is open and reads its
real GitHub CI (gh pr view --json state then gh pr checks, folded to one
verdict by bin/lib/ci_status.rb): a failing, still-running, or
closed/merged PR is refused — a closed PR's green checks are historical, not
a live target, so a stale pr_url never passes as green — an all-green open PR
passes, and a task with no PR yet stays silent (nothing to verify). This closes the blocker-analysis
#1 class — a PR green locally but red on CI, because the local cert runs
bin/rails test and not the browser test:system lane GitHub also runs. It
rides the existing gate-zero re-run: the feature agent's pre-PR run is silent on
CI, but the heartbeat's --gate merge gate zero runs after the PR is up and
refuses a red (or not-yet-green) PR before any review-judgment tokens are spent. A
gh/network error or a PR with no checks degrades to a note, never a hard block
— we don't trade a flaky CI lane for a flaky gate.
bin/dor-check itself stays a fast, deterministic verdict — it does not
run the suite; bin/full-suite-check is the (slower, run-once-before-handoff)
runner that produces the evidence (format + fingerprint live in
bin/lib/full_suite_gate.rb). It closes the retro gap where a build passed only
the files it touched while the full suite or rubocop broke post-merge. For
those who want the lanes wired locally, bin/full-suite-check --install-hook
installs an opt-in pre-push hook (off by default; runs the gate before each
push, blocks a red push; remove with --uninstall-hook) — pre-push, not
pre-commit, because a full suite on every commit is untenable. But the
authoritative gate is bin/dor-check validating the recorded evidence: the
hook is a convenience, and evidence on the task record survives a fresh checkout
where a local hook artifact would not.
| Tier | Author | When |
|---|---|---|
| Unit | Feature agent | During build, before first commit |
| Component | Feature agent | Before submitted |
| Integration | Feature agent | Before submitted (mandatory for any migration/solana/payment/auth risk tag) |
| E2E (happy path) | Feature agent | Before submitted for ui+db / vertical shapes |
| Full suite + rubocop | Feature agent | Before submitted — bin/full-suite-check <task> certifies the WHOLE suite + lint (not the touched-file subset); records fingerprint-bound evidence bin/dor-check re-grades |
| E2E (edge/regression) | QA lane (Avi/Steffon) | May add during review; becomes a follow-up task if large |
| Manual | Mr. McRitchie | At the release QA stop (this is the manual tier) |
Pruning is a recurring chore task owned by the QA/infra lane (Steffon),
on a monthly cadence, tracked like any other task.
quarantine lane + a follow-up task (never silently
skip); redundant → delete the higher-tier test when a lower tier now covers
it (push coverage down the pyramid); dead → remove tests for removed
behavior.bin/devops-tests; daily steering still comes from task status.Runs on the OpenClaw box every ~10 minutes. Builds directly on the
devops-cycle/qa-intake toolchain and the "Future Heartbeats" lease spec.
# Workflow 1 — per task (review). Each submitted task, one safe step.
for each task in {submitted}:
acquire lease (claimed_by, claim_expires_at) # resilience: reclaimable
1. bin/dor-check --gate merge (gate-zero: metadata + tiers + FRESH full-suite/rubocop evidence) — fail ⇒ block(rework) + qa_feedback, release
2. run pr_review_gate suite (base: unit/component) — fail ⇒ classify, block(rework), release
3. Avi delegates: 2 seniors (domain fit + LOGGED tiebreak), 1 PRIMARY + 1 LIGHT — §1.2
each: diff-vs-acceptance + standards/smell/scalability — changes ⇒ ONE complete qa_feedback + block
4. on 2 approvals → reviewed ✅ — Discord: approved
# Workflow 2 — the ONE active release (singleton).
release.assembling:
bin/release prepare (Steffon, SELF-HEALING): detect reviewed + assembled stragglers, honoring dependencies + lanes (§4.2)
sweep: overlap planner (warn) → gh pr merge each (base release; SKIP merged: release/main) → sweep! ALL in ONE heroku run (ensure) — conflict ⇒ leave reviewed, keep the rest
members reviewed + merged:release; pre-QA gate (qa_test_cmd: integration + e2e-smoke on origin/release) — regression ⇒ bin/release eject <task> + revert, keep the rest
deploy origin/release → QA + Discord notes → wait-for-boot /up → QA-GREEN ⇒ qa_green!: members → assembled + release.assembled — failure ⇒ members stay reviewed (next run self-heals)
# full e2e + highest tier runs at ship, on the FROZEN ship SHA (Avi) — §1.2
release.assembled:
Avi: full e2e + highest tier on the FROZEN ship SHA # §1.2 — closes "shipped ≠ tested"
if operator_made_the_release: bin/release ship → PREFLIGHT (each app on clean main, else abort) → ff release → main, bin/deploy → production_smoke → notes → members shipped # ONLY here
else: no-op (HARD STOP — wait for the operator to Make the release)
update last_heartbeat_at, current_command, blocked_reason; emit progress
Properties that give resilience + scale:
claimed_by, claim_expires_at, last_heartbeat_at) →
an interrupted task is reclaimable by the next heartbeat; this is exactly the
interruption-resilience you asked for, at the task level.The heartbeat agent will not merge-race conflicting work:
bin/release merge a b c
prints, before merging, the files each PR shares with the others, a
suggested merge order (smallest-footprint first), and which PRs will need a
post-merge rebase — so siblings that all touched task.rb / a shared helper /
the docs don't conflict on release after passing review. Warning-only (it
never blocks); the conductor reads it to choose order / rebase the loser.db/schema.rb or migrations → serialize
via the existing backend_migration advisory lock; second one holds with a
note.release_slug lane; the agent promotes the
train in order, never a consumer ahead of its gem.release_slug deployed in order
(Squads program upgrade first, then app IDL re-pin via bin/deploy's
allow-list dance). The agent refuses to deploy two program upgrades
concurrently.migration/solana/payment) still land in your one-click queue
with full context — the agent sequences, you approve.Three message classes, deterministic templates with a small freeform
notes slot. Posted freely by the heartbeat agent.
| Class | Trigger | Shape (deterministic) |
|---|---|---|
| Heartbeat digest | every tick (or every N) | 🔄 DevOps tick HH:MM — N in review · M in QA · K awaiting approval. Blockers: … |
| Task event | stage advance / send-back | ✅ <title> merged → QA <url> · ⛔ <title> sent back: <reason> · 🟡 <title> QA-passed — approve to ship: <qa url> |
| Release notes | after prod deploy | existing POST /api/v1/release_notes (already standardized, grouped-by-app, task-linked) |
The 1000ft view: blockers + "awaiting approval" are the only two classes you
must read; the digest is ambient. Webhooks: reuse
DISCORD_RELEASE_NOTES_WEBHOOK_URL; add DISCORD_DEVOPS_PROGRESS_WEBHOOK_URL
for digests/events so release notes stay clean.
Add a routing block to AGENTS.md so a fresh agent self-selects its SOP:
## DevOps Routing
Before implementing, identify your role and read the matching section of
docs/agents/system/devops-cycle-design.md:
- Handling a FEATURE → § Feature SOP. Classify the feature SHAPE and load its
test contract before writing code. Build: designed → building → submitted.
- Handling a BUG → § Bug SOP. Write the failing regression test first.
- Running the airgapped/QA cycle → § Heartbeat agent. One safe step per task;
review moves submitted → reviewed or blocked; never ship a release without the
operator OK.
Everything else the agent needs already loads via the existing Start Here
table. No per-session explanation from you.
Compartmentalize tokens: deterministic scripts carry the 80%; escalate to a
capable model only for genuine review judgment, and only to Opus for high-risk
surfaces.
| Step | Nature | Engine | Model |
|---|---|---|---|
| DoR gate, metadata presence | deterministic | bin/dor-check |
none |
| Run test suites | deterministic | CI / bin/devops-tests |
none |
| Conflict / lane check | deterministic | bin/ + advisory locks |
none |
| Classify a check failure (real / flaky / stale) | light judgment | small model | Haiku |
| QA acceptance evaluation | suite + light judgment | suite + small model | Haiku |
| PR diff vs acceptance review | judgment | capable model | Sonnet, Opus if solana/payment/migration/auth |
| Merge decision | rules-gated judgment | rules + model | Sonnet |
| QA deploy / prod deploy | deterministic | bin/qa-server / bin/deploy |
none |
| Release-notes formatting | deterministic | POST /api/v1/release_notes |
none |
| Release-notes highlights prose | light judgment | small model | Haiku |
| Discord digest / event messages | deterministic templates | script | none |
| Production approval | human | — | Mr. McRitchie |
shape field vs inferring shape from risk_tags — add an explicit
devops.shape field, or derive it? (Recommend explicit; it's the contract key.)hotfix severity that goes
straight to building and shortens review, still regression-tested +
ship-gated? (Recommend yes.)DISCORD_DEVOPS_PROGRESS_WEBHOOK_URL, or
reuse the release channel? (Recommend separate.)Done
bin/dor-check + the shape→contract matrix in config/feature_shapes.yml.AGENTS.md / CLAUDE.md routing block.bin/task / bin/dor-check / board, the data migration, blocked metadata
(blocked_from + block_kind), and the DoR-to-Build / DoR-to-Merge gates.Release singleton model + release_slug / dependencies on Task + the
board's "current release" header.release branch cutover: bin/release init|merge|prepare|
eject|ship on the persistent per-repo release branch — membership records at
the sweep (merge/prepare → gh pr merge + Release::Conductor.sweep!,
stage flip on QA-green via qa_green!), prepare deploys origin/release to
QA, ship fast-forwards release → main stamping merged: "main" (§1.1).bin/agent-worktree release-base default: new cuts the feature branch from
origin/release (falling back to origin/main), and finish --pr opens the PR
with --base release — feature agents no longer pass --base release by hand.bin/devops-cycle stage-name migration: the heartbeat planner (+ its snapshot
fixture + bin/devops-tests lanes) speaks the new stages
(submitted/reviewed/assembled).ship: producer-first, hub-before-satellites deploy across every
release repo (gems → re-pin consumers → hub → satellites) with the per-repo
test_cmd gate and partial-ship recovery (§1.1).Next
config/devops_test_suites.yml.DISCORD_DEVOPS_PROGRESS_WEBHOOK_URL.We emailed a one-tap sign-in link to . It expires shortly and can only be used once.
No email? Check spam, or close this and try again.