Created
Jul 06, 02:11
Started
Jul 06, 03:26
Completed
Jul 06, 05:02
DevOps handoff
Type
Feature
Shape
backend
Worktree Slug
test-scope-telemetry-registry
Repositories
mcritchie-studio
Release Slug
—
Branch
feat/test-scope-telemetry-registry
Local URL
—
QA URL
—
Production URL
—
Acceptance Criteria
Expected Test Plan
Checks Run
Agent Context
Observability epic Workstream A1 (foundation #407 + seal hardening #410 both on main). GOAL: every test scope bin/release runs becomes a logged, gradeable unit — start + result with {command, host, tier, duration, pass/fail, parsed counts} — visible as AgentActions in the heartbeat timeline. DESIGN (operator-locked): promote config/devops_test_suites.yml to the canonical scope registry (stable key per lane + phase/tier/host/blocks/mutates; grep existing consumers first and stay compatible). Add run_test_scope wrapper in bin/release.rb that gates call instead of raw sh(): emit start, run via sh(), measure duration, parse counts (minitest 'X runs, Y assertions, Z failures...'; playwright 'N passed'; lenient, nil ok), emit completed/failed. EMISSION: primary channel = the #407 self-report primitive (release.rb's agent_action()/step() helpers -> bin/agent-activity action, gated on open role span, best-effort non-fatal); ALSO fix the qa_smoke started-without-completed gap via record_release_event; check ReleaseEvent/Conductor.record_event! step validation before inventing new step names. GATES TO WRAP (line refs pre-#410, expect drift): pre_qa_gate (~1009, qa_test_cmd, emits NOTHING today), test_gate/avi_ship_gate (~1668/~1884, frozen-SHA bin/rails test, only outer ship_gate pair today), wait_for_boot /up QA smoke (~515), deploy_app prod /up smoke (~1930), run_post_deploy (~551), production_smoke_seal (now w/ chdir+rescue from #410 — already fully instrumented, route through wrapper without changing its semantics). CRITICAL: blocking gates still abort! exactly as today; the seal stays non-blocking; wrapper returns [out, ok]. Scope: bin/release.rb + config/devops_test_suites.yml + optional small bin/lib helper + tests. OUT OF SCOPE (A2/A3): no new AR model, no UI, no bin/full-suite-check, no CI changes. Tests follow test/lib/release_cli_test.rb NEUTRALIZED_ENV pattern.
Stage Timeline
Who handled each stage, the time it took (measured), and the model / tokens / cost reported (best-effort) — plus who's on it right now. — means the agent didn't report that metric.
Conversation
QA review feedback, agent handoffs, and follow-up notes for this task.
Scout report: request-changes - Request changes: gem release_check remains unregistered/unwrapped; focused release CLI suite is green under Ruby 3.3. Reporter: alex Outcome: request-changes Findings: - Blocker: bin/release.rb:747-752 still runs a gem release_check via sh(rc, '--build') while config/devops_test_suites.yml release_scopes has no gem scope, so gem releases miss start/result telemetry. - Non-blocker: PR #361 also touches bin/release.rb and test/lib/release_cli_test.rb; git merge-tree auto-merges the pair, but conductor should sequence the same-file overlap. Questions: - none Checks: - Ruby 3.3.11 focused run: ruby -Itest test/lib/release_cli_test.rb — 140 runs, 709 assertions, 0 failures. - bin/devops-tests --app mcritchie-studio prints the updated catalog; PR #414 is mergeable with GitHub CI success.
Scout report: request-changes - Request changes: release telemetry misses gem release_check and qa_smoke can mark Live on QA before a blocking post-deploy hook aborts. Reporter: carl Outcome: request-changes Findings: - Blocker: bin/release.rb:749-752 still runs gem release_check via raw sh and config/devops_test_suites.yml:205-254 has no release scope for it, so gem pre-publish tests emit no start/result telemetry. - Blocker: bin/release.rb:1494-1516 records qa_smoke:completed before QA post-deploy hooks run; Release::EVENT_STAGE_STAMPS maps that to qa_deployed, so a failing hook leaves the tracker Live on QA while members stay reviewed. - Non-blocker: PR #361 also touches bin/release.rb and test/lib/release_cli_test.rb; merge-tree is clean both orders, but Steffon should sequence same-file overlap. Questions: - none Checks: - Read AGENTS, Avi pr-review SOP, shared pr-review SOP, README, testing/deployment runbooks, and release conductor docs. - Reviewed task metadata, PR body, changed files, diff, ReleaseEvent/stage semantics, CI, qa-intake, and prior light scout report. - Focused test passed under Ruby 3.3.11: ruby -Itest test/lib/release_cli_test.rb — 140 runs, 709 assertions. - DoR passed from PR worktree; GitHub CI green; local merge-tree with PR #361 clean both orders.
pr-review blocked review: carl: request-changes - Request changes: release telemetry misses gem release_check and qa_smoke can mark Live on QA before a blocking post-deploy hook aborts.; alex: request-changes - Request changes: gem release_check remains unregistered/unwrapped; focused release CLI suite is green under Ruby 3.3.
Rework done on feat/test-scope-telemetry-registry (PR #414): (1) gem_release_check now registered in release_scopes + wrapped through run_test_scope in publish_gem, abort-before-publish preserved; (2) qa_smoke completed deferred into the else branch, recorded only after the blocking QA post-deploy hook returns green (failed on boot failure) — no premature green. Regression tests added for both; full suite + rubocop certified green, CI green.
pr-review request-changes (both reviewers). (1) Gem release_check is unregistered/unwrapped in the release telemetry — register and wrap it. (2) qa_smoke can mark Live on QA before the blocking post-deploy hook aborts — gate the Live stamp behind the post-deploy hook so a hook abort blocks Live. Focused release-CLI suite is green under Ruby 3.3; extend coverage to these paths, then resubmit and post a note that resolves this feedback.
Approved to reviewed (deploy owner, operator-assigned lane). Re-block was a DUPLICATE of already-resolved feedback: verified on PR head 0d125c74 both findings fixed — (1) gem_release_check registered (registry:234) + wrapped via run_test_scope (release.rb:751), abort-before-publish intact; (2) qa_smoke completed deferred to AFTER the blocking run_post_deploy(:qa) (release.rb:1514-1520). GitHub CI green (4 checks); full suite 2941/0. Taking to production via full-cycle.
Sealed-bid sizing
Edit →Alex (PM)
—
Avi (PO)
MEDIUM
Dev
MEDIUM
Actual
XL
We emailed a one-tap sign-in link to . It expires shortly and can only be used once.
No email? Check spam, or close this and try again.