Agents Builders

Test scope telemetry registry

Archived
test-scope-telemetry-registry

Created

Jul 06, 02:11

Started

Jul 06, 03:26

Completed

Jul 06, 05:02

DevOps handoff

Type

Feature

Shape

backend

Worktree Slug

test-scope-telemetry-registry

Repositories

mcritchie-studio

Release Slug

Branch

feat/test-scope-telemetry-registry

Local URL

QA URL

Production URL

observability release-conductor

Acceptance Criteria

  • Registry declares every release test scope with stable keys
  • Release gates run through one wrapper emitting start and result
  • Emitted results carry duration, pass fail, parsed counts
  • Blocking and seal gate semantics remain unchanged

Expected Test Plan

  • [unit] wrapper emission, count parsing, failure paths
  • [integration] release gates emit through wrapper end-to-end

Checks Run

  • [unit] ruby -Itest test/lib/release_cli_test.rb — run_test_scope START/COMPLETED/FAILED emission, minitest/playwright/up count parsing, failure+dry-run+role-span gating
  • [integration] ruby -Itest test/lib/release_cli_test.rb — run_post_deploy + production_smoke_seal drive their commands through run_test_scope end-to-end
  • [full-suite@3a78a0d36fcfbd16ba74ca635672b8491213d5dd] bin/rails db:test:prepare test test:system green
  • [rubocop@3a78a0d36fcfbd16ba74ca635672b8491213d5dd] bin/rubocop clean

Agent Context

Observability epic Workstream A1 (foundation #407 + seal hardening #410 both on main). GOAL: every test scope bin/release runs becomes a logged, gradeable unit — start + result with {command, host, tier, duration, pass/fail, parsed counts} — visible as AgentActions in the heartbeat timeline. DESIGN (operator-locked): promote config/devops_test_suites.yml to the canonical scope registry (stable key per lane + phase/tier/host/blocks/mutates; grep existing consumers first and stay compatible). Add run_test_scope wrapper in bin/release.rb that gates call instead of raw sh(): emit start, run via sh(), measure duration, parse counts (minitest 'X runs, Y assertions, Z failures...'; playwright 'N passed'; lenient, nil ok), emit completed/failed. EMISSION: primary channel = the #407 self-report primitive (release.rb's agent_action()/step() helpers -> bin/agent-activity action, gated on open role span, best-effort non-fatal); ALSO fix the qa_smoke started-without-completed gap via record_release_event; check ReleaseEvent/Conductor.record_event! step validation before inventing new step names. GATES TO WRAP (line refs pre-#410, expect drift): pre_qa_gate (~1009, qa_test_cmd, emits NOTHING today), test_gate/avi_ship_gate (~1668/~1884, frozen-SHA bin/rails test, only outer ship_gate pair today), wait_for_boot /up QA smoke (~515), deploy_app prod /up smoke (~1930), run_post_deploy (~551), production_smoke_seal (now w/ chdir+rescue from #410 — already fully instrumented, route through wrapper without changing its semantics). CRITICAL: blocking gates still abort! exactly as today; the seal stays non-blocking; wrapper returns [out, ok]. Scope: bin/release.rb + config/devops_test_suites.yml + optional small bin/lib helper + tests. OUT OF SCOPE (A2/A3): no new AR model, no UI, no bin/full-suite-check, no CI changes. Tests follow test/lib/release_cli_test.rb NEUTRALIZED_ENV pattern.

Stage Timeline

Who handled each stage, the time it took (measured), and the model / tokens / cost reported (best-effort) — plus who's on it right now. means the agent didn't report that metric.

Sizing Avi · PO MEDIUM Dev MEDIUM Actual XL ≠ forecast
  1. Created Designed
    L Lugia
    Lugia
    Model
    Duration
    Tokens
    Cost
    Completed Jul 06, 02:11 · 2 days ago
    api
  2. Designed Building
    L Lugia
    Lugia
    Model
    claude-fable-5
    Duration
    1 minute
    Tokens
    2,173,809
    Cost
    ~$3.18
    Started Jul 06, 02:11
    Completed Jul 06, 02:12 · 2 days ago
    cli
  3. Building Submitted
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    35 minutes
    Tokens
    6,992,845
    Cost
    ~$15.29
    Started Jul 06, 02:12
    Completed Jul 06, 02:47 · 2 days ago
    cli
  4. Submitted Blocked
    A Avi
    Avi
    Model
    gpt-5.5
    Duration
    16 minutes
    Tokens
    1,403,252
    Cost
    ~$1.24
    Started Jul 06, 02:47
    Completed Jul 06, 03:03 · 2 days ago
    cli
  5. Blocked Building
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    23 minutes
    Tokens
    16,773,848
    Cost
    ~$17.19
    Started Jul 06, 03:03
    Completed Jul 06, 03:26 · 2 days ago
    cli
  6. Building Submitted
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    under a minute
    Tokens
    Cost
    Started Jul 06, 03:26
    Completed Jul 06, 03:26 · 2 days ago
    cli
  7. Submitted Blocked
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    Cost
    Started Jul 06, 03:26
    Completed Jul 06, 04:35 · 2 days ago
    cli
  8. Blocked Reviewed Review Activity
    Model
    claude-opus-4-8
    Duration
    6 minutes
    Tokens
    10,580,245
    Cost
    ~$19.69
    Started Jul 06, 04:35
    Completed Jul 06, 04:42 · 2 days ago
    cli
  9. Reviewed Assembled
    S Steffon
    Steffon
    Model
    claude-opus-4-8
    Duration
    8 minutes
    Tokens
    5,481,955
    Cost
    ~$3.17
    Started Jul 06, 04:42
    Completed Jul 06, 04:50 · 2 days ago
  10. Assembled Shipped
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    12 minutes
    Tokens
    8,386,949
    Cost
    ~$5.47
    Started Jul 06, 04:50
    Completed Jul 06, 05:02 · 2 days ago
  11. Shipped Archived
    Model
    Duration
    about 11 hours
    Tokens
    Cost
    Started Jul 06, 05:02
    Completed Jul 06, 15:43 · 2 days ago

Conversation

QA review feedback, agent handoffs, and follow-up notes for this task.

Comment alex 2 days ago

Scout report: request-changes - Request changes: gem release_check remains unregistered/unwrapped; focused release CLI suite is green under Ruby 3.3. Reporter: alex Outcome: request-changes Findings: - Blocker: bin/release.rb:747-752 still runs a gem release_check via sh(rc, '--build') while config/devops_test_suites.yml release_scopes has no gem scope, so gem releases miss start/result telemetry. - Non-blocker: PR #361 also touches bin/release.rb and test/lib/release_cli_test.rb; git merge-tree auto-merges the pair, but conductor should sequence the same-file overlap. Questions: - none Checks: - Ruby 3.3.11 focused run: ruby -Itest test/lib/release_cli_test.rb — 140 runs, 709 assertions, 0 failures. - bin/devops-tests --app mcritchie-studio prints the updated catalog; PR #414 is mergeable with GitHub CI success.

Comment carl 2 days ago

Scout report: request-changes - Request changes: release telemetry misses gem release_check and qa_smoke can mark Live on QA before a blocking post-deploy hook aborts. Reporter: carl Outcome: request-changes Findings: - Blocker: bin/release.rb:749-752 still runs gem release_check via raw sh and config/devops_test_suites.yml:205-254 has no release scope for it, so gem pre-publish tests emit no start/result telemetry. - Blocker: bin/release.rb:1494-1516 records qa_smoke:completed before QA post-deploy hooks run; Release::EVENT_STAGE_STAMPS maps that to qa_deployed, so a failing hook leaves the tracker Live on QA while members stay reviewed. - Non-blocker: PR #361 also touches bin/release.rb and test/lib/release_cli_test.rb; merge-tree is clean both orders, but Steffon should sequence same-file overlap. Questions: - none Checks: - Read AGENTS, Avi pr-review SOP, shared pr-review SOP, README, testing/deployment runbooks, and release conductor docs. - Reviewed task metadata, PR body, changed files, diff, ReleaseEvent/stage semantics, CI, qa-intake, and prior light scout report. - Focused test passed under Ruby 3.3.11: ruby -Itest test/lib/release_cli_test.rb — 140 runs, 709 assertions. - DoR passed from PR worktree; GitHub CI green; local merge-tree with PR #361 clean both orders.

QA Feedback avi 2 days ago

pr-review blocked review: carl: request-changes - Request changes: release telemetry misses gem release_check and qa_smoke can mark Live on QA before a blocking post-deploy hook aborts.; alex: request-changes - Request changes: gem release_check remains unregistered/unwrapped; focused release CLI suite is green under Ruby 3.3.

Handoff steffon 2 days ago

Rework done on feat/test-scope-telemetry-registry (PR #414): (1) gem_release_check now registered in release_scopes + wrapped through run_test_scope in publish_gem, abort-before-publish preserved; (2) qa_smoke completed deferred into the else branch, recorded only after the blocking QA post-deploy hook returns green (failed on boot failure) — no premature green. Regression tests added for both; full suite + rubocop certified green, CI green.

QA Feedback avi 2 days ago

pr-review request-changes (both reviewers). (1) Gem release_check is unregistered/unwrapped in the release telemetry — register and wrap it. (2) qa_smoke can mark Live on QA before the blocking post-deploy hook aborts — gate the Live stamp behind the post-deploy hook so a hook abort blocks Live. Focused release-CLI suite is green under Ruby 3.3; extend coverage to these paths, then resubmit and post a note that resolves this feedback.

Handoff avi 2 days ago

Approved to reviewed (deploy owner, operator-assigned lane). Re-block was a DUPLICATE of already-resolved feedback: verified on PR head 0d125c74 both findings fixed — (1) gem_release_check registered (registry:234) + wrapped via run_test_scope (release.rb:751), abort-before-publish intact; (2) qa_smoke completed deferred to AFTER the blocking run_post_deploy(:qa) (release.rb:1514-1520). GitHub CI green (4 checks); full suite 2941/0. Taking to production via full-cycle.

Sealed-bid sizing

Edit →

Alex (PM)

Avi (PO)

MEDIUM

Dev

MEDIUM

Actual

XL