Agents Builders

Test telemetry local and CI

Archived
test-telemetry-local-and-ci

Created

Jul 06, 07:29

Started

Jul 06, 07:36

Completed

Jul 06, 08:16

DevOps handoff

Type

Feature

Shape

backend

Worktree Slug

test-telemetry-local-and-ci

Repositories

mcritchie-studio

Release Slug

Branch

feat/test-telemetry-local-and-ci

Local URL

QA URL

Production URL

observability ci

Acceptance Criteria

  • full-suite-check lanes emit gradeable test_scope actions
  • Emitted lanes carry scope key, pass fail, duration
  • CI telemetry implemented or scoped as documented follow-up

Expected Test Plan

  • [unit] full-suite-check emits a tagged action per lane
  • [integration] cert run self-reports test_scope actions end-to-end

Checks Run

  • [unit] full_suite_check_test: each lane emits tagged test_scope (kind/event_slug/result_slug/duration_ms); no-op with no session; non-fatal on broken emit seam — 19 green
  • [integration] full_suite_check_test: green + red-lane cert self-reports all 3 lanes end-to-end through the bin/agent-activity action seam, in run order
  • [unit] heartbeat_helper_test: full_suite_scopes keys enrich /alex/pipeline phase/tier/host chips via merged registry — 42 green
  • [full-suite@9a92942a313084b24e6dacf5c2509797021ec92a] bin/rails db:test:prepare test test:system green
  • [rubocop@9a92942a313084b24e6dacf5c2509797021ec92a] bin/rubocop clean

Agent Context

Observability epic A3 — the final slice. Extend the A1/A2 test-scope telemetry beyond the release conductor to LOCAL certs and CI. A1 shipped run_test_scope in bin/release.rb emitting kind=test_scope AgentActions via bin/agent-activity action (self-report to the open span); A2 (just shipped) tags event_slug/result_slug/duration_ms and surfaces a gradeable Test-runs band in /alex/pipeline. Now: (1) LOCAL full-suite-check — bin/full-suite-check runs test-db-reset / bin/rails test / rubocop lanes (see feature_shapes + the file). Wrap each lane's run so it self-reports a kind=test_scope AgentAction (event_slug=<lane key>, result_slug=pass|fail, duration_ms) via the SAME bin/agent-activity action verb, gated on an open activity/span, best-effort/non-fatal/inert when no span (mirror run_test_scope's guards exactly — telemetry must NEVER break a cert). Register the full-suite lanes as scopes in config/devops_test_suites.yml with stable keys (phase: build, tier: full/unit/build, host: local). This makes local cert runs gradeable like release runs — the clean, cohesive win. (2) CI — assess feasibility: GitHub Actions (.github/workflows/ci.yml) runs off-box and can't touch the local activity bus; emitting would mean POSTing a test_scope event to the board API at job start/end. JUDGMENT CALL: if this is a clean board-API POST that needs NO new CI secrets and has a sensible attribution (e.g. tied to the PR/branch), implement it; if it needs new CI secrets, a new API attribution path, or non-trivial infra decisions, DO NOT half-build it overnight — scope it as a documented follow-up task/comment instead and ship the solid full-suite-check half. Prefer a solid partial over a risky full. NO changes to bin/release.rb's run_test_scope (A1/A2 own it) beyond reusing shared helpers if cleanly extractable. Shape backend; unit + integration tiers.

Stage Timeline

Who handled each stage, the time it took (measured), and the model / tokens / cost reported (best-effort) — plus who's on it right now. means the agent didn't report that metric.

Sizing Avi · PO MEDIUM Dev MEDIUM Actual XL ≠ forecast
  1. Created Designed
    L Lugia
    Lugia
    Model
    Duration
    Tokens
    Cost
    Completed Jul 06, 07:29 · 2 days ago
    api
  2. Designed Building
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    7 minutes
    Tokens
    5,935,484
    Cost
    ~$3.93
    Started Jul 06, 07:29
    Completed Jul 06, 07:36 · 2 days ago
    cli
  3. Building Submitted
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    15 minutes
    Tokens
    Cost
    Started Jul 06, 07:36
    Completed Jul 06, 07:51 · 2 days ago
    cli
  4. Submitted Reviewed Review Activity
    C Carl
    Carl primary
    J Jasper
    Jasper light
    Model
    claude-opus-4-8
    Duration
    7 minutes
    Tokens
    15,815,565
    Cost
    ~$9.82
    Started Jul 06, 07:51
    Completed Jul 06, 07:58 · 2 days ago
    cli
  5. Reviewed Assembled
    S Steffon
    Steffon
    Model
    claude-opus-4-8
    Duration
    9 minutes
    Tokens
    6,836,084
    Cost
    ~$3.74
    Started Jul 06, 07:58
    Completed Jul 06, 08:07 · 2 days ago
  6. Assembled Shipped
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    10 minutes
    Tokens
    3,745,181
    Cost
    ~$1.95
    Started Jul 06, 08:07
    Completed Jul 06, 08:16 · 2 days ago
  7. Shipped Archived
    Model
    Duration
    about 14 hours
    Tokens
    Cost
    Started Jul 06, 08:16
    Completed Jul 06, 22:06 · 1 day ago

Conversation

QA review feedback, agent handoffs, and follow-up notes for this task.

Handoff avi 2 days ago

Avi review approved: A3 local-cert telemetry faithfully mirrors bin/release.rb run_test_scope — triple-gated best-effort emit (session gate + action-verb marker no-op + swallowed errors), lane result captured before/independent of the discarded emit so it can never corrupt a cert. bin/release.rb untouched, no migration, CI green; CI-telemetry correctly deferred to test-telemetry-from-ci.

Sealed-bid sizing

Edit →

Alex (PM)

Avi (PO)

MEDIUM

Dev

MEDIUM

Actual

XL