McRitchie Studio
Say Hi 👋
Agents Builders
← All Tasks
Sizing

Test telemetry from CI

Archived
test-telemetry-from-ci

Created

Jul 06, 07:45

Started

Jul 06, 13:39

Completed

Jul 06, 16:20

DevOps handoff

Type

Feature

Shape

backend

Worktree Slug

test-telemetry-from-ci

Repositories

mcritchie-studio

Release Slug

—

Branch

feat/test-telemetry-from-ci

Pull Request

https://github.com/amcritchie/mcritchie-studio/pull/421

Local URL

http://localhost:3018

QA URL

—

Production URL

—

observability ci

Acceptance Criteria

  • Each CI job captured as a gradeable test_scope row
  • Capture reads gh pr checks; no CI secret needed
  • Emit reuses agent-activity action: best-effort, non-fatal, inert
  • Re-reading CI does not duplicate rows (idempotent)

Expected Test Plan

  • [unit] capture parses gh-checks JSON, emits one test_scope per pass/fail job (event_slug/result_slug/duration_ms), skips pending
  • [unit] no-op without a session/open span; idempotent (same key twice -> one row); non-fatal when the emit seam is broken
  • [integration] end-to-end capture over a stubbed gh fixture self-reports the CI lanes to the open activity

Checks Run

  • [unit] ci-scope-capture parses gh-checks JSON, emits one test_scope per pass/fail job (event_slug/result_slug/duration_ms) — test/lib/ci_scope_capture_test.rb
  • [unit] skips pending/skipping/cancel; normalizes job names to ci_ keys; no-op without session; non-fatal on broken emit seam + malformed JSON
  • [unit] idempotency key stable across same-sha re-read, distinct per head sha (ci:<pr>:<sha>:<job>)
  • [unit] AgentAction.capture dedupes on idempotency_key (same key -> one row); distinct/blank key -> distinct rows — test/models/atomic_action_test.rb
  • [unit] action verb forwards --idempotency-key into the POST body, drops it when blank — test/lib/atomic_event_cli_test.rb
  • [integration] end-to-end ingest of a full PR fixture (pass/fail/pending/cancel) self-reports the ci_ lanes — test/lib/ci_scope_capture_test.rb
  • [integration] endpoint permits idempotency_key; a re-POST with the same key persists exactly one row — test/controllers/api/v1/atomic_actions_controller_test.rb
  • [full-suite@7494c34961888661a0ba83e7aab15a21a9a4854a] bin/rails db:test:prepare test test:system green
  • [rubocop@7494c34961888661a0ba83e7aab15a21a9a4854a] bin/rubocop clean

Agent Context

APPROACH CHANGED to Option A (local ingestion) per operator 2026-07-06 — NOT the in-CI POST originally filed. Goal unchanged: CI runs become gradeable kind=test_scope rows in /alex/pipeline alongside local certs (A3) and release runs (A1/A2). DESIGN: a session-side capture reads gh pr checks for a PR and self-reports one test_scope AgentAction per completed CI job, reusing the EXACT A3 emit path (bin/agent-activity action → kind=test_scope, event_slug, result_slug, duration_ms), attributed to the open span. NO CI secret, NO ci.yml change, NO server/API/model change, NO fork-PR issue. Read: gh pr checks <pr> --json name,bucket,startedAt,completedAt (bucket=pass/fail/pending/skipping/cancel). For each job whose bucket is pass or fail: event_slug=ci_<normalized job name> (ci_test, ci_lint, ci_scan_ruby, ci_scan_js), result_slug=pass|fail, duration_ms=(completedAt-startedAt) in ms; skip pending/skipping. Register a ci_scopes: section in config/devops_test_suites.yml (phase: ci, host: ci, stable keys) and broaden the heartbeat_helper scope-meta merge to include it so pipeline rows get phase/tier/host facets. IDEMPOTENCY (important): carry a stable idempotency_key like ci:<pr>:<headSha>:<job> on the emit so re-reading CI (dor-check/preflight run twice) does not create duplicate rows — check whether bin/agent-activity action / AgentAction.capture support idempotency_key (release events use it); if not, add a lightweight guard. HOME: pick the cleanest integration — a standalone bin helper (e.g. bin/ci-scope-capture <task-or-pr>) is fine; do NOT bolt side-effects onto dor-check (keep it a pure read-only gate). Mirror A3's guards EXACTLY (session gate, span self-check no-op, output discarded, errors swallowed — telemetry NEVER breaks anything). See bin/full-suite-check emit_test_scope + config full_suite_scopes for the pattern.

Stage Timeline

Who handled each stage, the time it took (measured), and the model / tokens / cost reported (best-effort) — plus who's on it right now. — means the agent didn't report that metric.

Sizing Avi · PO SMALL Dev SMALL Actual XL ≠ forecast
  1. Created → Designed
    L Lugia
    Lugia
    Model
    —
    Duration
    —
    Tokens
    —
    Cost
    —
    Completed Jul 06, 07:45 · 2 days ago
    api
  2. Designed → Building
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    about 6 hours
    Tokens
    42,356,534
    Cost
    ~$36.30
    Started Jul 06, 07:45
    Completed Jul 06, 13:39 · 2 days ago
    cli
  3. Building → Submitted
    L Lugia
    Lugia
    Model
    claude-opus-4-8
    Duration
    26 minutes
    Tokens
    —
    Cost
    —
    Started Jul 06, 13:39
    Completed Jul 06, 14:05 · 2 days ago
    cli
  4. Submitted → Reviewed Review Activity
    C Carl
    Carl primary
    J Jasper
    Jasper light
    Model
    claude-opus-4-8
    Duration
    32 minutes
    Tokens
    —
    Cost
    —
    Started Jul 06, 14:05
    Completed Jul 06, 14:37 · 2 days ago
    cli
  5. Reviewed → Assembled
    S Steffon
    Steffon
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    5,473,319
    Cost
    ~$4.87
    Started Jul 06, 14:37
    Completed Jul 06, 15:58 · 2 days ago
  6. Assembled → Shipped
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    22 minutes
    Tokens
    —
    Cost
    —
    Started Jul 06, 15:58
    Completed Jul 06, 16:20 · 2 days ago
  7. Shipped → Archived
    —
    Model
    —
    Duration
    about 6 hours
    Tokens
    —
    Cost
    —
    Started Jul 06, 16:20
    Completed Jul 06, 22:06 · 1 day ago

Conversation

QA review feedback, agent handoffs, and follow-up notes for this task.

Handoff avi 2 days ago

Avi review approved; ready for Steffon's qa-release sweep. Two approvals (Avi primary + Carl backend light), no blockers: all 4 acceptance criteria met, idempotency airtight (partial unique index + find-first + RecordNotUnique rescue ordered before StandardError; idempotency_key is the only unique index on agent_actions so the rescue can't mis-fire), emit is best-effort/non-fatal/session-gated, and the no-ci.yml/no-CI-secret scope claim holds. DoR fingerprint-fresh at PR tree (7494c34 == d342414^{tree}); CI green (4). Non-blocking advisory for deploy: the idempotency unique index builds non-concurrently (no disable_ddl_transaction! + algorithm: :concurrently), a brief write lock on agent_actions during migration — acceptable since telemetry writes are best-effort, but worth watching at ship.

Sealed-bid sizing

Edit →

Alex (PM)

—

Avi (PO)

SMALL

Dev

SMALL

Actual

XL