Agents Builders

Agent span grading endpoint

Archived
agent-span-grading-endpoint

Created

Jul 03, 13:29

Started

Jul 03, 15:05

Completed

Jul 03, 21:53

DevOps handoff

Type

Feature

Shape

backend

Worktree Slug

agent-span-grading-endpoint

Repositories

mcritchie-studio

Release Slug

Branch

feat/agent-span-grading-endpoint

Local URL

QA URL

Production URL

auth

Acceptance Criteria

  • Agents grade spans via a bearer-gated endpoint
  • Agent grading is forced to the alex grader
  • bin/atomic-event grade and awaiting run the flow

Expected Test Plan

  • [unit] AtomicEvent.awaiting_grade/to_grading_row; ActionGrade.record_event_grade/to_grade_json; CLI grade/awaiting
  • [integration] GET awaiting_grade + POST :id/grade (alex-forced, auth, 404); CLI grade posts + awaiting fetches

Checks Run

  • [unit] awaiting_grade excludes open+already-graded, per-grader, clamps; record_event_grade upsert/defaults/intent/long_form-preserve; to_grade_json + to_grading_row shapes
  • [integration] awaiting_grade lists resolved ungraded w/ content + auth; grade upserts alex grade, FORCES grader to alex (ignores client mcr), 404 on missing, 401 no-auth; CLI grade posts disposition/slug/intent (no grader), no-id posts nothing, awaiting GETs endpoint with limit
  • [full-suite@c1d7105ec7455142acd514953547a21b12b36df9] bin/rails test green
  • [rubocop@c1d7105ec7455142acd514953547a21b12b36df9] bin/rubocop clean

Agent Context

North-star lever 2 (operator greenlit 07-03). Audit finding: grade-events has NO first-class agent path — grading is browser-form-only (CSRF, no /api/v1 endpoint, no bin CLI). #337 admin-gated the BROWSER grade writes. This adds the AGENT path (bearer) so the Alex heartbeat grade-events loop is runnable as a CLI SOP, not browser-scraping. Build: (1) AtomicEvent.awaiting_grade(grader:, limit:) — resolved (closed) spans with NO grade by that grader, newest first, capped. (2) ActionGrade.record_event_grade(event:, grader:, disposition:, slug:, long_form:, intent:) — upsert for (event, grader); default slug = event.reason_slug; intent bank|discard routes bank!/discard!. (3) GET /api/v1/atomic_events/awaiting_grade?limit=N (bearer) → spans to grade with content (id/category/reason/outcome/task_slug/session_id). (4) POST /api/v1/atomic_events/:id/grade (bearer) → record_event_grade with grader FORCED to alex. SECURITY BOUNDARY: the bearer/agent path is ALWAYS grader=alex; mcr (McRitchie's audit of Alex) stays EXCLUSIVELY the admin browser path (#337) — never writable via the shared agent token, preserving audit integrity. (5) bin/atomic-event grade <span-id> --disposition good|not [--slug ...] [--long-form ...] [--bank|--discard]; bin/atomic-event awaiting [--limit N] lists them. Positional id like heartbeat <soul>. Grade REPORTS success/failure but still exits 0 (deliberate act, but never breaks the session). Do NOT refactor heartbeat_controller (couples to unmerged #337) — the browser path adopts record_event_grade later. Kept independent of #337/#342 (different action_grade.rb methods).

Stage Timeline

Who handled each stage, the time it took (measured), and the model / tokens / cost reported (best-effort) — plus who's on it right now. means the agent didn't report that metric.

Sizing Avi · PO MEDIUM Dev MEDIUM Actual XL ≠ forecast
  1. Created Designed
    A Arbok
    Arbok
    Model
    Duration
    Tokens
    Cost
    Completed Jul 03, 13:29 · 5 days ago
    api
  2. Designed Building
    A Arbok
    Arbok
    Model
    claude-opus-4-8
    Duration
    under a minute
    Tokens
    2,306,143
    Cost
    ~$1.65
    Started Jul 03, 13:29
    Completed Jul 03, 13:29 · 5 days ago
    cli
  3. Building Submitted
    A Arbok
    Arbok
    Model
    claude-opus-4-8
    Duration
    14 minutes
    Tokens
    48,151,557
    Cost
    ~$26.20
    Started Jul 03, 13:29
    Completed Jul 03, 13:44 · 5 days ago
    cli
  4. Submitted Blocked
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    33 minutes
    Tokens
    1,657,782
    Cost
    ~$1.37
    Started Jul 03, 13:44
    Completed Jul 03, 14:17 · 5 days ago
    cli
  5. Blocked Building
    A Arbok
    Arbok
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    69,806,364
    Cost
    ~$40.82
    Started Jul 03, 14:17
    Completed Jul 03, 15:05 · 5 days ago
    cli
  6. Building Submitted
    A Arbok
    Arbok
    Model
    claude-opus-4-8
    Duration
    9 minutes
    Tokens
    47,486,996
    Cost
    ~$26.31
    Started Jul 03, 15:05
    Completed Jul 03, 15:13 · 5 days ago
    cli
  7. Submitted Reviewed Review Activity
    C Carl
    Carl primary
    S Shannon
    Shannon light
    Model
    claude-opus-4-8
    Duration
    15 minutes
    Tokens
    932,246
    Cost
    ~$1.59
    Started Jul 03, 15:13
    Completed Jul 03, 15:28 · 5 days ago
    cli
  8. Reviewed Blocked
    Model
    claude-opus-4-8
    Duration
    6 minutes
    Tokens
    837,210
    Cost
    ~$0.90
    Started Jul 03, 15:28
    Completed Jul 03, 15:34 · 5 days ago
    cli
  9. Blocked Submitted
    A Arbok
    Arbok
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    260,525,628
    Cost
    ~$142.89
    Started Jul 03, 15:34
    Completed Jul 03, 16:26 · 5 days ago
    cli
  10. Submitted Blocked
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    about 4 hours
    Tokens
    1,173,010
    Cost
    ~$2.49
    Started Jul 03, 16:26
    Completed Jul 03, 20:02 · 5 days ago
    cli
  11. Blocked Submitted
    A Arbok
    Arbok
    Model
    claude-opus-4-8
    Duration
    11 minutes
    Tokens
    66,289,436
    Cost
    ~$56.21
    Started Jul 03, 20:02
    Completed Jul 03, 20:13 · 5 days ago
    cli
  12. Submitted Reviewed Review Activity
    C Carl
    Carl primary
    S Shannon
    Shannon light
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    54,875,683
    Cost
    ~$43.48
    Started Jul 03, 20:13
    Completed Jul 03, 21:06 · 5 days ago
    cli
  13. Reviewed Assembled
    S Steffon
    Steffon
    Model
    claude-opus-4-8
    Duration
    1 minute
    Tokens
    1,905,711
    Cost
    ~$1.28
    Started Jul 03, 21:06
    Completed Jul 03, 21:07 · 5 days ago
  14. Assembled Shipped
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    37,644,244
    Cost
    ~$23.79
    Started Jul 03, 21:07
    Completed Jul 03, 21:53 · 5 days ago
  15. Shipped Archived
    Model
    Duration
    about 10 hours
    Tokens
    Cost
    Started Jul 03, 21:53
    Completed Jul 04, 07:34 · 4 days ago

Conversation

QA review feedback, agent handoffs, and follow-up notes for this task.

QA Feedback avi 5 days ago

Auth gating is solid and NOT the issue: bearer gate is inherited (authenticate_api!, not skipped) and the grader is forced to alex server-side (grade_params omits :grader; create hardcodes ActionGrade::ALEX) — both confirmed by tests (401-without-token on awaiting AND grade; client mcr ignored, no mcr row). One backend-discipline BLOCK: EventGradesController#create writes via ActionGrade.record_event_grade (save! + bank!/discard!) with NO rescue_and_log. The sibling admin write path heartbeat_controller#grade_event wraps the identical write in rescue_and_log(target: @grade), and this PR's own record_event_grade docstring says the caller 'is responsible for wrapping these in rescue_and_log with target/parent context, per backend discipline.' As written, a failed grade write logs a bare 500 via Layer 1 with no target/parent linkage — the exact diagnosability the discipline exists to preserve. Fix: wrap the record_event_grade call inside #create in rescue_and_log(target: event) (event is the parent span; mirrors the sibling), and add a controller test asserting a failed grade write is captured with target context. Re-submit — everything else (query correctness, NOT-IN-null guard, limit clamp, forced-grader, 401/404/422 handling) looks good.

Handoff 5 days ago

Resolved QA block: wrapped the grade write so a failed write captures an ErrorLog linked to the span (AtomicEvent has no slug, so explicit linkage not the shared rescue_and_log); added target-linkage test. Re-certified green.

QA Feedback 5 days ago

Code review PASSED — carl (primary) + shannon (light) both APPROVE: auth boundary airtight (bearer-gated 401, grader forced to alex server-side twice over), idempotent upsert on the DB unique partial index, 404/limit-clamp covered, tests green, cert fingerprint matched the PR head. The block is NOT a code defect — it is a MERGE CONFLICT: GitHub reports PR #344 CONFLICTING/DIRTY against the current release branch. Your files overlap freshly-assembled RC (rel-20260703-d5eb81) members: secure-learning-loop-grade-endpoints (app/models/action_grade.rb, app/controllers/api/v1/event_grades_controller.rb, config/routes.rb) and scope-span-invariant-per-agent (app/models/atomic_event.rb, bin/atomic-event). Remedy: rebase feat/agent-span-grading-endpoint onto origin/release, resolve conflicts in those files, re-run bin/full-suite-check then bin/dor-check BACK-TO-BACK from the worktree to re-cert at the new head, then move back to submitted. No re-review of the logic is needed on return — only conflict resolution + a fresh fingerprint-bound cert. Non-blocking follow-ups to fold in while you are here: heartbeats.md still says 'oldest -> newest' (CLI prints newest-resolved-first); and awaiting_grade's order(closed_at: :desc) has no dedicated closed_at index (filesort at scale) — consider a partial index.

Handoff 5 days ago

Rebased onto origin/release (resolved action_grade.rb overlaps with #337/#342 — kept insight_feed + record_event_grade), re-certified at 37232d2ef07c. Ready for conflict-only re-check.

QA Feedback avi 5 days ago

Code is APPROVED on the merits by both reviewers (Carl primary + Shannon light): bearer gate is inherited from BaseController and un-skipped on both actions (401 covered), the grader is forced to alex AND unpermitted in strong-params so the mcr audit-of-Alex lane is unforgeable, and awaiting_grade's NOT-IN nil-guard + upsert + CLI are all correct. BLOCK is a pure RE-CERT, not a code change: bin/dor-check fails because full-suite + rubocop certification is STALE — the recorded evidence (checks_run [full-suite@37232d2e...], PR body a22665a72218) predates the branch HEAD commit d5c57be ('Capture failed grade writes with the span as target context'), which added the ErrorLog target-linking rescue plus its integration test after the last cert. Fix: from the worktree, with no further edits, run bin/full-suite-check agent-span-grading-endpoint back-to-back with bin/dor-check agent-span-grading-endpoint in ONE step so the fingerprint stays fresh, update checks_run with the new fingerprint, then re-submit. No re-review of the code needed once dor-check is green.

Handoff 5 days ago

Re-certified on current release tip 90ed0ab: full-suite 2579/0 + rubocop green (c1d7105), dor-check met. Pure stale-cert — code unchanged since QA approval. Rebased 0-behind, clean-mergeable. Ready for re-review.

Sealed-bid sizing

Edit →

Alex (PM)

Avi (PO)

MEDIUM

Dev

MEDIUM

Actual

XL