Capture execution data; aggregate, surface, and act on signals
Status: open/published
Summary
Section titled “Summary”- Every workflow captures structured execution data: what happened, what surprised, what failed, what took too long.
- Three nested loops: per-run capture (post-run assessment), cross-run aggregation (pattern surfacing), automated or human-mediated improvement.
- Where safe, the system autonomously proposes and applies improvements; otherwise it surfaces the signal so a human acts.
- Match instrumentation to the stage per P0003; don’t instrument maximally.
Statement
Section titled “Statement”Data collection and self-assessment are first-class concerns of the SDLC system, not incidental add-ons. Execution of system workflows captures structured data: what happened, what surprised, what failed, what took too long. The system aggregates that data across runs, surfaces the patterns, and proposes improvements using the SDLC system itself. Where safe, the system autonomously suggests and implements improvements. Where not, it makes the signal visible so a human can act on it.
The point of the SDLC system is to operationalize and automate as much as possible. That is intrinsically an iterative process: it requires identifying and capturing lessons in a way that can influence system and policy changes downstream.
Adding instrumentation and a framework for capture and analysis feeds into the iterative improvement process; self-reflective tools (analytics or LLM review) help surface needed fixes.
How it applies
Section titled “How it applies”Three nested loops:
Per-run capture. Every skill and workflow executes a structured post-run assessment that records:
- goal achievement: did the work hit its declared goal?
- performance metrics: which tool calls were made and how many, how many failures, execution time, token usage, task acceptance rate, etc.
- anomaly detection: surprises the workflow’s instructions didn’t anticipate.
- self-review: what about the workflow’s own shape made the work harder than it should have been?
- follow-ups: surfacing through dashboard, alerts, or SDLC tracking mechanisms (bug reports).
- full-loop automation (eventually): capturing the data, reviewing, and driving changes through the SDLC process in a fully automated fashion.
The skill ADR’s “Post-execution assessment” section
(docs/planning/decisions/D-TQHZ-skill.md) is the concrete
shape for skill-level capture.
Cross-run aggregation. A separate layer surfaces patterns across the per-run artifacts: “this step has failed five times in a row,” “this skill’s average runtime has tripled,” “three independent runs surfaced the same missing capability,” “the verifier rejects this section 60% of the time.” Patterns feed the next loop.
Automated or human-mediated improvement. Where the system can safely act (anti-pattern detected, verifier producing a deterministic fix, skill self-updating from its own self-critique), it acts. Otherwise it surfaces the pattern as a task, backlog item, or open question on the affected entity. The signal does not evaporate either way.
Examples
Section titled “Examples”- The skill ADR’s
Post-execution assessmentsection (docs/planning/decisions/D-TQHZ-skill.md): per-run capture pattern for every skill execution. - The
need_human_reviewfrontmatter flag combined withdocs/dashboard.base: a manual instance of capture-and-surface for review state. - git commit history as a long-tail capture: every change carries a rationale that an aggregator can later mine.
- Future: an eval harness reading per-run assessment artifacts; a skill-performance dashboard surfacing failure-rate trends.
Implications
Section titled “Implications”- The skill ADR (
docs/planning/decisions/D-TQHZ-skill.md) is one direct instantiation: post-execution assessment is the per-run capture layer for skills. - A future cross-run aggregation layer (undesigned, flagged across the planning artifacts as “eval harness,” “skill performance,” “continuous improvement”) is the second loop.
- The natural-progression model in P-0001-prefer-deterministic-over-llm gets evidence-based answers to “what stage is this workflow actually at?” instead of subjective judgment.
- Self-hosting (P-0006-self-hosting-development) gains teeth: sdlc collects data about its own work, so the self-improvement loop runs on real production signal.
- Surfacing system shape (P-0009-surface-system-shape) gains a data-driven companion: not just what we built but how the built system is performing.
- Standards expected to emerge: an assessment-artifact schema (shape of per-run capture data); an aggregation manifest (where cross-run data lives); a “safely automatable” boundary (when the system can act vs. when it must defer to a human).
- This principle does NOT say “instrument everything maximally.” Per P-0003-pragmatic-architecture, instrumentation is itself staged: a brand-new prose skill doesn’t need a per-run aggregation pipeline; a stable Stage-3 workflow benefits from more rigorous capture. Match instrumentation to the stage.
- The “safely automatable” boundary is the hardest piece. Too aggressive turns into noise that erodes trust; too conservative loses the compounding benefit. Natural place for a future ADR.