Capture execution data; aggregate, surface, and act on signals
State: open/published
Summary
Section titled “Summary”- Every workflow captures structured execution data: what happened, what surprised, what failed, what took too long.
- Three nested loops: per-run capture, cross-run aggregation, and automated or human-mediated improvement.
- Where safe, the system proposes and applies improvements itself; otherwise it surfaces the signal so a human acts.
- Instrumentation matches the workflow’s stage per P-0003-pragmatic-architecture; do not instrument maximally.
Statement
Section titled “Statement”Data collection and self-assessment are first-class concerns of SDLC, not add-ons. Executing a workflow captures structured data: what happened, what surprised, what failed, what took too long. The system aggregates that data across runs, surfaces the patterns, and proposes improvements using SDLC itself. Where safe, it applies them; where not, it makes the signal visible so a human can act.
SDLC exists to operationalize and automate as much as possible, and that is iterative: lessons have to be captured in a form that can drive later changes to the system and its policies.
Without capture, a lesson dies in the session that learned it. The same step fails for the fifth time and nobody knows it is the fifth; a workflow’s cost triples unnoticed; changes to skills and policy get argued from recollection instead of evidence. Capture turns those into facts a later run, or a later reviewer, can act on.
How it applies
Section titled “How it applies”Three nested loops:
Per-run capture. A workflow past the thin prose-skill stage (see Notes) executes a structured post-run assessment that records:
- goal achievement: did the work hit its declared goal?
- performance: tool calls made, failures, execution time, token usage, task acceptance rate
- anomalies: surprises the workflow’s instructions did not anticipate
- self-review: what about the workflow’s own shape made the work harder than it should have been?
- follow-ups: surfaced through the dashboard, alerts, or SDLC tracking (bug reports)
The skill Decision’s “Post-execution assessment” section (D-TQHZ-skill, still proposed) is the drafted shape for skill-level capture.
Cross-run aggregation. A separate layer surfaces patterns across per-run artifacts: “this step has failed five times in a row”, “this skill’s average runtime has tripled”, “three independent runs surfaced the same missing capability”, “the verifier rejects this section 60% of the time”. Patterns feed the next loop.
Automated or human-mediated improvement. Where the system can act safely (an anti-pattern detected, a verifier producing a deterministic fix, a skill updating itself from its own self-critique), it acts. Otherwise it surfaces the pattern as a task, backlog item, or open question on the affected entity. The signal does not evaporate either way.
Examples
Section titled “Examples”- The skill Decision’s
Post-execution assessmentsection (D-TQHZ-skill): the per-run capture pattern for skill execution. - The
need_human_reviewfrontmatter flag withdocs/dashboard.base: a manual instance of capture-and-surface for review state. - git commit history as long-tail capture: every change carries a rationale an aggregator can mine.
- Future: an eval harness reading per-run assessment artifacts; a skill-performance dashboard surfacing failure-rate trends.
Implications
Section titled “Implications”- P-0001-prefer-deterministic-over-llm’s progression gets evidence-based answers to “what stage is this workflow at?” instead of subjective judgment.
- P-0006-self-hosting-development gains teeth: SDLC collects data about its own work, so the self-improvement loop runs on real signal.
- P-0009-surface-system-shape gains a data-driven companion: not just what was built but how it performs.
- Standards expected to emerge: an assessment-artifact schema, an aggregation manifest (where cross-run data lives), and a “safely automatable” boundary (when the system acts versus defers to a human).
- This principle does not say “instrument everything maximally”. Per P-0003-pragmatic-architecture, instrumentation is staged: a new prose skill does not need a per-run aggregation pipeline; a stable Stage 3 workflow benefits from more rigorous capture.
- The “safely automatable” boundary is the hardest piece. Too aggressive becomes noise that erodes trust; too conservative loses the compounding benefit. A natural place for a future Decision.