Typed LLM invocation instead of shell pipes and sentinel scraping

State: open/active · Genre: analysis

Use case: typed-llm-agent-step RFC mapping: UC1 Source: cross-repo survey of sksizer/* (2026-07-05), companion to the workflow-constellation RFC (PR #682).

The need

Across nearly every repo, invoking an LLM agent means piping a concatenated prompt into the claude CLI and recovering results by regex over raw stdout — sentinel strings, PR-URL greps, or slug-namespaced marker grammars with documented collision incidents. The recurring need is a first-class typed agent step: prompt in, typed object out, with an invoker abstraction, timeout/cancellation, session semantics, and token/cost accounting.

Evidence

DarkFactory

618-line invoke_claude: stream-json parsing, markdown-forgiving sentinel regexes (PRD_EXECUTE_OK/FAILED), watchdog thread with timeout-salvage, 4-level timeout resolution chain; multiple hardening PRDs (542.x, 560.x, 603/604) are pure maintenance on this layer.

Where: python/darkfactory/utils/claude_code/_invoke.py · python/darkfactory/timeouts.py · .darkfactory/data/prds/PRD-604-jsonl-transcript-validation.md

Pain: Structured output faked with anchorless regex over accumulated agent text; timeout/cost/cancellation all bespoke; ongoing PRD stream exists just to keep it working.

dev

Project-wide SCREAMING-KEBAB stdout marker grammar (TASK-WORK-DONE pr=#42, ENSURE-READY-OK:) as the typed-result channel between agents, with a recorded 2026-05-28 marker-collision incident that cost five dispatched runs; callers parse with regex allowlists and truncation guards.

Where: plugin/skills/CLAUDE.md · plugin/skills/pr-check/SKILL.md · plugin/skills/orchestrate/SKILL.md

Pain: Every skill re-invents a one-line verdict grammar and every caller re-implements parsing; failure mode (prose before the marker, marker collision) has already caused lost work.

rust-agent-adapt

Seven single-repo scripts each duplicate find_runner()/compose_prompt()/ALLOWED_TOOLS scaffolding and pipe prompts into claude-code —print; the only structured datum recovered from any run is a PR URL grepped from stdout.

Where: scripts/bring_up_to_date.sh · scripts/lib/pr_prompt.sh · scripts/cargo_update.sh

Pain: Agent-step harness rebuilt seven times with visible copy-paste drift; parameters smuggled via exported env vars; no typed result at all.

rust-fluent-git

fluent-claude: an entire crate hand-building the typed invoker layer — session resume/continue/fork builders, JSON output format, typed PromptResult/error types — with no workflow layer consuming it.

Where: crates/fluent-claude/src/ops/session.rs · crates/fluent-claude/src/ops/prompt.rs · crates/fluent-core/src/run.rs

Pain: The author wants agent invocation as a first-class typed task badly enough to build the invoker crate; the composition layer above it is the missing piece.

ralph

Bash loop decides termination by grepping the transcript for <promise>COMPLETE</promise>; failures swallowed with || true; no distinction between agent failure, tool crash, and incomplete work.

Where: ralph.sh · CLAUDE.md

Pain: The entire control plane is string-matching on untyped CLI output — no typed result, no cost reporting, no cancellation.

2b

Three independent prompt-to-typed-object stacks (instructor+zod in Deno, instructor+pydantic in Python, plus a stubbed generic get_structured_response helper) each hand-wiring schema plumbing.

Where: apps/pro/cli.ts · apps/professional/professional/util/llm.py · apps/professional/professional/generate.py

Pain: The abandoned generic helper is precisely the typed LLM step @flow/llm ships; each stack re-invents it.

Flowline fit

@flow/llm is the direct answer (typed agent step + invokers). Gaps the evidence demands: session resume/continue/fork semantics (fluent-claude, kiosk’s —continue gate loops), per-step tool-allowlist config as typed input, and cost/token accounting as a shipped field — several implementations (ontological, DarkFactory) designed cost parsing and never built it.

Proposed API

Builds on: — (the base primitive) Introduces: agent(), AgentRequest/AgentResponse, invokers

What pr-check’s marker grammar and DarkFactory’s sentinel regexes become — the step asks for typed output; the invoker decides how the model is reached:

import { agent, cliInvoker } from "flowline/llm";
import { z } from "zod";

// Today: the skill prints "PR-CHECK-VERDICT: NEEDS-RESPONSE reason=..." and every
// caller regex-parses stdout.
const Verdict = z.object({
  verdict: z.enum(["CLEAN", "NEEDS-RESPONSE", "CONFLICTS", "CI-FAILED", "MERGED", "CLOSED"]),
  reason: z.string(),
});

const r = await agent({
  prompt: `Classify the state of PR #${prNumber} ...`,
  tools: ["Bash", "Read"],
  output: Verdict,                 // parsed + zod-validated, or a typed step failure
  invoker: cliInvoker({ bin: "claude", model: "sonnet" }),
  timeout: 300_000,
});

if (r.ok) dispatch(r.value);       // { verdict, reason } — no marker grammar, no collisions
report(r.usage?.costUsd, r.toolCounts);  // accounting shipped, not bespoke

Extensions the evidence demands beyond the RFC sketch: session resume/continue/fork on the invoker (what fluent-claude hand-built), and the per-step tool allowlist as typed input rather than an exported env var (rust-agent-adapt’s ALLOWED_TOOLS).

---

← Back to Notes