What problem this demonstrates
Kiro CLI (the ACP agent backend openab runs Kiro through) is a plain command-line tool: you pipe it a prompt, it prints an answer, it exits. Nothing about that tells you whether the answer was good, what it cost, or whether a prompt change actually helped. Langfuse exists to answer exactly those questions for LLM applications — but Kiro CLI does not natively emit Langfuse traces. Someone has to bridge the two.
This lab builds that bridge, then uses it for a real evaluation workflow: run a small support-FAQ dataset through Kiro CLI under two different system prompts, score every answer with an LLM judge, and compare the two runs — plus a smaller side task swapping models mid-task to see whether a more expensive model is actually earning its cost.
Minimum concepts involved:
- LLM observability — trace → observation → score as the unit of record for one Kiro CLI call.
- LLM-as-judge evaluation — a second, independent model call grades the first call's output against a rubric.
- ACP agent backend — why Kiro CLI behaves like a session-aware service (stdin in, first response out) rather than a one-shot function call.
System architecture & flow
(ACP agent, per-call subprocess)
grouped into a named dataset run
side-by-side comparison
Two data paths run in parallel per dataset item: the answer path (task → Kiro CLI → output) and the score path (output + expected_output → judge → Evaluation). A third, independent path — the cost path — runs entirely inside Kiro CLI's own process via the Stop hook, writing usage snapshots that a run-level evaluator later aggregates. All three paths land in the same Langfuse dataset run, which is what makes "was it better, and what did it cost" answerable from one screen.
What each tool does — and where responsibility crosses
| Tool | Responsible for | Not responsible for |
|---|---|---|
| Kiro CLI | Running the agent, producing an answer, firing lifecycle hooks (Stop, etc.) | Structured tracing, scoring, cost aggregation — it emits text and exits |
| Harness (this lab's Python) | Turning subprocess calls into Langfuse observations; owning the judge rubric | Being the source of truth for "was Kiro CLI actually reachable" — that's Setup |
The Stop hook | Triggering a credit-usage check after every response, without the harness having to ask | Anything if its trigger field doesn't match a real Kiro trigger name — see Failure case 2 |
| Langfuse | Storing traces/scores, grouping them into comparable dataset runs, rendering the UI | Judging answer quality itself — that's the judge call, which is just another Kiro CLI call |
The seam most worth paying attention to: the judge is not a Langfuse feature. It is an ordinary, independent Kiro CLI call that happens to run inside an "evaluator" function. Langfuse only stores whatever score that call's output gets parsed into.
Learner steps & checkpoints
- Read the scaffold, starting with the Kiro CLI wrapper and the Langfuse glue module.
- Write down predictions for 5 checkpoints before running anything.
- Run the setup/smoke-test script — it verifies Kiro CLI is reachable, Langfuse is reachable, and the exact stdin invocation shape the harness depends on.
- Implement the one stubbed core function — an LLM-as-judge scorer.
- Run the baseline variant, then the improved variant, watching for a data gap between them.
- Run the comparison check script and read what it reports.
- Run the model-switch task across 2–3 models on a small item subset.
- Open both dataset runs in the Langfuse UI and use its built-in run-comparison view.
Observable signals at each stage: Kiro CLI's stdout (does a single piped input produce a single clean reply?), usage-log.jsonl (does it gain a line after every response?), the Langfuse UI (do traces, generations, and scores show up per item?), and the comparison script's own pass/fail output.
Failure cases — what to inspect (not the fix)
Case 1 — a judge score that should not be trusted at face value
One dataset item is deliberately constructed so that no single answer is unambiguously "correct" — the question depends on missing policy context. Run your judge on it anyway and read its reasoning, not just its number. What to inspect: whether your rubric has any way to represent "this shouldn't be scored the same way as a factual lookup," and what happens to your aggregate metrics if it doesn't.
Case 2 — a cost-tracking hook that looks configured but isn't
A Stop-triggered hook is supposed to log a credit-usage snapshot after every Kiro CLI response, so nobody has to babysit a usage command by hand. After running one of the two dataset variants, the usage log for that run will be suspiciously empty. Kiro CLI does not error or warn about this — the hook file is valid JSON and is silently never matched. What to inspect: the hook's trigger field against Kiro's current list of valid trigger names, and why a syntactically valid hook config can still do nothing.
The exact root cause and fix are intentionally not published here — they're the lab's private answer key. Compare your findings after running lab-review.md.
Acceptance criteria
- Two named dataset runs exist in Langfuse, each with one scored item per dataset row.
- The comparison script exits successfully once both runs are healthy.
- The improved prompt variant scores higher than baseline on most — not necessarily all — items; the deliberately ambiguous item is allowed to be the exception.
- The model-switch task produces a score and a latency figure for every (model × item) pair you ran.
- You can explain, in your own words, why the ambiguous item doesn't falsify your judge, and how you diagnosed the missing cost data — not just that you fixed it.
Prerequisites, time, cost, and fallback
| Kiro CLI | Installed and authenticated (API key). Not bundled with every environment — verify before starting. |
| Langfuse | A reachable Langfuse project (in-cluster instance or cloud.langfuse.com) with your own API keys. |
| Estimated time | 90–120 minutes, including reading the scaffold and both failure investigations. |
| Cost risk | Roughly 40–50 total Kiro CLI calls across both dataset variants, judge calls, and the model-comparison task. Check your usage before and after if metered. |
| Mock fallback | None. This lab is intentionally real-tool — the failure modes being taught only exist when a real CLI process and a real hook-matching engine are involved. |
Why this lab has no local-mock fallback
A mocked Kiro CLI would need to fake exactly the two things this lab teaches: the exact non-interactive stdin/stdout shape real CLIs vary on, and a hook-trigger-matching engine that silently no-ops on an unrecognized name. Faking both would mean writing (and implicitly trusting) a second implementation of the very thing being verified — which defeats the point of a real-tool lab.