LiveLLM testing · CI/CD

LLM test gate

Test an LLM feature the way the post says: structure, facts, then meaning. Compare two prompts and see which one the gate lets ship.

Ship it7/8 cases · 88%

Cosmetic surgery

Rhinoplasty, marked cosmetic by the treating doctor. Bill ₹1,20,000.

Must say: 5.4Never: 4.23.1

StructureValid JSON with every field, and an allowed decision.
FactsHas everything it must, and nothing it must not.
Meaning · SUPPORTEDRight clause, and the reason matches the doctor’s note.

Answer (prompt v1) — edit it

{
  "decision": "reject",
  "clause": "5.4",
  "reason": "The doctor marks the procedure as cosmetic, which clause 5.4 excludes."
}

The made-up policy

3.1: Room rent: up to 1% of the sum insured per day.
3.2: Hospital stay must be at least 24 hours.
3.6: Day-care procedures are covered, up to ₹40,000.
3.8: Road ambulance: up to ₹2,000 per claim.
4.2: Pre-existing diseases: 2-year waiting period.
5.4: Cosmetic treatment is not covered.

1 · Structure (JSON, fields, allowed values)2 · Facts (must / never say)3 · Meaning (judge label)Instead: exact match to last week’s answer

Pass the build at85%

Structure and critical cases must always pass

The gate is open. Prompt v1 passes. Switch to v2, the shorter prompt, and see what it breaks.

Every case, clause and answer here is made up by hand — none of it is real model output.

How it works

1Exact match fails even correct answers, because the wording changes every run.
2Structure and facts are plain code checks — edit any answer and they re-run as you type.
3Meaning uses a judge model’s label. This page has no model, so those labels are written with the cases, and an edited answer is marked “not re-judged”.
4The build passes on a percentage, but structure checks and critical cases must always pass.

Read the post: How to test an LLM feature in CI

More demos

All demos

LiveDatabases · Built from scratch

QueryLite: a SQL database

A SQL database written from scratch — parser, B+ tree indexes, a query planner, joins and transactions — running live on 22,000 rows in your browser.

Try it

LiveDistributed systems · Built from scratch

Raft consensus, live

The algorithm that keeps etcd and Kubernetes consistent. Five servers elect a leader and replicate a log — crash them and split the network while it runs.

Try it

LiveReal-time · Built from scratch

Live collaborative editor

Real-time editing with no server. Three devices share a note — take one offline, edit everywhere, reconnect, and they merge. Open a second tab and it syncs live.

Try it