Hamza Ouhssaine
Contact

Prompty

Prompts shipped like code: versioned, evaluated, A/B tested

Prompty treats a prompt as a versioned artifact, not a string someone edits in production and then watches for complaints. Every version is graded against a golden set by an LLM judge before it can ship, then rolled out to a share of real traffic and judged by statistics. Deploying and rolling back are the same operation: moving a label.

Type
Personal project, 2026, designed and built end to end
Built
FastAPI service, Python SDK, React dashboard
Statistics
Wilson intervals, z-test, Welch’s t-test, paired t-test, power analysis, SRM check
Models
Any OpenAI-compatible endpoint, or a deterministic mock
Tests
151 test functions (pytest), statistics checked by simulation

A working local prototype: single user, no authentication. Its demo data comes from a deterministic mock model, so it shows the machinery, not the quality of any model.

One completion, end to end

Applications call Prompty instead of the model. One call decides which version this user gets, renders it, calls the model and records everything an experiment will need later.

One live request through the SDK
SDK call prompt key + user id Resolve target experiment or label Assign variant sticky hash bucket Render {{variables}} only Model call OpenAI-compatible Completions log cost, tokens, latency Response text, variant, cost SDK call prompt key + user id Resolve target experiment or label Assign variant sticky hash bucket Render {{variables}} only Model call OpenAI-compatible Completions log cost, tokens, latency Response text, variant, cost
  • Input or output
  • LLM call
  • Code
  • Data store
  1. SDK call client.complete("support-reply", variables={"message": "My order never arrived"}, unit_id="user-42").
  2. Resolve target An experiment is running on support-reply, so the experiment decides the version. Without one, the “production” label would.
  3. Assign variant Bucket = sha256(experiment:unit) mod 10,000, walked through the variants’ weights in basis points. The assignment is stored, so user-42 keeps the same variant even if the weights change.
  4. Render Plain {{name}} substitution with no template logic. A missing variable is a 422, never a half-filled prompt.
  5. Model call The variant’s own model and settings. Rate limits and connection errors are recorded as status “error” instead of raised; a content filter becomes “refusal”.
  6. Completions log One row in the single completions table, priced per million tokens and tagged source = live. Eval and judge calls share the table under their own source.
  7. Response Text, variant, tokens, cost and latency. Later, client.track() attaches the outcome, a thumbs-up or a number, to the same variant.

Other routes through the graph

  • No experiment running: the deployment label picks the version, and nothing is stored.
  • No unit id: a random draw that is not persisted.
  • Provider error: the logged completion comes back inside a 502, so the caller sees what was recorded.

From a new version to production

A version has to pass two different tests: offline, against answers already judged good; online, against real traffic. Either can stop it.

One prompt version on its way to production
New version immutable Offline eval golden set + judge Deploy gate paired t-test A/B experiment sticky traffic split Results SRM, z-test, Welch Production the label moves New version immutable Offline eval golden set + judge Deploy gate paired t-test A/B experiment sticky traffic split Results SRM, z-test, Welch Production the label moves
  • Input or output
  • LLM call
  • Code
  • Check or gate
  1. New version v3 of support-reply. Versions never change once created; labels are pointers to them.
  2. Offline eval Each golden case runs through v3, five at a time, then a judge model scores the answer from 1 to 5 on an absolute rubric, not side by side, which avoids position bias.
  3. Deploy gate Compared case by case with the version in production: a paired t-test over at least 5 shared cases. Only a significant drop blocks; absence of evidence never does.
  4. A/B experiment Traffic is split by weight in basis points, sticky per user, with the sample size planned in advance by a power analysis.
  5. Results Wilson intervals per variant, a pooled z-test for rates, Welch’s t-test for continuous metrics. A sample-ratio mismatch at p < 0.001 suppresses every “significant” flag; cost, latency and error-rate guardrails watch for regressions.
  6. Production Completing the experiment moves the production label to the winner. A guardrail regression refuses with a 409 unless forced, the one human override.

Other routes through the graph

  • Deploy gate fails: the deploy is refused and the label stays where it was.
  • Fewer than 5 events in a cell: the z-test isn’t trusted, and the error-rate guardrail switches to Fisher’s exact test.

Statistics chosen on purpose

An A/B platform is only as honest as its tests. Each choice here answers a specific way experiments mislead.

  • Pooled for the test, unpooled for the interval

    The z-test uses the pooled standard error, as the null hypothesis implies; the interval on the difference uses the unpooled one. Each variant’s own rate gets a Wilson interval, which stays sensible near 0 and 1.

  • Paired when the cases are shared

    Offline runs score the same golden cases, so comparisons are paired. In the test suite, 40 shared cases give p < 0.001 paired and p > 0.05 unpaired on the same scores.

  • Sample ratio mismatch first

    If a 50/50 split delivers 45/55, assignment is broken and every number downstream is suspect. The check runs from 100 observations at p < 0.001, and silences all significance flags when it fires.

  • Guardrails beside the goal

    A variant that wins on thumbs-up but costs more, runs slower or fails more often is flagged. Cost and latency need both a 10% relative regression and significance.

  • Aggregates, not rows

    The statistics module has no database access. One GROUP BY feeds it, so the results endpoint costs the same at any traffic.

  • Checked by simulation

    The power calculation is tested by simulation: 20,000 runs land at 0.80 ± 0.02 power, and with no true effect the false-positive rate stays at 0.05 ± 0.01.

Engineering choices

  • One table for every call

    Live, eval and judge calls land in one completions table with a source column. Aggregation must filter on live traffic, and a test fails if it stops doing so.

  • Deploy and rollback are one operation

    Versions are immutable and labels point at them. Rolling back is moving the label back.

  • Sticky, and stored

    Assignment is a hash of experiment and user, persisted on first sight, so it survives a change of weights. Two first requests racing for one user resolve to the stored row.

  • Errors are data

    Provider failures are logged like any completion, so the error-rate guardrail can see a variant that fails more often.

  • Immutable judges

    A judge can’t be edited after creation, so scores from different weeks stay comparable.

  • Fail at startup, not on the bill

    An unknown provider name stops the service at startup, so a typo can’t fall through to a paid endpoint.

Stack

Backend
Python, FastAPI, async SQLAlchemy, SQLite (WAL), Pydantic settings
Statistics
SciPy, NumPy
LLM
OpenAI-compatible SDK, LLM judge with JSON output, deterministic mock provider
SDK
Python client on httpx
Frontend
React 19, TanStack Query, React Router, Radix, Tailwind CSS 4, Vite
Quality
pytest, pytest-asyncio, simulation tests of the statistics

Building something similar?

I’m happy to walk through the design, the trade-offs, or what I would change next.

Email me All projects