Hamza Ouhssaine
Contact

ML Engineer

An AutoML agent that never sees its own holdout set

ML Engineer is an autonomous agent that profiles a dataset, plans experiments, writes scikit-learn code and runs it in a sandbox, inside a harness built to check whether it works honestly. The agent never sees the holdout set: not by convention, by construction. Its headline number isn’t its best score but the gap between the cross-validation score it optimised and the holdout score it never saw.

Type
Personal project, 2026, designed and built end to end
Agent
Custom loop with four LLM roles: planner, coder, critic, submitter
Sandbox
Docker, no network, 2 GB, 2 CPUs, read-only root, 10-minute fits
Tasks
Six tabular tasks: five with a planted trap, one clean control
Tests
188 test cases (pytest), 16 of them in Docker

A working prototype. The whole pipeline runs end to end with a scripted stand-in for the model; no benchmark against a model has been published yet.

The holdout never meets the agent

The split happens before the agent exists, and the function that runs the agent has no parameter that could carry a holdout. Scoring happens in another container, after the agent has returned.

One task through the harness
Dataset seeded task or a CSV Holdout split by time, group or class Agent run sees train data only Holdout set masked to predict time Holdout scoring separate container Report gap, traps, baseline Dataset seeded task or a CSV Holdout split by time, group or class Agent run sees train data only Holdout set masked to predict time Holdout scoring separate container Report gap, traps, baseline
  • Input or output
  • LLM call
  • Code
  • Data store
  • Check or gate
  1. Dataset A churn task generated from a seed, with one trap planted: a column filled in after the outcome. Or your own CSV, up to 64 MB.
  2. Holdout split Time tasks hold out the latest 20%, grouped tasks 25% of the groups, the rest a stratified 20%. Columns only known after the outcome are blanked in the holdout, as they would be at prediction time.
  3. Agent run Planner, coder, critic and submitter work on train.parquet only, for up to 4 experiments. The holdout isn’t hidden from them; it was never passed in.
  4. Holdout scoring After the agent returns, another container loads each fitted model and the holdout. Every experiment is scored, not only the submitted one.
  5. Report The CV − holdout gap per experiment (above 0.05 counts as self-deceived), traps caught, missed and falsely flagged, and a naive baseline fitted on every column in the same sandbox.

Other routes through the graph

  • No sandbox available: the benchmark stops rather than run code on the host. Host execution needs an explicit flag, and every result says so.
  • One task fails: it is logged and the other tasks carry on.

Inside the agent loop

Four language-model roles, called in turn by plain Python. None of them can call another, and none of them can overrule the arithmetic in between.

Two experiments, one of them caught leaking
Briefing profile + column audit Planner ranked hypotheses Coder writes candidate.py Sandbox fit no network, train only Leakage checks 7, arithmetic only Critic a smaller model Submitter chooses on CV alone Briefing profile + column audit Planner ranked hypotheses Coder writes candidate.py Sandbox fit no network, train only Leakage checks 7, arithmetic only Critic a smaller model Submitter chooses on CV alone
  • Input or output
  • LLM call
  • Code
  • Check or gate
  1. Briefing Code, not a model, profiles the training data, audits the columns and suggests a CV scheme, labelled “advice, not a rule”: whether the agent follows it is part of what is measured.
  2. Planner Returns ranked hypotheses as schema-validated JSON.
  3. Coder Writes candidate.py for the first hypothesis. It sees CV scores, the last 4 lessons and the last 3 failures; no holdout score exists yet.
  4. Sandbox fit Docker with no network, 2 GB of memory, 2 CPUs, a read-only filesystem and train.parquet only. Out: CV score and strategy, features used, the fitted model.
  5. Leakage checks Say the model kept the post-outcome column: one feature separating the classes with an AUC of 0.97 or more is a critical finding, and the experiment is quarantined.
  6. Critic A smaller model, to partly control for a model favouring its own work, reads the findings and pivots: drop the column. One pivot is allowed per run.
  7. Coder The next hypothesis, with the lesson.
  8. Sandbox fit A second experiment in a fresh container.
  9. Leakage checks No critical finding this time.
  10. Critic Stop.
  11. Submitter Chooses one experiment from CV scores and findings alone. If it names one that doesn’t exist, the harness falls back to the best CV that wasn’t quarantined.

Other routes through the graph

  • Coder fails: the attempt is recorded as a coder failure and the loop moves on, with no sandbox or critic run.
  • Planner fails: the run ends with an error.
  • Critic fails: the experiment is accepted and the loop continues.
  • At most 4 experiments; an empty queue also goes to the submitter.

Checks a model can’t argue with

Leakage detection is plain arithmetic, with no language model anywhere in the module: a detector a model can talk itself out of reports whatever the model was hoping for.

Five of the seven leakage checks, with their thresholds in the code
CheckFlags when
One feature separates the classesAUC ≥ 0.90 (critical from 0.97)
Missing values separate the classesseparation ≥ 0.85
Cross-validation too good to be trueCV ≥ 0.985
Near-duplicate rowsshare ≥ 0.15
Class imbalancepositive rate < 0.05

What gets measured

  • The gap

    CV score minus holdout score, for every experiment. A gap above 0.05 is labelled self-deceived.

  • Traps

    Five tasks carry one planted trap each (target leakage, time order, groups, duplicates, class imbalance) and one is a clean control. Caught, missed and false alarms are counted separately.

  • A naive baseline

    Gradient boosting on every column, fitted in the same sandbox. It is meant to fall into the traps, which shows what the agent avoided.

  • Cost

    Tokens, cost and wall time for every run, from the same records.

Engineering choices

  • Isolation by signature

    The agent’s entry point has no parameter that could carry a holdout. Tests check that signature, check the loop never imports the scorer, and scan every recorded prompt for every holdout score at 2 to 8 decimals.

  • Masked to prediction time

    Post-outcome columns are blanked in the holdout; otherwise a planted leak would just look like a good feature.

  • Advise, don’t enforce

    The split guard recommends a CV scheme instead of imposing one. Imposing it would make the traps unreachable.

  • Fail closed

    No sandbox, no run. Running on the host needs an explicit flag and is labelled in every result.

  • Features live in the pipeline

    Data preparation may only drop rows, so every transformation sits inside the pipeline that also runs at prediction time.

  • Failures are data

    A crashed fit exits cleanly with the stage that failed and becomes a recorded experiment, not a crash.

Stack

Agent
custom Python loop, OpenAI-compatible SDK, schema-validated JSON, strict mode with a fallback
Machine learning
scikit-learn, pandas, NumPy, SciPy, PyArrow, joblib
Sandbox
Docker: no network, capped CPU, memory and processes, read-only root
Backend
Python 3.12, FastAPI, WebSocket, SQLModel, SQLite, record/replay cassettes
Frontend
React 18, Vite, Tailwind CSS 4, hand-drawn SVG charts
Quality
pytest, ruff, GitHub Actions

Building something similar?

I’m happy to walk through the design, the trade-offs, or what I would change next.

Email me All projects