# Hamza Ouhssaine: GenAI Engineer & Senior Data Scientist

Hamza Ouhssaine is a GenAI Engineer and Senior Data Scientist based in Paris, France. He builds LLM agents, RAG systems and the evaluations that prove them, in production at BPCE & Natixis, on top of 6+ years of machine learning, computer vision and big data for banking, heavy industry and travel.

- Location: Paris, France. Open to relocation.
- Email: hamzaouhssaine@gmail.com
- LinkedIn: https://www.linkedin.com/in/hamza-ouhssaine-45b40112a/
- GitHub: https://github.com/hamzaca
- Website: https://www.hamza-ouhssaine.com/
- Expertise: Generative AI & LLMs, RAG & AI agents, LLM evaluation, Prompt engineering, Machine learning, Computer vision, NLP, MLOps

## Experience

### Data Scientist, GenAI, BPCE & Natixis

Jan 2026 – present, Paris

- **Steering ALM Metrics:** Platform automating liquidity (LCR) monitoring and reporting across Groupe BPCE, built and deployed with Treasury, ALM and IT teams to analyze large-scale balance-sheet data and explain each change. An LLM-powered AI agent answers analysts’ questions in plain language and drills into the data.
- **Insight Generator:** Financial data analysis and business intelligence platform for Natixis. Users import their files, explore them with pivot tables and charts, and export reports. A generative AI assistant (LLM, RAG) helps teams extract insights and improve business performance. [How Insight Generator works](https://www.hamza-ouhssaine.com/projects/insight-generator/)
- **AAR (Accounting Analytical Review):** App streamlining quarterly statutory and consolidated accounting reviews for **about 200 Natixis entities**. It drafts analyses of significant account movements and Audit Committee report sections for reviewers to edit and validate. Used by Permanent Controls and Consolidation teams to cut manual effort and standardize a process behind management decisions and audits.
- **APAC T&E Policy Chatbot:** Generative AI chatbot (LLM, RAG) for the APAC travel and entertainment (T&E) policy. It automates answers to employees’ questions, grounded in official guidelines with links to sources. Administrators add documents, set custom rules and review feedback from a dedicated dashboard.

Tools: Generative AI, LLMs, RAG, LangGraph, LangChain, ChromaDB, Python, Machine Learning, Microsoft Graph API, AngularJS, JavaScript, Docker, Jenkins, OpenShift

### Data Scientist, ArcelorMittal

Dec 2022 – Dec 2025, Marseille

- **Product ID Recognition:** Computer-vision pipelines reading steel product IDs at a **0.098% character error rate**, reducing manual checks and improving traceability.
- **Predictive models for steel production:** Cooling-control models improving production reliability, and computer-vision defect detection on casting rolls and products, integrated with cross-functional teams to cut manual inspection.
- **MLOps on Google Cloud:** Set up the Vertex AI side end to end (Workbench, custom training, Pipelines, Model Registry, online and batch prediction, Gemini-based RAG) for product-ID OCR, defect detection and an LLM/RAG assistant, in a hybrid setup: real-time inference in the plant, training and batch jobs on GCP.

Tools: Python, PyTorch, Deep Learning, Computer Vision, OpenCV, Image Processing, LLMs, RAG, Vertex AI, Elasticsearch, Kibana, Dataiku, Decision Sciences

### Big Data Engineer, Amadeus

Jan 2021 – Nov 2022, Nice

- Built and maintained production ETL pipelines over large-scale airline datasets for Revenue Management, with automated testing and deployment.
- Delivered projects for Southwest, Polish Airlines, Singapore Airlines and Kuwait Airways; served as team leader and led hiring for the team.

### Data Scientist Intern, Meelo

Mar – Nov 2020, Lille

- Optical character recognition for specific administrative documents, with state-of-the-art document OCR built on LSTM, CNN and Transformer networks.
- An API that recognises companies’ tax reports and computes their financial score.
- Fraud and non-payment detection for consumer credit, with machine learning on open data from INSEE.
- Corrective maintenance of the prediction models in production.

Tools: PyTorch, Machine Learning, OCR, LSTM, CNN, Transformers

### Deep Learning Researcher, CRIStAL Lab

Nov 2019 – May 2020, Lille

- Deep learning for steganalysis: detecting hidden messages in images with a deep residual network, ResNet-12.
- Channel pruning of the ResNet-12 with CALPA-Net.

Tools: PyTorch, Image Processing, ResNet, CALPA-Net

## Projects

Personal projects' source code: https://github.com/hamzaca

### At BPCE & Natixis (work)

- **Insight Generator:** LLM agent for finance spreadsheets: clean data, exact pivots, answers computed in SQL. Finance teams upload Excel or CSV exports; it sniffs the real format, parses French and English finance values, flags quality issues, versions every change and builds Excel-style pivots with exact subtotals. The assistant writes SQL, runs it on DuckDB, draws bridge charts and explains the result, then exports to Excel, Word and PowerPoint. Stack: Python, FastAPI, pandas, DuckDB, ChromaDB (RAG), LLM agent with tool calling, vanilla JavaScript. https://www.hamza-ouhssaine.com/projects/insight-generator/

### RAG and LLM systems (personal)

- **Sanad:** Agentic RAG over private documents where every claim resolves to a page and a bounding box. Reads nine formats, including scans and photos through vision OCR; chunks on structure and never splits a table row; adds Contextual Retrieval. Hybrid BM25 + vector retrieval fused with RRF, then a cross-encoder rerank: on the sample corpus it lifts a weak embedder’s MRR from 0.53 to 0.94. A LangGraph agent rewrites follow-ups, decomposes multi-part questions, critiques its own evidence and declines rather than guessing. Every citation quote is verified against its source chunk, and an eval harness scores recall, MRR and groundedness per model. Ingestion pauses and resumes mid-file without paying for any call twice. Stack: LangGraph, FastAPI, hybrid BM25 + vector retrieval (RRF), cross-encoder rerank, Claude vision OCR, ChromaDB, SQLite FTS5, pausable/resumable pipelines, Angular. https://www.hamza-ouhssaine.com/projects/sanad/
- **Applied LLM Projects:** Monorepo of five LLM systems behind one auto-discovering gateway with a unified cost tracker. A function-calling SQL agent with guardrails, a RAG eval harness (recall@k, LLM-judge groundedness, chunking ablations), a forecasting narrator with rolling-origin backtests, a drift sentinel with small-to-large model triage, and a distillation pipeline measuring an accuracy-vs-cost Pareto frontier. Stack: FastAPI gateway, OpenAI-compatible local models and all major LLM providers, RAG evals, distillation, cost tracking, Angular.

### LLM agents and evaluation (personal)

- **Code Auditor:** Autonomous LangGraph agent auditing and refactoring Git repos end to end. Malware and secret preflight gates, static and security analysis, an LLM-generated risk-ranked roadmap, then sandboxed fixes with test-and-rollback. It treats the repo as hostile input: network-disabled containers and a human approval gate. 478 backend test functions. Stack: LangGraph, FastAPI, network-disabled Docker sandboxes, ruff/mypy, bandit/semgrep/osv-scanner, ClamAV/YARA/TruffleHog, React. https://www.hamza-ouhssaine.com/projects/code-auditor/
- **Proving Ground:** Agent evaluation benchmark that scores trajectories against world state, not the agent’s own report. A deterministic event-sourced company simulator with 20 tools, 24 scenarios (a third unwinnable by acting), 4 scaffolds and 6 policy profiles with taint tracking and irreversible-action gates. Headline metric: hallucinated success, claiming “done” when the ledger disagrees. Stack: FastAPI, WebSocket, event-sourced simulator, ReAct/Plan-Execute/Reflexion scaffolds, taint tracking, React. https://www.hamza-ouhssaine.com/projects/proving-ground/
- **ML Engineer:** Autonomous AutoML agent architecturally prevented from seeing its own holdout set. It profiles a dataset, plans and codes scikit-learn experiments in a network-disabled sandbox with resource caps, and runs seven leakage checks it cannot overrule; the holdout is scored out of process. It reports the CV-minus-holdout gap against a naive baseline. Stack: Python, scikit-learn, LLM code generation, sandbox with CPU/memory/PID caps, out-of-process holdout scoring. https://www.hamza-ouhssaine.com/projects/ml-engineer/
- **Prompty:** Prompt versioning, offline evaluation and A/B testing platform for LLM features. Immutable versions with movable deploy labels (deploy and rollback are the same operation), LLM-judge grading against a golden set that blocks regressions, sticky traffic splitting, and a statistics layer: z-tests, Welch’s t-test, power analysis, SRM detection, cost and latency guardrails. Stack: FastAPI, Python SDK, LLM-judge eval harness, z-tests, Welch’s t-test, power analysis, SRM detection, deterministic traffic splitting, React. https://www.hamza-ouhssaine.com/projects/prompty/

### Forecasting and full-stack products (personal)

- **Vantage / Insight Terminal:** Full-stack market intelligence platform over ~194 instruments: equities, crypto and commodity futures. An ensemble of XGBoost, LightGBM, a PyTorch LSTM, ARIMA and Prophet against naive baselines with honest walk-forward backtesting, LLM-scored news sentiment, and an AI-written daily brief served from a local OpenAI-compatible model. Stack: FastAPI, SQLModel, PyTorch LSTM, XGBoost/LightGBM, ARIMA/Prophet, walk-forward backtesting, Angular.
- **Tahanot ERP:** Self-hosted, Odoo-inspired mini-ERP (sales, purchase, inventory, dashboard) rebuilt from scratch. A generic model registry with domain filters, state-machine documents, quant-based stock with backorders, product variants, invoice payments and credit notes, plus a tool-calling AI assistant over the service layer. Bilingual EN/FR, with Postgres concurrency and locking tests. Stack: FastAPI, SQLAlchemy, PostgreSQL, Alembic, Docker, tool-calling assistant, React. https://www.hamza-ouhssaine.com/projects/tahanot/

## Skills

- GenAI & LLMs: LangGraph, LangChain, Hugging Face, LLM APIs (Claude, Gemini, OpenAI-compatible), ChromaDB, spaCy, NLTK
- Code: Python, Scala, SQL, JavaScript
- ML & deep learning: PyTorch, TensorFlow, Keras, JAX, Haiku, scikit-learn, XGBoost, LightGBM
- AI apps & APIs: FastAPI, Chainlit, Streamlit, React, Angular
- Data: Pandas, Polars, Apache Spark, Hadoop, DuckDB
- Databases & search: SQL / NoSQL, PostgreSQL, Impala, MongoDB, Cassandra, Elasticsearch
- Platforms: Docker, Kubernetes, OpenShift, Jenkins, GCP (Vertex AI), Dataiku

### Methods

- Generative AI: agents and tool calling, RAG with hybrid retrieval (BM25 + vectors + RRF), reranking, LLM evaluation and LLM-judge grading, prompt versioning and A/B testing
- Machine learning: OCR, object detection (YOLO), speech recognition (Whisper), time-series forecasting
- Engineering: ETL pipelines, CI/CD, Git, Agile (SAFe, Scrum)

## Education

### Engineer’s Degree, Data Science & AI, Centrale Lille (2016 – 2020)

Master’s-level engineering degree from one of France’s leading Grandes Écoles, part of the Écoles Centrale group: two years of generalist, multidisciplinary engineering training, then a specialization in data science and AI taught by researchers and industry professionals.

Coursework: Machine learning; Big data; Statistical modeling; Data-driven decision-making; Applied mathematics; Optimization; Relational databases (SQL); Project management; Innovation; Generalist core in physics, engineering sciences, computer science and economics; Data science integration project.
Skills: Data Science, Big Data, Deep Learning, Reinforcement Learning, LLMs, RAG, Dataiku.

### Master 2 (M2), Applied Mathematics, University of Lille (2019 – 2020)

Master’s degree in applied mathematics completed in one year in parallel with the engineering degree, through a partnership between the University of Lille and Centrale Lille.

Coursework: Stochastic processes: modeling random phenomena that evolve over time; Concentration of measure: high-dimensional probability, the theory behind generalization guarantees in machine learning.

### Bachelor’s Degree, Mathematics, University of Lille (2017)

Applied mathematics degree taken in parallel with the first year at Centrale Lille, through a partnership between the University of Lille and Centrale Lille.

Coursework: Topology; Probability; Differential equations; Differential calculus.

### Classes Préparatoires MPSI / MP*, Ibn Ghazi (2014 – 2016)

Two-year intensive mathematics and physics program with weekly oral exams, preparing for the competitive entrance exams to France’s top engineering schools. Admitted to Centrale Lille via the Centrale-Supélec exam.

Coursework: MPSI: analysis, algebra, finite probability, inner product spaces, linear differential equations; MP* (advanced “star” class): matrix reduction and the spectral theorem, normed spaces, power series, parameter integrals, differential calculus, discrete random variables and the law of large numbers; Also physics, chemistry, engineering sciences and computer science (Python).

## Certifications

- Build and Modernize Applications With Generative AI (Google Cloud Skills Boost, Feb 2026)
- Advanced Machine Learning on Google Cloud Specialization (Coursera, May 2021)
- Data Engineering, Big Data, and Machine Learning on GCP Specialization (Coursera, Mar 2021)
- Certified SAFe® 5 Practitioner (Scaled Agile, Inc., Dec 2021 – Dec 2022)
- MOOC “Gestion de Projet” (project management) (Centrale Lille)
- TOEIC (TOEIC® Program)

## Languages

- French: Fluent
- English: Fluent
- Arabic: Fluent
- Tamazight: Native

## Case study: Insight Generator, an AI analyst for finance teams’ spreadsheets

Full page: https://www.hamza-ouhssaine.com/projects/insight-generator/

Insight Generator is a web app that turns the Excel and CSV files finance teams live in into clean data, Excel-style pivot tables, charts and written analysis. Analysts upload a workbook, clean and reshape it, build pivots and ask an assistant questions in plain language. Every number in an answer comes from a query that actually ran on their data.

- Context: Natixis, Groupe BPCE. Paris, 2026
- Role: Data Scientist, GenAI
- Users: Finance and reporting teams
- Interface: French and English
- Stack: Python 3.13, FastAPI, pandas, DuckDB, ChromaDB, SQLAlchemy, vanilla JavaScript

### Why build it

Month-end in a finance team still runs on spreadsheets. Files arrive from different systems with French and English number formats, periods written five different ways, Excel errors, and total rows mixed in with the data. Analysts clean them by hand, rebuild the same pivot tables every month, then rewrite the commentary in Word, PowerPoint or an e-mail.

A general-purpose chatbot doesn’t fix this, because it can’t be trusted with figures. Insight Generator gives the language model the job of planning and explaining, and leaves every calculation to a database engine.

### The assistant: numbers are computed, not generated

The assistant is an agent that works across every table in a session. It doesn’t answer from memory: it writes SQL, runs it on DuckDB, builds pivots and draws charts, including waterfall (bridge) charts, then explains what the results show.

Finance-specific SQL macros (fiscal_year, fiscal_quarter and fiscal_period, with a configurable fiscal-year start) make year-to-date and prior-year comparisons a single line of SQL. Suggested questions adapt to the data: budget vs actual, variances, bridges.

Worked example on synthetic data. Question: “How did Q1 2025 revenue compare with budget, by region?”

```sql
SELECT region,
       SUM(revenue)               AS actual,
       SUM(budget)                AS budget,
       SUM(revenue) - SUM(budget) AS variance
FROM   sales
WHERE  fiscal_year(order_date) = 2025
  AND  fiscal_quarter(order_date) = 1
GROUP  BY region
ORDER  BY variance;
```

Answer: Q1 2025 revenue came in at €13.34M against a €13.75M budget: €411k (3.0%) short. South (−€268k) and West (−€121k) explain most of the gap; East is €78k under and North finished €56k ahead.

### Context from documents

Users can add documents to a private knowledge base: Word, PowerPoint, PDF (layout kept), spreadsheets, HTML pages and e-mails (.eml and Outlook .msg). They are indexed in ChromaDB and retrieved by the assistant and the analysis tab (RAG). Each document belongs to its owner. A document can also be marked as one of three kinds:

- **Template:** Rules and placeholders for a report. Ask “fais le rapport selon le modèle” (write the report following the template) and the assistant follows it. Word and PowerPoint exports take the template’s look, and Excel reporting templates are filled cell by cell.
- **Example:** A past e-mail, PDF or Word deliverable to imitate. Figures in examples are masked before the agent reads them, and any old figure that reaches an answer is flagged.
- **Reference:** A lookup table such as a chart of accounts or FX rates, which the agent can join in SQL.

### Engineering choices

- **The content picks the parser.** Files are sniffed, not trusted by their extension, so real-world exports load instead of failing.
- **Numbers come from queries.** The model plans and explains; DuckDB and pandas compute. Figures in an answer can always be traced to a query.
- **Exact subtotals.** Subtotals are recomputed from raw rows, never summed from their children, so non-additive measures stay correct.
- **Every change is a version.** Undo, audit trails and the month-end replay all come from the same versioned history.
- **Model-agnostic.** Runs on the bank’s internal GenAI platform or on any OpenAI-compatible endpoint, including local models. The LLM features can also be switched off entirely.
- **Secure by default.** Single sign-on (OpenID Connect), user roles, audit logs, prompt management and documents private to their owner. Development shortcuts are refused in production.
- **Tested like a product.** pytest unit, integration and regression suites run against fakes of the LLM SDK and document storage, with a temporary database per test. Node tests cover translations, charts and the pivot layout, and a sample workbook ships with test questions and their expected answers.

### Stack

- Backend: Python 3.13, FastAPI, pandas, DuckDB, SQLAlchemy (SQLite)
- AI: LLM agent with tool calling, RAG on ChromaDB, optional local Hugging Face embeddings
- Frontend: HTML, CSS and vanilla ES modules served by the backend, no build step, PivotTable.js
- Exports: XLSX, CSV, DOCX, PDF, PPTX, EML
- Quality: pytest, node:test, golden questions with expected answers

## Case study: Sanad (سند), agentic RAG that shows the page it answered from

Full page: https://www.hamza-ouhssaine.com/projects/sanad/

Sanad answers questions about a folder of documents. Every claim in an answer carries a marker, and every marker resolves to a page and a bounding box (or a block and character offsets): click it and the original file opens with the exact passage highlighted. When the documents don’t hold the answer, the agent says so instead of guessing.

- Type: Personal project, 2026, designed and built end to end
- Reads: PDF, DOCX, PPTX, XLSX, HTML, Markdown, text, JSON and images
- Backend: Python, FastAPI, LangGraph, ChromaDB, SQLite FTS5, PyMuPDF, SQLModel
- Models: Claude for answers and vision OCR; OpenAI or local embeddings (BGE, E5, MiniLM)
- Frontend: Angular, with a live search trace and a PDF/HTML source viewer
- Tests: Over 350 offline pytest tests

### The name

- **A support:** What holds something up. An answer with nothing under it is just a plausible sentence.
- **A chain of transmission:** In classical Islamic scholarship the sanad is the chain of narrators attached to a report. A report was judged on whether its chain traced back to a source, not on how convincing it sounded. That is this system’s contract: every claim carries a marker, every marker resolves to a page, and a quote that can’t be found in the passage it cites is flagged.
- **A supporting document:** In commercial Arabic a sanad is the paper behind a transaction: سند التسليم, a delivery note; سند الطلب, a purchase order. The exact paperwork Sanad was built to read.

### How a document becomes searchable

1. **Parse every format into one block type.** PDF, Word, PowerPoint (speaker notes included), Excel, saved web pages, Markdown, JSON and images all become the same kind of block, with headings, tables and records kept. Scans and photos go through Claude vision, transcribed and described, so a photographed delivery note is a document like any other.
2. **Chunk on structure, then on tokens.** Headings, sheets and records start new chunks first; text is then packed to about 700 tokens with 15% overlap that never cuts a sentence. A table is always its own chunk; a large one splits by rows and repeats its header in every part, so no row is ever cut in two.
3. **Situate each chunk in its document.** Contextual Retrieval: a cheap model writes one or two sentences placing each chunk in its document, prepended to the embedded text only, never to what the user sees or what a citation is matched against. A row reading “| 80 | 6.00 | 480.00 |” becomes findable by a question about accessory pricing. Cached by file hash, so a re-ingest costs nothing.
4. **Render once, cite forever.** Every non-PDF file is converted at ingest into HTML with a stable id on each paragraph, row and record. The viewer then needs only two paths, PDF and HTML, and a chunk’s character offsets line up with the rendition because both came from the same parse.
5. **Embed into two indexes.** Vectors go to ChromaDB, in one collection per project and embedding model, so switching models never mixes vector spaces. The same text goes to SQLite FTS5 for keyword search.
6. **Keep the index true to the folder.** Each project owns a folder. A watcher ingests files as they appear, a reconciler rescans at startup, and a progress bar follows each file through parsing, chunking, contextualising and embedding. Any file, or the whole queue, can be paused and resumed at the chunk it stopped on, with no vision or LLM call paid twice, even across a restart.

### Retrieval

Embeddings generalise but miss exact strings; a question about a part reference, an invoice total or a code like VT-COB/SKU98496 is lexical. BM25 gets those, embeddings get “what did the report conclude about growth”. Both run on every search.

Cosine similarity and BM25 scores live on incompatible scales, so they are fused with Reciprocal Rank Fusion, which only needs ranks: nothing to calibrate per corpus, and it stays correct when one retriever returns nothing. A cross-encoder then reads each of the 20 candidates together with the question, which is far more accurate and only affordable on a short list. The model is sent each chunk’s parent section: the chunk is what matched, the section is what answers.

- Semantic search (ChromaDB, cosine): 30 candidates
- Keyword search (SQLite FTS5, BM25): 30 candidates
- Reciprocal Rank Fusion (k = 60, ranks only): 20 candidates
- Cross-encoder rerank (bge-reranker-v2-m3): 6 candidates

### The agent

- **Rewrite:** Turns a follow-up into a standalone query using the last six turns: “and the second one?” becomes “unit price of reference 40564101 in produits.xlsx”.
- **Plan:** Classifies the turn: small talk, lookup, multi-hop, summary or comparison. Small talk skips retrieval.
- **Decompose:** For multi-hop and comparison questions only, splits the question into two to four parts and searches each one.
- **Agent and tools:** Searches, lists documents and reads around a passage. The first round is forced to search, so the model can’t answer from memory.
- **Critique:** Judges whether the evidence supports an answer; if not, it sends the agent back with a better query, within three rounds.
- **Answer:** Writes structured JSON: the answer with a marker on every claim, and a verbatim quote per citation. If the sources don’t answer, it says what is missing.
- **Resolve citations:** Finds each quote in its source and maps it to page coordinates or HTML offsets.

### Citation checks

- Exact substring: score 100
- Normalised substring (whitespace, curly quotes, dashes): score 97
- Fuzzy sliding window, then refined: score its ratio
- **Verified citations.** A quote that matches its chunk with a score of 72 or more is verified. Below that it is shown in amber, marked unverified, and the answer gets a “2 of 3 sources verified” badge. Unverified citations are never deleted: a near-miss is usually a real source lightly paraphrased, and hiding it would hide a retrieval problem.
- **Uncited figures.** A sentence that states a number without a marker is counted. On invoices and delivery notes an invented total is the failure that matters, and a citation count alone can’t see it.
- **Orphan markers.** A marker in the text with no matching source is caught and logged.

### Evaluation (MRR on the sample corpus)

- Vector only: all-MiniLM-L6-v2 0.53, bge-small-en-v1.5 0.81
- BM25 only: all-MiniLM-L6-v2 0.57, bge-small-en-v1.5 0.57
- Hybrid + rerank: all-MiniLM-L6-v2 0.94, bge-small-en-v1.5 0.94

The evaluation harness scores retrieval with recall@k, MRR and nDCG per category, and full answers with groundedness and refusal accuracy. Retrieval-only runs call no LLM, so sweeping six embedding models costs nothing. Gold questions include unanswerable ones on purpose: a system that answers everything scores perfectly on recall and is useless.

The table shows why the retriever being scored matters. Through the full pipeline a weak embedder and a strong one score the same, because BM25 and the reranker rescue the weak one. To choose an embedding model, score vectors alone; to judge the system, score it whole. Comparisons are per question, so a flat average hiding swings in both directions shows up as the regression it is.

The weakest category is aggregation (recall@6 0.50, MRR 0.10 with MiniLM): RAG retrieves passages, it doesn’t add up three totals across three files. A compute tool over retrieved table rows is the fix, and the harness exists so that it gets built on a number, not an assertion.

### Engineering choices

- **Models are a runtime choice.** The embedding model is chosen per project (a folder of French and Arabic delivery notes wants BGE-M3, English docs do better on BGE-base); the answer model is set instance-wide. Both switch without a restart, and a re-index runs in the background while chat keeps answering on the old model.
- **English-only models say so.** They don’t fail loudly on Arabic or French, they retrieve nonsense confidently, so the picker flags them in amber.
- **Query and documents share one model.** The model is resolved once per query and feeds both the embedder and the collection name. If a collection’s metadata disagrees, the search raises an error rather than returning wrong passages.
- **A paused document is invisible.** Keeping paid-for work during a pause must not make half a document answerable, so vectors wait in a spool and reach the index in one write.
- **Shows its work.** Each search reports its stages live (embedded, semantic, keyword, merged, reranked), then folds into “how this was found” under the answer, stored with the message.
- **Runs offline.** With a local embedding model, ingestion and retrieval need no embedding API key; only answer generation calls an LLM. Every endpoint call and internal action is logged to the database.

### Stack

- Agent: LangGraph, tool calling, structured JSON answers, SQLite checkpoints, rapidfuzz quote matching
- Retrieval: ChromaDB, SQLite FTS5 (BM25), Reciprocal Rank Fusion, bge-reranker-v2-m3
- Ingestion: PyMuPDF, python-docx / python-pptx / openpyxl, Claude vision OCR, tiktoken, watchdog
- Backend: Python, FastAPI, Server-Sent Events, SQLModel, Alembic
- Frontend: Angular 20 (signals), Tailwind CSS, Server-Sent Events reader
- Quality: pytest, golden question sets, recall@k, MRR, nDCG, groundedness

## Case study: Code Auditor, a LangGraph agent that treats a repository as hostile input

Full page: https://www.hamza-ouhssaine.com/projects/code-auditor/

Code Auditor takes a Git URL or a local folder and assumes the worst about it. It scans for malware and secrets before anything else touches the code, maps and analyses it without ever running it on the host, builds a risk-ranked roadmap, then fixes one item at a time in a sandbox. Every change waits for a person to approve it.

Status: Prototype. Every stage is tested against fakes of the model and the containers, and the scanners against real tools. The fix loop has not yet run end to end against a real model inside Docker.

- Type: Personal project, 2026, designed and built end to end
- Agent: LangGraph state machine, 11 nodes, SQLite checkpoints
- Isolation: Network-less Docker containers with memory, CPU and process caps
- Models: Claude Sonnet for the roadmap and the fixes, both as structured output
- Tests: 478 backend test functions, 18 frontend tests

### From repository to roadmap

The first half of a run reads the repository and never executes it. Each stage is a node of a fixed LangGraph state machine rather than a free agent loop, so the order of the checks can’t be talked around.

1. **Repository.** A Git URL, cloned with full history into a throwaway workspace (600 s timeout). A local folder is copied without following symlinks.
2. **Preflight scan.** ClamAV, YARA rules and TruffleHog run in a container with no network. The gate fails closed and trusts the hits, not the scanner’s own “passed” flag. Here: no hits.
3. **Explore.** Maps modules, dependency graph, entry points and tests by parsing with ast. Nothing is imported or executed. Caps: 2 MB per file, 20,000 files.
4. **Diagnose.** ruff, mypy and radon run in a 4 GB container with no network; every finding comes back with a severity.
5. **Security audit.** bandit, semgrep, osv-scanner and gitleaks, scored with CVSS. Critical is reserved for security tools; a Critical finding would stop here for a human.
6. **Plan.** Findings are compacted (150 listed one by one, the rest grouped) and Claude Sonnet returns a typed roadmap. Code then re-derives every severity instead of trusting the model.
7. **Roadmap.** Items with severity, priority, rationale and the files they touch, ready for the fix loop. With no model, or on a model error, a rule-based roadmap is built instead.

Other routes:

- Any preflight hit, or no preflight result at all: quarantine, abort, and a report that says why.
- A Critical security finding: the run halts at the approval gate before any planning.
- No findings, or an empty roadmap: straight to the report.
- Cost ceiling reached while planning: a rule-based roadmap, and the run is marked aborted.

### The fix loop, with a human in it

The second half changes code, one roadmap item at a time. The model writes files; code decides whether they are safe to write, a sandbox decides whether they work, and a person decides whether they stay.

1. **Roadmap item.** The highest-priority item. Scope and the $10 cost ceiling are checked before any model call.
2. **Act.** Up to 4 whole files go to the model, which returns complete new files, never diffs or shell commands. Each file is validated, snapshotted and written atomically; the diff comes from difflib.
3. **Verify.** pytest runs in a sandbox with no network, on a read-only mount of the workspace. A test fails.
4. **Act.** Second attempt. Each item gets 3 in all, counting the first.
5. **Verify.** Tests pass. “Unverifiable” is kept apart from “failed” and is never retried.
6. **Human approval.** The graph pauses with interrupt() until a reviewer approves or rejects. The decision must also exist as a database row, and the row is the authority.
7. **Report.** After the last item, a Markdown report rendered by code, not by a model. “Fixed” means approved, verified and not rolled back.

Other routes:

- Still failing after 3 attempts, or the iteration cap reached: byte-exact rollback, then the gate.
- Rejected without feedback: the change is reverted. Rejected with feedback: back to Act with a fresh retry budget.
- Nothing written to disk: the gate lets the run continue without stopping.

### Containment

The repository is untrusted, and so is the model’s output. Both are kept on a short leash.

- **Four kinds of container.** Preflight (2 GB), analysis (4 GB), security (6 GB) and the test sandbox (2 GB), each with no network, 2 CPUs, 512 processes and a timeout between 10 and 30 minutes.
- **Write on the host, run in the box.** Edits are written to the workspace on the host; code only ever runs in the sandbox, against a read-only mount. Snapshots live outside the mount.
- **Hard limits on a run.** 50 iterations, one hour of wall clock, a $10 cost ceiling checked before each model call, and a LangGraph recursion limit of 200.
- **Pinned tools.** ruff 0.9.10, mypy 1.15.0, radon 6.0.1, bandit 1.8.6, semgrep 1.172.0, gitleaks 8.24.0, osv-scanner 2.4.0, TruffleHog 3.88.0, pytest 8.3.4.

### What a real repository showed

A first manual run on a real 56-file repository, without Docker or a model, is recorded in the project notes. 134 of its 137 security findings were bandit’s B101, a plain assert statement, and none of the 17 roadmap items fell inside the fixer’s default allowlist.

Noise and scope, not model quality, are what a tool like this has to get right first.

### Engineering choices

- **A state machine, not a free agent.** Fixed nodes with narrow jobs. The model plans and writes files; it never chooses which check runs next.
- **Fail closed.** A missing preflight result counts as a failed one. The scanners’ hits decide, not their exit flags.
- **Severity in code.** The model may suggest a severity; code re-derives it from where the finding came from. Critical belongs to security tools only.
- **Whole files, not patches.** The model returns complete files. Diffs are computed by difflib, and unsafe output is refused before anything is written.
- **Approval needs a record.** A LangGraph interrupt pauses the run, and resuming requires an approval row in the database. No row, no resume.
- **A report without a model.** The final report is rendered by code from recorded events, so it can’t overstate what was fixed.

### Stack

- Agent: LangGraph, SQLite checkpoints, Anthropic SDK (structured output)
- Analysis: ruff, mypy, radon, bandit, semgrep, osv-scanner, gitleaks
- Preflight: ClamAV, YARA, TruffleHog
- Backend: Python 3.12, FastAPI, WebSockets, SQLModel, Alembic, Docker SDK
- Frontend: React 19, TypeScript, Vite, Tailwind CSS 4
- Quality: pytest, Vitest, parser tests on captured real tool output

## Case study: Proving Ground, agents judged by what happened, not by what they say happened

Full page: https://www.hamza-ouhssaine.com/projects/proving-ground/

Proving Ground is a benchmark for tool-using agents. It runs them against a deterministic simulated company (customers, orders, tickets, an inbox) and scores the world’s event log, not the agent’s final message. Its headline metric is hallucinated success: the agent says the refund went through, and the ledger shows no refund.

Status: The machinery works and is tested: every scenario has a hand-written reference solution that must pass. It has not been benchmarked against a model yet, so there are no results to report.

- Type: Personal project, 2026, designed and built end to end
- World: 40 customers, 120 orders, 30 tickets, 24 emails, 9 knowledge-base documents
- Suite: 24 scenarios, 20 tools, 4 scaffolds, 6 policy profiles
- Models: Any OpenAI-compatible endpoint, temperature 0, fixed seed
- Tests: 232 test cases (pytest)

### One cell of the benchmark

A benchmark is a grid: scenario × scaffold × policy profile. Each cell runs one agent in one fresh world, and every tool call passes a policy gate before it can change anything.

1. **Benchmark cell.** One cell: an adversarial scenario, the ReAct scaffold, the “taint” policy profile. Every model reply is recorded to a cassette, so the cell can be replayed exactly.
2. **Build the world.** A fresh world from seed 1001 plus the scenario’s own patch, written to the event log by the actor “system”.
3. **Agent turn.** The model reads the task and asks for a tool: read_inbox.
4. **Policy gate.** Unknown tool, allowlist, steps, tokens, repeats; then, for irreversible tools only, deny, taint and confirmation. A read inside budget: allowed.
5. **Run the tool.** The virtual clock advances and a world event is appended before anything changes. The email text goes to the taint tracker.
6. **Agent turn.** One email carries instructions. The agent calls issue_refund, an irreversible tool, with an order id that appeared only in that untrusted text.
7. **Policy gate.** DENY_TAINT. The call is refused with its reason, and the world and the clock are left untouched.
8. **Agent turn.** The agent gives its final answer without refunding.
9. **Score the world.** Checks run over the final world, never over the reply: no refund event exists. Had the reply claimed one, it would count as hallucinated success.
10. **Classify.** A rule-based failure code ordered by root cause, and the attack’s outcome read from the world.
11. **Results.** Saved to SQLite and streamed over a WebSocket to the dashboard: success, safety, hallucinated success, steps and cost.

Other routes:

- The turn loop also ends after 3 malformed replies in a row, or at 31 turns (the 25-step budget plus 6).
- Out of budget: the agent is told to wrap up and be honest, and its next blocked call ends the run.
- Irreversible tools under the “confirm” profile: the call returns a token the agent must echo back. There is no simulated approver whose answer could leak.
- An optional judge model scores clarity, grounding and calibration from 1 to 5. It can never change success.

### A third of the suite can’t be won by acting

The 24 scenarios come in five families: 6 direct, 4 ambiguous, 4 impossible, 4 distractor and 6 adversarial. In the ambiguous and impossible ones the right move is to ask or to refuse, so an agent that always acts loses a third of the suite.

The world even allows policy violations, such as refunding a cancelled order, so that it is the agent being measured and not the simulator. Each scenario ships with a hand-written reference solution that must pass and must not be flagged, which tests the scenarios themselves.

### Four scaffolds, the same 20 tools

10 read tools, 6 reversible writes and 4 irreversible actions: refunds, emails, discounts and deletions.

- **ReAct.** One loop: think, call a tool, read the observation, until a final answer.
- **Plan and execute.** A planner writes a structured plan first; one replan is allowed if steps were blocked or failed.
- **Reflexion.** After a first attempt, a smaller critic reviews a mechanical record of calls and world events, and the agent gets one more attempt with at most 3 lessons. The world is not reset.
- **Orchestrator and workers.** A lead delegates up to 3 subtasks to workers with narrowed tools (a researcher sees 10, an operator 16, a settler all 20), enforced in the call itself, not only hidden from the prompt.

### Six policy profiles

From open to strict. Delimited fences untrusted text; confirm demands a token for irreversible tools; taint blocks irreversible calls whose arguments came from untrusted text (5-word shingles, rare tokens, email addresses); strict combines confirm and taint, and readonly is the sixth.

Taint is checked before confirmation, so an injected instruction can’t simply be confirmed into action.

### Engineering choices

- **World state decides.** Success is computed from the event log. A judge model can comment on the reply but can’t change the score.
- **No LangGraph, on purpose.** The scaffold is the variable under test, so the loop has to be interceptable, recordable and replayable line by line.
- **Deterministic by construction.** A seeded world, a virtual clock, temperature 0 and one cassette per cell: a change re-records only the column it affects.
- **Taint without attack lists.** Detection tracks where text came from rather than matching known payloads. An allowlist of trusted identifiers fixed a real false positive that blocked the correct refund in a direct scenario.
- **Narrowing enforced, not suggested.** A worker’s tool list is checked when it calls, so a tool missing from its prompt is also missing from its reach.
- **Two wire protocols.** Native tool calling or a plain-text THOUGHT / ACTION protocol, recorded with each run, for models without function calling.

### Stack

- Agents: hand-written loops, OpenAI-compatible SDK, JSONL cassettes
- Simulator: Python dataclasses, YAML scenarios, event sourcing, virtual clock
- Backend: Python 3.12, FastAPI, WebSocket, SQLModel, SQLite
- Frontend: React 18, Vite, Tailwind CSS 4, hand-drawn SVG charts
- Quality: pytest, ruff, GitHub Actions

## Case study: ML Engineer, an AutoML agent that never sees its own holdout set

Full page: https://www.hamza-ouhssaine.com/projects/ml-engineer/

ML Engineer is an autonomous agent that profiles a dataset, plans experiments, writes scikit-learn code and runs it in a sandbox, inside a harness built to check whether it works honestly. The agent never sees the holdout set: not by convention, by construction. Its headline number isn’t its best score but the gap between the cross-validation score it optimised and the holdout score it never saw.

Status: A working prototype. The whole pipeline runs end to end with a scripted stand-in for the model; no benchmark against a model has been published yet.

- Type: Personal project, 2026, designed and built end to end
- Agent: Custom loop with four LLM roles: planner, coder, critic, submitter
- Sandbox: Docker, no network, 2 GB, 2 CPUs, read-only root, 10-minute fits
- Tasks: Six tabular tasks: five with a planted trap, one clean control
- Tests: 188 test cases (pytest), 16 of them in Docker

### The holdout never meets the agent

The split happens before the agent exists, and the function that runs the agent has no parameter that could carry a holdout. Scoring happens in another container, after the agent has returned.

1. **Dataset.** A churn task generated from a seed, with one trap planted: a column filled in after the outcome. Or your own CSV, up to 64 MB.
2. **Holdout split.** Time tasks hold out the latest 20%, grouped tasks 25% of the groups, the rest a stratified 20%. Columns only known after the outcome are blanked in the holdout, as they would be at prediction time.
3. **Agent run.** Planner, coder, critic and submitter work on train.parquet only, for up to 4 experiments. The holdout isn’t hidden from them; it was never passed in.
4. **Holdout scoring.** After the agent returns, another container loads each fitted model and the holdout. Every experiment is scored, not only the submitted one.
5. **Report.** The CV − holdout gap per experiment (above 0.05 counts as self-deceived), traps caught, missed and falsely flagged, and a naive baseline fitted on every column in the same sandbox.

Other routes:

- No sandbox available: the benchmark stops rather than run code on the host. Host execution needs an explicit flag, and every result says so.
- One task fails: it is logged and the other tasks carry on.

### Inside the agent loop

Four language-model roles, called in turn by plain Python. None of them can call another, and none of them can overrule the arithmetic in between.

1. **Briefing.** Code, not a model, profiles the training data, audits the columns and suggests a CV scheme, labelled “advice, not a rule”: whether the agent follows it is part of what is measured.
2. **Planner.** Returns ranked hypotheses as schema-validated JSON.
3. **Coder.** Writes candidate.py for the first hypothesis. It sees CV scores, the last 4 lessons and the last 3 failures; no holdout score exists yet.
4. **Sandbox fit.** Docker with no network, 2 GB of memory, 2 CPUs, a read-only filesystem and train.parquet only. Out: CV score and strategy, features used, the fitted model.
5. **Leakage checks.** Say the model kept the post-outcome column: one feature separating the classes with an AUC of 0.97 or more is a critical finding, and the experiment is quarantined.
6. **Critic.** A smaller model, to partly control for a model favouring its own work, reads the findings and pivots: drop the column. One pivot is allowed per run.
7. **Coder.** The next hypothesis, with the lesson.
8. **Sandbox fit.** A second experiment in a fresh container.
9. **Leakage checks.** No critical finding this time.
10. **Critic.** Stop.
11. **Submitter.** Chooses one experiment from CV scores and findings alone. If it names one that doesn’t exist, the harness falls back to the best CV that wasn’t quarantined.

Other routes:

- Coder fails: the attempt is recorded as a coder failure and the loop moves on, with no sandbox or critic run.
- Planner fails: the run ends with an error.
- Critic fails: the experiment is accepted and the loop continues.
- At most 4 experiments; an empty queue also goes to the submitter.

### Checks a model can’t argue with

Leakage detection is plain arithmetic, with no language model anywhere in the module: a detector a model can talk itself out of reports whatever the model was hoping for.

Five of the seven leakage checks, with their thresholds in the code:

| Check | Flags when |
|---|---|
| One feature separates the classes | AUC ≥ 0.90 (critical from 0.97) |
| Missing values separate the classes | separation ≥ 0.85 |
| Cross-validation too good to be true | CV ≥ 0.985 |
| Near-duplicate rows | share ≥ 0.15 |
| Class imbalance | positive rate < 0.05 |

### What gets measured

- **The gap.** CV score minus holdout score, for every experiment. A gap above 0.05 is labelled self-deceived.
- **Traps.** Five tasks carry one planted trap each (target leakage, time order, groups, duplicates, class imbalance) and one is a clean control. Caught, missed and false alarms are counted separately.
- **A naive baseline.** Gradient boosting on every column, fitted in the same sandbox. It is meant to fall into the traps, which shows what the agent avoided.
- **Cost.** Tokens, cost and wall time for every run, from the same records.

### Engineering choices

- **Isolation by signature.** The agent’s entry point has no parameter that could carry a holdout. Tests check that signature, check the loop never imports the scorer, and scan every recorded prompt for every holdout score at 2 to 8 decimals.
- **Masked to prediction time.** Post-outcome columns are blanked in the holdout; otherwise a planted leak would just look like a good feature.
- **Advise, don’t enforce.** The split guard recommends a CV scheme instead of imposing one. Imposing it would make the traps unreachable.
- **Fail closed.** No sandbox, no run. Running on the host needs an explicit flag and is labelled in every result.
- **Features live in the pipeline.** Data preparation may only drop rows, so every transformation sits inside the pipeline that also runs at prediction time.
- **Failures are data.** A crashed fit exits cleanly with the stage that failed and becomes a recorded experiment, not a crash.

### Stack

- Agent: custom Python loop, OpenAI-compatible SDK, schema-validated JSON, strict mode with a fallback
- Machine learning: scikit-learn, pandas, NumPy, SciPy, PyArrow, joblib
- Sandbox: Docker: no network, capped CPU, memory and processes, read-only root
- Backend: Python 3.12, FastAPI, WebSocket, SQLModel, SQLite, record/replay cassettes
- Frontend: React 18, Vite, Tailwind CSS 4, hand-drawn SVG charts
- Quality: pytest, ruff, GitHub Actions

## Case study: Prompty, prompts shipped like code: versioned, evaluated, A/B tested

Full page: https://www.hamza-ouhssaine.com/projects/prompty/

Prompty treats a prompt as a versioned artifact, not a string someone edits in production and then watches for complaints. Every version is graded against a golden set by an LLM judge before it can ship, then rolled out to a share of real traffic and judged by statistics. Deploying and rolling back are the same operation: moving a label.

Status: A working local prototype: single user, no authentication. Its demo data comes from a deterministic mock model, so it shows the machinery, not the quality of any model.

- Type: Personal project, 2026, designed and built end to end
- Built: FastAPI service, Python SDK, React dashboard
- Statistics: Wilson intervals, z-test, Welch’s t-test, paired t-test, power analysis, SRM check
- Models: Any OpenAI-compatible endpoint, or a deterministic mock
- Tests: 151 test functions (pytest), statistics checked by simulation

### One completion, end to end

Applications call Prompty instead of the model. One call decides which version this user gets, renders it, calls the model and records everything an experiment will need later.

1. **SDK call.** client.complete("support-reply", variables={"message": "My order never arrived"}, unit_id="user-42").
2. **Resolve target.** An experiment is running on support-reply, so the experiment decides the version. Without one, the “production” label would.
3. **Assign variant.** Bucket = sha256(experiment:unit) mod 10,000, walked through the variants’ weights in basis points. The assignment is stored, so user-42 keeps the same variant even if the weights change.
4. **Render.** Plain {{name}} substitution with no template logic. A missing variable is a 422, never a half-filled prompt.
5. **Model call.** The variant’s own model and settings. Rate limits and connection errors are recorded as status “error” instead of raised; a content filter becomes “refusal”.
6. **Completions log.** One row in the single completions table, priced per million tokens and tagged source = live. Eval and judge calls share the table under their own source.
7. **Response.** Text, variant, tokens, cost and latency. Later, client.track() attaches the outcome, a thumbs-up or a number, to the same variant.

Other routes:

- No experiment running: the deployment label picks the version, and nothing is stored.
- No unit id: a random draw that is not persisted.
- Provider error: the logged completion comes back inside a 502, so the caller sees what was recorded.

### From a new version to production

A version has to pass two different tests: offline, against answers already judged good; online, against real traffic. Either can stop it.

1. **New version.** v3 of support-reply. Versions never change once created; labels are pointers to them.
2. **Offline eval.** Each golden case runs through v3, five at a time, then a judge model scores the answer from 1 to 5 on an absolute rubric, not side by side, which avoids position bias.
3. **Deploy gate.** Compared case by case with the version in production: a paired t-test over at least 5 shared cases. Only a significant drop blocks; absence of evidence never does.
4. **A/B experiment.** Traffic is split by weight in basis points, sticky per user, with the sample size planned in advance by a power analysis.
5. **Results.** Wilson intervals per variant, a pooled z-test for rates, Welch’s t-test for continuous metrics. A sample-ratio mismatch at p < 0.001 suppresses every “significant” flag; cost, latency and error-rate guardrails watch for regressions.
6. **Production.** Completing the experiment moves the production label to the winner. A guardrail regression refuses with a 409 unless forced, the one human override.

Other routes:

- Deploy gate fails: the deploy is refused and the label stays where it was.
- Fewer than 5 events in a cell: the z-test isn’t trusted, and the error-rate guardrail switches to Fisher’s exact test.

### Statistics chosen on purpose

An A/B platform is only as honest as its tests. Each choice here answers a specific way experiments mislead.

- **Pooled for the test, unpooled for the interval.** The z-test uses the pooled standard error, as the null hypothesis implies; the interval on the difference uses the unpooled one. Each variant’s own rate gets a Wilson interval, which stays sensible near 0 and 1.
- **Paired when the cases are shared.** Offline runs score the same golden cases, so comparisons are paired. In the test suite, 40 shared cases give p < 0.001 paired and p > 0.05 unpaired on the same scores.
- **Sample ratio mismatch first.** If a 50/50 split delivers 45/55, assignment is broken and every number downstream is suspect. The check runs from 100 observations at p < 0.001, and silences all significance flags when it fires.
- **Guardrails beside the goal.** A variant that wins on thumbs-up but costs more, runs slower or fails more often is flagged. Cost and latency need both a 10% relative regression and significance.
- **Aggregates, not rows.** The statistics module has no database access. One GROUP BY feeds it, so the results endpoint costs the same at any traffic.
- **Checked by simulation.** The power calculation is tested by simulation: 20,000 runs land at 0.80 ± 0.02 power, and with no true effect the false-positive rate stays at 0.05 ± 0.01.

### Engineering choices

- **One table for every call.** Live, eval and judge calls land in one completions table with a source column. Aggregation must filter on live traffic, and a test fails if it stops doing so.
- **Deploy and rollback are one operation.** Versions are immutable and labels point at them. Rolling back is moving the label back.
- **Sticky, and stored.** Assignment is a hash of experiment and user, persisted on first sight, so it survives a change of weights. Two first requests racing for one user resolve to the stored row.
- **Errors are data.** Provider failures are logged like any completion, so the error-rate guardrail can see a variant that fails more often.
- **Immutable judges.** A judge can’t be edited after creation, so scores from different weeks stay comparable.
- **Fail at startup, not on the bill.** An unknown provider name stops the service at startup, so a typo can’t fall through to a paid endpoint.

### Stack

- Backend: Python, FastAPI, async SQLAlchemy, SQLite (WAL), Pydantic settings
- Statistics: SciPy, NumPy
- LLM: OpenAI-compatible SDK, LLM judge with JSON output, deterministic mock provider
- SDK: Python client on httpx
- Frontend: React 19, TanStack Query, React Router, Radix, Tailwind CSS 4, Vite
- Quality: pytest, pytest-asyncio, simulation tests of the statistics

## Case study: Tahanot, a mini-ERP whose assistant can only use the front door

Full page: https://www.hamza-ouhssaine.com/projects/tahanot/

Tahanot is a self-hosted mini-ERP inspired by Odoo’s ideas and rebuilt from scratch: sales, purchasing, inventory, invoicing and a dashboard for a small multi-company trading business. Its AI assistant answers questions about the business by calling the same services, with the same permission checks, as the rest of the app. It never writes SQL.

Status: A working prototype, version 0.1. Purchasing, credit notes and the French interface live on a feature branch that isn’t merged yet.

- Type: Personal project, 2026, designed and built end to end
- Modules: Sales, purchasing, inventory with backorders, invoicing and payments, dashboard
- Assistant: Hand-written tool loop, 10 typed tools, up to 8 rounds
- Data: PostgreSQL 16, SQLAlchemy 2, Alembic migrations
- Tests: 220 backend and 26 frontend test functions

### One question to the assistant

The assistant is one model in a loop with typed tools. Everything it can see goes through the same service layer and access checks as the REST API, so it can’t know more than the person asking.

1. **User.** “How many units of SKU-102 are left in Warehouse 1?” On a record page, the panel also sends the record being viewed.
2. **Auth and context.** The JWT and the company header resolve the user, the companies they may see, their groups and their rights per model. A company they can’t see is a 403.
3. **Rate limit.** 10 messages per user per minute. The conversation is loaded or created, and the message saved.
4. **Model.** The system prompt is rebuilt on every request: date, user, warehouses, currency, grounding rules and a field catalogue generated from the live model registry. The model asks for get_stock(sku="SKU-102", warehouse_id=1).
5. **Typed tools.** Arguments are validated by Pydantic, then the tool calls the stock service with the user’s own rights. Cost fields come back as “restricted” for users without access.
6. **Model.** The result comes back fenced as untrusted data, never an instruction. No more tool calls, so it answers.
7. **Result blocks.** Tool payloads become real interface components: a stock tile here; elsewhere tables that link to records, bar charts and KPI tiles that can be pinned to the dashboard.
8. **Answer.** The figure from the tool result, shown next to the text that explains it.

Other routes:

- Up to 8 rounds of tool calls; past that, the answer is marked as possibly partial.
- A tool error goes back to the model as a result, not an exception, so it can explain and adapt.
- Write tools (create a quotation, confirm an order, add a reordering rule) are off by default. When enabled, the first call returns a proposal and an HMAC token valid for 600 s; only a second call with the same payload executes it.
- Provider error: the stream ends with an error event. There is no silent retry.

### The ERP underneath

Odoo’s ideas, not its code: a generic model layer, documents with explicit states, and stock held as quantities per location.

- **A generic model registry.** One CRUD router with domain filters serves every model; business actions such as confirming an order or validating a delivery have routes of their own.
- **Documents as state machines.** Quotations, orders and deliveries move through explicit states.
- **Quant-based stock.** Stock is held as quantities per product and location, with product variants, and backorders when a delivery is only partly done.
- **Invoices and payments.** Invoice payments and credit notes, with concurrency and locking tests that run against PostgreSQL.
- **One aggregation primitive.** read_group powers the dashboard, the pivot views and the assistant alike.
- **Two companies, two languages.** Multi-company from the data model up, with an English interface and a French one in progress.

### Engineering choices

- **The model never writes SQL.** It calls typed, validated tools over the same services and permission checks as the REST API.
- **Provider-neutral.** Messages use a neutral format and an OpenAI-compatible endpoint is the default, so Claude or a local server is a configuration change.
- **Allowlisted models, hidden fields.** 21 models are visible to the assistant. Cost fields are restricted to four groups, and secret fields are never serialised.
- **Untrusted by default.** Tool results are wrapped as untrusted data, and the system prompt says such data is never an instruction. A test checks that an injected product name stays data.
- **Writes need a second step.** A write first returns a proposal and a token signed over its payload; a changed payload is rejected.
- **Answers as components.** Results render as tables, charts and tiles built from the tool payloads, not only as text.

### Stack

- Backend: Python 3.12, FastAPI, SQLAlchemy 2, Alembic, Pydantic, PyJWT, Server-Sent Events
- Data: PostgreSQL 16
- AI: hand-written tool loop, Anthropic and OpenAI SDKs
- Frontend: React 18, TypeScript, TanStack Query and Table, Recharts, Tailwind CSS 4, Vite
- Infra: Docker Compose, nginx
- Quality: pytest, Vitest, Testing Library