[{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/claude-code/","section":"Tags","summary":"","title":"Claude Code","type":"tags"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/fastapi/","section":"Tags","summary":"","title":"Fastapi","type":"tags"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/pwa/","section":"Tags","summary":"","title":"Pwa","type":"tags"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/react/","section":"Tags","summary":"","title":"React","type":"tags"},{"content":" Knowing where the household stands in the face of a water cut, a storm or an evacuation: what is ready, what is missing, what has expired. The app runs on the phone, without a network, and syncs with a PC at home. StatusIn development · offline v2 shipped, catalogue reviewed at each council TimelineSince November 2026 RoleSolo: specification, content framework, development Stack Python 3.13FastAPISQLModelSQLiteReactTypeScriptVitePWAIndexedDBGSAPTailscale Source code 279\ngoals\nacross 14 areas and 9 tiers, written in YAML 2\nscoring engines\nPython and TypeScript, kept identical by shared test cases 98\nautomated tests\n57 on the server, 41 in the app Transparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of this page. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nWork in progress. This page describes the repository as of 9 November 2026, one week after the first commit (9 merged pull requests). The figures are counted in the catalogue and the tests, and will be updated as the project moves on. Every screenshot shows a fictional household (Camille, Sam, their daughter Léa and Moka the cat), entered on a demo instance. The app only exists in French, so the screenshots show its French interface. The problem # Official preparedness guides (government, Red Cross) say what to do: a water supply, a first-aid kit, a go-bag, contacts on paper. They do not say where my household stands, or what has stopped being true since: the bottles that expire, the power bank never recharged, the first-aid course from three years ago, the baby who has arrived in the meantime.\nI wanted a tool that:\nmeasures preparedness with checkable criteria (\u0026ldquo;14 days of water for this household\u0026rdquo;), not with a feeling; regresses when reality changes: an expired supply no longer counts, a skill not practised expires; fits the household: a 6th-floor flat does not have the same priorities as an isolated house, nor does a household with a child or a hard-to-refill prescription; works without a network, since the day the network is down is precisely the day it is needed; keeps the data at home: the household\u0026rsquo;s members, their health conditions and where their equipment is stored have no business on a third-party server. Where the project stands # v1 · The engine and the catalogue 2 November A specification drawn from an interview A specification written from an interview, then a YAML catalogue (levels, areas, goals, needs, questionnaire), a level engine in pure Python, a FastAPI API and a React interface with the column drawn in SVG. v2 · Offline on the phone 2 November The engine ported to TypeScript An installable app (PWA), an operation log in the browser, sync with the PC's server over Tailscale. The Python engine remains the reference; its TypeScript copy replays the same test cases. Four review councils 6 to 9 November The catalogue tested against three viewpoints At each council, three agents review the app and the catalogue, each from a different angle, then debate; I approve the plan before it is implemented. They brought the Crisis screen, the 30-day grace period, the Morale, Self-defence and Cybersecurity areas, physical fitness, and corrected content that was wrong or dangerous. Card inventory and skills 8 and 9 November 99 items, 117 talents The inventory becomes a deck of cards with five grades each and a quantified overview; skills become nine talent trees. What the app does # A monument with nine tiers # The household\u0026rsquo;s level is a monument with nine tiers, a Roman column by default. Each tier stands for a resilience horizon:\nLevel Name What the household can withstand 1 Initiate A few hours of outage: 24 h of water and food, a light, the emergency numbers 2 Aware Knows its risks, has its papers, its contacts and a basic plan 3 Ready 3 days without running water, electricity or resupply 4 Provident 14 days confined or cut off from the grid 5 Enduring 3 months of partial disruption, an emergency savings fund 6 Seasoned 6 months of supplies, practical skills kept up 7 Self-reliant Produces part of its own water, food and energy 8 Resilient Part of a mutual-aid community, passes on what it knows 9 Antifragile Everything is tested for real, each crisis improves the plan The tiers are blocking: level 4 is reached only once every required goal of levels 1 to 4 is validated, in every area. A household far ahead on water but with no evacuation plan stays at the tier of its weakest link, and the home screen says which one.\nThe fictional household is at level 3, with 22 of the 38 goals of level 4. Drum III is cracked: a validation expires within the month. On the right, the three most useful actions. Level 9 cannot be bought: it requires testing the household (7 days without mains water, 72 hours with the main breaker off, a real evacuation in under 15 minutes, a surprise drill with one\u0026rsquo;s group), and almost all its criteria must be redone every year.\nThe monument is chosen from fourteen: a pyramid, a tree, the Eiffel Tower, the Great Mosque of Djenné, a moai, a rocket, an undersea volcano… Changing monument changes nothing about the level.\nEach monument has its nine tiers and its crown, which lights up at level 9. A level that can drop # The level is not kept for good. It goes down when:\na supply expires: an expired item immediately stops counting towards self-sufficiency; a validation expires: a skill to practise, a piece of equipment to test, a drill to repeat each have a period of validity; a seasonal deadline passes: firewood stored before November, the chimney swept before October; the household changes: one more member raises the needs and can make new goals apply. So that a single forgotten power bank does not bring down several tiers at once, an expired validation first enters a 30-day grace period: the drum cracks but stays up. Expired supplies get no grace period. That compromise came out of the first council: a supply that is not renewed does not exist, but a sudden drop makes people give up on the tool.\nThe fictional household climbs to level 3, falls back to level 1 when a grandmother on medication joins it (new goals become required), then climbs back up when she leaves. 279 goals that fit the household # The goals are spread over 14 areas: water, food, energy, health, shelter, safety, communication, mobility, skills, finances, community, morale, self-defence and cybersecurity. Each has a level, an explanation (\u0026ldquo;why it matters\u0026rdquo;) and a kind: an action ticked by hand, a skill to revalidate, or a supply computed from the inventory.\nA goal only shows up for the households it concerns:\nwhere they live: a flat does not need a campfire or fuel stored in the cellar, but must know how to use a camping stove without poisoning itself; the members: child, baby, senior, pet; health conditions: twelve in total, from glasses to a prescription that is dispensed in limited amounts. For glasses: a spare pair and a copy of the prescription at level 2, a pair in the go-bag at level 3, two spare pairs and a repair kit at level 5; declared risks: flooding, cold, heatwave, industrial hazard. Physical tests (walking 15 km with a pack, being able to swim) have an adapted variant for households with a member who has reduced mobility: the requirement is about the household\u0026rsquo;s capability, not about each person\u0026rsquo;s fitness.\nFilters by level, area and status. Each goal shows where its validation came from (interview or action) and its date. At sign-up, an adaptive questionnaire of 26 questions describes the household, where it lives now and where it plans to live (\u0026ldquo;a house in the countryside in two years\u0026rdquo;), what is already in place and the target level. The app infers a starting level from it. Goals specific to the planned home are offered separately, in a \u0026ldquo;Prepare for what\u0026rsquo;s next\u0026rdquo; section.\nAn inventory measured in days of self-sufficiency # The inventory answers one question: how many days can the household hold out? Each item has a quantity, an expiry date and a storage place; self-sufficiency is the sum of unexpired items divided by the household\u0026rsquo;s daily need, computed member by member (4 L of water per adult, 2.5 L per child, 0.3 L for a cat; 2,200 kcal per adult).\nEquipment takes the form of 99 cards in ten categories, each with five grades (rustic, common, confirmed, epic, legendary). A card\u0026rsquo;s grade depends on its quality, its options, the quantity, the minimum contents (for a first-aid kit or a bag) or, for a supply, the days of self-sufficiency. A card shows which goals it serves, but owning an item validates nothing: it has to be confirmed.\nOwned cards are in colour, with their grade; the others stay greyed out. Cards that do not concern the household sit behind a \u0026quot;See also\u0026quot; link. The water supply: its grade follows from self-sufficiency (25 days), and the card lists the goals it serves, validated or still to do. The overview sums up the inventory: the weakest link, self-sufficiency per category against the 3-day, 14-day, 3-month and 6-month marks, and a twelve-month projection showing when self-sufficiency drops as items expire, assuming nothing is consumed or bought.\nThe weakest link is cooking fuel (15 days). In the projection, water drops to zero in August 2027, when the packs expire. Nine skill trees # Skills are grouped into nine specialities (physical fitness, healer, handyman, cook, protector, gatherer…), each a talent tree with five ranks and three branches. A talent can be ticked once its prerequisites are acquired, with what proves it (a certificate, a test), a date and a note. As with items, ticking a talent validates no goal.\nThe Physical fitness speciality: three talents acquired at Novice rank, three available at the next rank. A screen for the day itself # The red \u0026ldquo;Crise\u0026rdquo; button stays visible on every page. It opens a screen designed for someone under stress, which works offline and can be printed:\ncall buttons (15, 112, 18, 17, and 114 by text message, the French emergency numbers) and a breathing reminder; folded reflex sheets: first aid (cardiac arrest, bleeding, carbon monoxide, heatstroke…) then situations (water cut, flood, evacuation, intrusion…), filtered by the household\u0026rsquo;s risks and members; the household\u0026rsquo;s own sheets: contacts, meeting points, who does what, medical sheets; where the equipment is, from the inventory, and the drinking-water supply. Emergency numbers first, then the first-aid sheets and the situations that concern this household. On the phone, without a network # The app installs on Android like a native app. The engine, the catalogue and the data live on the phone: in airplane mode, everything works. The PC\u0026rsquo;s server is used to install it, to sync the phone and the computer, and as a backup. It is reachable over Tailscale, in HTTPS, at home or away, with nothing exposed to the Internet.\nLeft, the home screen and its tab bar; right, the Crisis screen with the cardiac-arrest sheet open. How it is built # An operation log rather than a state # Nothing is stored as is: every change is an immutable operation (validate a goal, add an item, change the household), timestamped and identified. The household\u0026rsquo;s state, its level and its whole history are rebuilt by replaying the log in order.\nflowchart LR T[\"PhonePWA + IndexedDB\"] --\u003e|operations| S[\"PC serverFastAPI + SQLite\"] P[\"PC browserPWA + IndexedDB\"] --\u003e|operations| S S --\u003e|other devices' operations| T S --\u003e|other devices' operations| P T --\u003e M1[\"TypeScript enginereplays the log\"] S -.-\u003e|read only| M2[\"Python enginethe reference\"] That choice settles three problems at once:\nsync: each device sends the operations the server does not know and receives the others\u0026rsquo;. Two changes to the same thing are settled by timestamp: the last one written wins; history: the level curve is not recorded but recomputed, including on the dates when a supply expires or a grace period ends, the only moments the level changes without any action; rule changes: when a council moves a goal to another tier, the level is recomputed over the whole log, with no data migration. The one constraint: a goal\u0026rsquo;s identifier is never renamed, since it is stored. The server computes nothing for the app: it keeps the shared copy of the log and serves the files. A JSON backup of the log can be exported from the Household tab.\nTwo engines, one behaviour # To work offline, the level engine has to run on the phone, so in TypeScript. But the Python engine is easier to test and to review. Both exist, and Python is the reference: a script generates test cases from it (households, logs, dates), which the front-end tests replay to check that the TypeScript engine gives exactly the same result. A test fails if the cases were not regenerated after a catalogue change.\nContent outside the code # Goals, daily needs, the questionnaire, item cards, talents and Crisis sheets live in YAML files, not in the code. Adding a goal means adding an entry:\n- id: eau.exemple level: 4 # tier kind: check # check | skill | stock required: true # false: bonus, does not block the tier contexts: [periurbain, rural] revalidate_days: 365 # expires and lowers the level title: … why: … The catalogue is validated at start-up (references, levels, the order of an item\u0026rsquo;s grades), and a test checks that every type of home keeps at least one required goal per tier, so that no household reaches the top through empty tiers.\nA catalogue reviewed in council # The catalogue is the most delicate part: a misplaced criterion discourages, wrong content can be dangerous. I had it reviewed by a council of three AI agents, each with an angle: the Tracker (self-reliance in rural settings), the Tactician (urban crisis and logistics) and the Sentinel (health, vulnerable people, psychology). Each analyses alone, without knowing the others\u0026rsquo; views; the main agent checks their findings in the code, leads the debate and writes a summary that I approve or not. Each council has its dated report in the repository.\nA few corrections that came out of it:\nFinding Decision A campfire required in a flat Fire and fuel kept for suburban and rural homes, a safe stove in town Physical tests blocked a household with a member of reduced mobility for good An adapted variant, the requirement stays for the others Level 3 had 27 required criteria, several unrelated to \u0026ldquo;holding out 72 h\u0026rdquo; Cut to 19, the rest moved to level 4 A PSC1 first-aid course required at level 3, to be booked weeks ahead \u0026ldquo;Gestes qui sauvent\u0026rdquo; (2 h, free) at level 3, PSC1 at level 4 Supplies estimated at sign-up never expired They expire after six months unless confirmed Silver tablets given to purify water, an FFP2 mask against smoke Chlorine or chlorine dioxide for water; FFP2 stops particles, not CO or gases Known limitation: these reviews are done by AI agents, not by first-aiders, doctors or lawyers. The first-aid sheets repeat publicly taught gestures and point to the official courses (Gestes qui sauvent, PSC1), but the catalogue has not been validated by a professional. The app helps get organised; it replaces neither training nor the authorities\u0026rsquo; instructions. What\u0026rsquo;s next # The 9 November council is being implemented: showing the weapon, alcohol and tobacco cards only to households that declare them (they still show for everyone), removing duplicate cards that offer the same goal twice, and correcting some content. The next leads are revalidation reminders, a structured medical sheet per member, and a trial on a small home server. The end-to-end scenario (phone and PC, airplane mode, sync) is in the repository; it still has to run in continuous integration.\nTry it # git clone https://github.com/gwils28/Shelterkwak cd Shelterkwak make install # Python (uv) and Node dependencies make check # lint, tests, engine parity, build make run # the app on http://127.0.0.1:8000 Installing on an Android phone, with Tailscale, is described in docs/telephone.md (in French).\n","date":"9 November 2026","externalUrl":null,"permalink":"/en/projects/shelterkwak/","section":"Projects","summary":"An app installed on the phone, with no account and no third-party service: 279 goals in nine blocking tiers, a level that drops when supplies expire, an inventory that computes the household’s days of self-sufficiency, and an offline Crisis screen for the day itself.","title":"Shelterkwak: tracking a household's crisis preparedness, tier by tier","type":"projects"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/sqlite/","section":"Tags","summary":"","title":"Sqlite","type":"tags"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"9 November 2026","externalUrl":null,"permalink":"/en/tags/typescript/","section":"Tags","summary":"","title":"Typescript","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/backtest/","section":"Tags","summary":"","title":"Backtest","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/chronos-2/","section":"Tags","summary":"","title":"Chronos-2","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/conformal-prediction/","section":"Tags","summary":"","title":"Conformal Prediction","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/dagster/","section":"Tags","summary":"","title":"Dagster","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/tags/%C3%A9nergie/","section":"Tags","summary":"","title":"Énergie","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/energy/","section":"Tags","summary":"","title":"Energy","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/forecasting/","section":"Tags","summary":"","title":"Forecasting","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/foundation-models/","section":"Tags","summary":"","title":"Foundation Models","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/lightgbm/","section":"Tags","summary":"","title":"LightGBM","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/mlflow/","section":"Tags","summary":"","title":"MLflow","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/tags/mod%C3%A8les-de-fondation/","section":"Tags","summary":"","title":"Modèles De Fondation","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/tags/s%C3%A9ries-temporelles/","section":"Tags","summary":"","title":"Séries Temporelles","type":"tags"},{"content":"","date":"8 November 2026","externalUrl":null,"permalink":"/en/tags/time-series/","section":"Tags","summary":"","title":"Time Series","type":"tags"},{"content":" Moving from the tech-watch question, \u0026ldquo;when does a foundation model beat a specialist?\u0026rdquo;, to a client\u0026rsquo;s: what to nominate every day at 9 am for a portfolio of 3,517 commercial sites, with which intervals, and how much it is worth compared with what they already do. StatusCompleted · 5 blocks, local service, 16 October addendum Timeline18 August – 8 October 2026, addendum from 16 to 31 October Stack Python 3.12LightGBMChronos-2TimestralTimesFM 3.0statsforecastDagsterMLflowFastAPIDockerQuartouvpytest Source code 2.7%\nMAPE on the portfolio\nLightGBM + Chronos-2 blend, against 5.6% for RTE\u0026rsquo;s forecast scaled down to the portfolio 1.10\nagainst debiased RTE\nnational MAE ratio, 95% CI [0.99, 1.22]: a one-line correction ties with every model −6.8%\nMASE from blending\nagainst −1.8% for the best possible router, known after the fact Transparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of this page. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThis repository applies the article Time series foundation models: zero-shot against the specialist to a client brief. The article asked the question on a single series and rebuilt a router; this project asks it again with a real benchmark to beat, covariates, a metric in euros and a service to run. The repository, its online report and its figures are in French.\nThe goal # FlexGrid Énergie is a fictional balance responsible party (BRP). Every day, it must nominate its portfolio\u0026rsquo;s consumption for the next day in 30-minute steps, before the day-ahead market closes at noon. Any gap between nomination and actual consumption is settled at the balancing price. The brief put the current forecast at about 6% MAPE and set a blunt criterion: beat RTE\u0026rsquo;s day-ahead forecast published in éCO2mix, or the project has no demonstrable value.\nThe data is public and real: national consumption from éCO2mix with RTE\u0026rsquo;s day-ahead forecast, Enedis aggregated load curves (sites above 36 kVA, by region, power band and sector), archived weather forecasts from Open-Meteo, and day-ahead prices from Energy-Charts. The test year runs from 1 July 2025 to 30 June 2026: 365 delivery days, a forecast issued every day at 9 am with only what is known at that time.\nWhat I wanted to learn came down to three questions:\nwhat happens to the tech watch\u0026rsquo;s conclusion (a zero-shot TSFM beat univariate specialists) when the specialist is given weather and calendar; how to judge a model against a public benchmark when that benchmark has flaws of its own; what a backtest needs before it becomes a service that can be monitored. The approach # The work follows the brief\u0026rsquo;s five blocks, each with its own folder and results README. The decisions made before writing code are in SPEC.md, with fourteen blind spots found in the data; the evaluation protocol was frozen in configs/protocol.yaml before the first computation, and every deviation is dated in IMPLEMENTATION_NOTES.md. The working method is the one in Claude Code for data science: what changes in my working method.\n1 Data Sources frozen by sha256, clock changes repaired by an explicit rule, the nomination clock rebuilt: national load known up to 8 am, site load curves up to the end of D-2, weather for day D forecast 48 hours ahead. The portfolio is the sum of two real Enedis segments: 3,517 sites, 815 GWh a year. 2 Models and backtest Naive forecasts, MSTL and ETS; quantile LightGBM on lags, calendar and weather, refitted weekly; Chronos-2 with and without covariates; the blend of both models\u0026rsquo; quantiles. Rolling origin over the test year, after six months of validation. Pre-registered verdicts against published RTE and debiased RTE, under two readings of the quarter-hour. 3 Router and cold start A per-segment router on a panel of 81 commercial segments, learned on validation and compared with the oracle and the blend. New portfolios simulated on three held-out regions, with 7 to 56 days of history. 4 From MW to euros Nomination error valued at the day-ahead price, with several penalty levels, and avoided cost with a confidence interval from seven-day blocks. 5 Service and monitoring Dagster (one partition per delivery day, scheduled at 9 am the day before), MLflow registry, FastAPI for quantiles and nominations, a Docker image, then monitoring replayed over the test year: data drift, concept drift, calibration, monthly cost. An addendum, written on 16 October before any forecast from these models, added Timestral (Mistral AI) and TimesFM 3.0 (Google Research), both released after the project ended.\nWhat the data corrected in the brief # The brief says What the data says \u0026ldquo;5,000 commercial sites and EV chargers\u0026rdquo; No public 30-minute data for chargers. The portfolio becomes two real Enedis segments (3,517 sites, 815 GWh/year), close to the requested order of magnitude. \u0026ldquo;Beat RTE\u0026rsquo;s day-ahead forecast\u0026rdquo; RTE only forecasts the national load, and its forecast has under-predicted actual consumption by about 1.1 GW since 2023. The criterion becomes: beat published RTE and debiased RTE. \u0026ldquo;Temperatures by département\u0026rdquo; Observed temperature is a leak: at 9 am the day before, only a forecast exists. Verdicts use the weather forecast issued 48 hours ahead. \u0026ldquo;Cost = volume × imbalance price\u0026rdquo; The imbalance settlement price cannot be downloaded without an account. Cost is valued at the day-ahead price, with a penalty factor from 1 (the brief\u0026rsquo;s definition) down to 0.1. \u0026ldquo;Current forecast at 6% MAPE\u0026rdquo; Almost exactly the MAPE of RTE\u0026rsquo;s forecast scaled to the portfolio half-hour by half-hour (5.6%): that of a method with no model. The results # Every figure comes from the versioned tables in results/, produced by the flexgrid CLI. Confidence intervals are bootstraps over seven-day blocks.\nThe benchmark had a bias # From 2015 to 2021, RTE\u0026rsquo;s day-ahead forecast has a MAPE of 1.4 to 1.6%. Since 2023 it has been around 2.7%, and the forecast under-predicts actual consumption by about 1 GW every year. Before any model, the data block showed two things about the reference. Its bias is stable, positive in every month of the test year: a BRP can correct it in one line, by subtracting the mean error of the last 28 known days for each half-hour. And the way a quarter-hourly forecast is read on a 30-minute grid moves its MAE by 6.5% (1,400 or 1,310 MW), with nothing in the documentation to settle it. A model only \u0026ldquo;beats RTE\u0026rdquo; here if it beats both readings.\nThe sources had their own traps too: éCO2mix publishes hours that do not exist when clocks go forward (48 duplicated UTC timestamps over 2015-2026) and drops the repeated October hour every year.\nNational: published RTE beaten, debiased RTE not # Below 1, the model does better than the reference. On the left, almost every model with covariates beats published RTE. On the right, none beats debiased RTE; the LightGBM + Chronos-2 blend sits at 1.10. The LightGBM + Chronos-2 blend has an MAE of 897 MW against 1,401 MW for published RTE (ratio 0.64, 95% CI [0.56, 0.73]). The brief\u0026rsquo;s criterion is met literally. But debiased RTE scores 816 MW: against it, the blend shows no difference (1.10 [0.99, 1.22]) and loses under the other quarter-hour reading (1.14 [1.02, 1.26]).\nNational, test year MAE (MW) MAPE 90% coverage Debiased RTE day-ahead 816 1.61% (point forecast) LightGBM + Chronos-2 blend 897 1.73% 0.86 LightGBM 972 1.87% 0.76 Chronos-2 + covariates 1,109 2.15% 0.83 Published RTE day-ahead 1,401 2.79% (point forecast) Chronos-2 without covariates 1,752 3.33% 0.81 MSTL 2,284 4.47% 0.79 Three readings:\nCovariates matter more than the model. Chronos-2 drops from 1,752 to 1,109 MW once it is given weather and calendar. Zero-shot without covariates, a TSFM does not know it will be cold tomorrow. Perfect weather would have changed almost nothing. Fed with observed instead of forecast weather, LightGBM gains only 4% in MAE. What separates the models from RTE is information on consumption itself: RTE revises its forecast later, with its own telemetry. The blend wins in summer and loses in winter. It beats debiased RTE from July to October (592 against 706 MW) and on one day in two over the year, but its bad days are winter days, where errors are counted in GW. On the portfolio: a model halves the brief\u0026rsquo;s 6% # On the portfolio, the LightGBM + Chronos-2 blend has 0.49 times the MAE of RTE scaled to the portfolio and 0.36 times that of the weekly naive forecast. RTE does not forecast the portfolio, so there is no debiased RTE to fall back on. The best method without a model is RTE\u0026rsquo;s forecast multiplied by the portfolio\u0026rsquo;s recent share, half-hour by half-hour. This benchmark was added after the protocol, and declared as such: without it, the portfolio\u0026rsquo;s only reference was a naive forecast.\nPortfolio, test year MAE (MW) MAPE 90% coverage LightGBM + Chronos-2 blend 2.57 2.72% 0.85 LightGBM 2.64 2.78% 0.75 Chronos-2 + covariates 3.16 3.36% 0.84 RTE day-ahead, pro rata 5.28 5.59% (point forecast) Weekly naive 7.17 7.55% 0.87 The gap is much wider than at national level because the portfolio does not look like France: an office peak around 10 am, a flat Sunday, and summer air conditioning that the national load barely has. A national forecast, even rescaled half-hour by half-hour, carries the national shape. The blend beats pro-rata RTE every month and on 86% of days.\nPublic holidays, the TSFMs\u0026rsquo; weak spot # On 25 December, Chronos-2 and pro-rata RTE forecast a working day. LightGBM, which has seen ten years of holidays in training, knows the offices are closed. This was already the weak spot of zero-shot in the tech watch, and it remains so with a calendar covariate. On the year\u0026rsquo;s eleven public holidays, LightGBM has 0.42 times the MAE of pro-rata RTE, Chronos-2 1.25. The blend therefore inherits Chronos-2\u0026rsquo;s errors on those days. A rule like \u0026ldquo;LightGBM alone on holidays\u0026rdquo; would be an easy gain, but it was not pre-registered, so it is not applied. On the four bridge days, every model loses: a bridge day in commercial buildings means half the offices closed, which no calendar variable measures.\nThe intervals # Raw, they do not keep their promise. LightGBM\u0026rsquo;s 90% interval covers 75% of steps on the portfolio and 76% at national level. The SPEC attributed this under-coverage to the gap between observed weather in training and forecast weather in service; measurement refuted it (77% with observed weather as input). The main cause is quantile regression itself, which fits its training data too closely.\nThe tech watch\u0026rsquo;s conformal repair (median kept, quantiles rebuilt from the errors of the last 56 days) brings every model back to 0.90-0.91 over the year. It follows the seasons with a lag: the blend\u0026rsquo;s repaired coverage drops to 0.78 in November, when the error rises with heating.\nBlending rather than routing # Mean MASE of each policy divided by that of \u0026quot;always LightGBM\u0026quot;. The learned router gains 0.4%, the oracle 1.8%, the blend 6.8%. Below the dashed line, the addendum\u0026rsquo;s policies; the last blend gives ½ to LightGBM and ⅙ to each of the three TSFMs. On the 81 segments of the panel, a stump (one variable, one threshold) learned on validation sends only one segment to Chronos-2. Even the oracle router, which knows the right answer, only gains 1.8%: LightGBM has the lowest MASE on 66 segments out of 81, and where Chronos-2 wins (electrically heated segments, the largest sites), it wins by little. The blend gains 6.8% and beats the oracle on 70 segments. Two models that err at different times correct each other over time; a router only chooses once per segment.\nThe decision also drifts, as in the tech watch: reapplied every week, the stump sends between 0 and 50 segments to Chronos-2 depending on the season, and 4.3 segments switch branch each week on average. The service therefore nominates the blend, which has no decision to monitor.\nA new portfolio is first nominated with the naive forecast # With 7 days of history, the weekly naive forecast (green) beats almost everything; only Timestral with sibling segments in context goes below it. From 14 days on, the models converge below the naive forecast. WAPE 7 days 14 days 28 days 56 days Weekly naive 6.6% 6.6% 6.6% 6.6% LightGBM trained on other regions 12.6% 3.9% 3.9% 3.9% Chronos-2 + covariates 19.2% 4.4% 3.9% 3.6% Chronos-2 few-shot in-context 19.7% 4.4% 4.0% 3.8% Timestral few-shot (addendum) 5.7% 4.2% 4.0% 3.8% With one week of history, Chronos-2 sees a single weekly cycle, does not recognise it as a season, and forecasts a Tuesday halfway between a working day and a Sunday. LightGBM receives a missing 14-day lag it never saw in training. The naive forecast simply copies the right day. From 14 days on, LightGBM trained on other regions takes the lead, within 0.5 points of what it does with full history. Chronos-2\u0026rsquo;s few-shot in-context mode, which the brief proposed for the cold start, does not help. Timestral\u0026rsquo;s is the first model below the naive forecast at 7 days, without it being possible to conclude that it beats it (ratio 0.87 [0.73, 1.04]).\nThe resulting rule: the naive forecast for two weeks, then the global model. One limit: a segment aggregates hundreds of sites. What is simulated is a new portfolio arriving, not an isolated site, which would be far noisier.\nTimestral and TimesFM 3.0: context matters more than the model # MAE divided by that of Chronos-2 with weather and calendar. With equal context, neither Timestral nor TimesFM 3.0 does better; with almost two years of context, Timestral moves ahead. The addendum asked three questions, with the same origins, the same inputs and the published default settings:\nDoes a new TSFM beat Chronos-2? With equal context (2,048 steps), no: Timestral has 1.18 times its national MAE, TimesFM 3.0 1.19. Without covariates, all three are level: the gap comes from how each reads covariates. With its full context (32,768 steps, almost two years), Timestral moves ahead: 990 MW, 0.89 times Chronos-2. More context helps neither Chronos-2 nor TimesFM 3.0. Does a calendar covariate help on holidays? Yes, it removes about a quarter of Chronos-2\u0026rsquo;s and Timestral\u0026rsquo;s error on those days, almost nothing from TimesFM 3.0\u0026rsquo;s. The TSFMs remain far behind LightGBM: a binary \u0026ldquo;holiday\u0026rdquo; covariate does not replace ten years of holidays seen in training. Do the new TSFMs improve the blend? No. Two blends pass the pre-registered switch rule on validation; neither is confirmed on the test year. The service keeps LightGBM + Chronos-2. An exploratory result, declared as such: the LightGBM + long-context Timestral blend, built after seeing the results, is the best model in the repository. At national level it scores 832 MW and ties with debiased RTE (1.02 [0.92, 1.13]). Chosen by looking at the test year, this figure is optimistic; it counts in no verdict and will be a candidate for a future protocol, on a window it has not seen.\nFrom MW to euros: cost does not follow error # The blend\u0026rsquo;s error is spread over the office day; its cost concentrates on the two price peaks, around 8 am and 7 pm. The solar dip at midday makes a midday error cheap. Under the brief\u0026rsquo;s definition (gap × day-ahead price), the blend avoids €1.24M a year compared with pro-rata RTE (95% CI [0.93, 1.56] €M), or €0.12M to €0.62M with a more realistic penalty. The ranking of models does not depend on the penalty, only the amounts do. The interval is wide because a year holds only one winter: from November to January, three months account for 47% of the annual cost.\nCost is not a disguised MAE. Pro-rata RTE errs mostly at midday, when prices are low. The blend avoids 51% of the energy in imbalance but only 47% of the cost. A model aiming at cost rather than MAE would weight its errors by the expected price: that is not done here, and it is the natural next step.\nThe service and its monitoring # One Dagster partition per delivery day, scheduled at 9 am the day before. LightGBM is refitted weekly and registered in MLflow; Chronos-2 runs on GPU, with a LightGBM fallback beyond 300 seconds. Three partitions were materialised end to end, including both clock-change days (46 and 50 steps). Production reproduces the backtest: identical quantiles for Chronos-2, and for LightGBM on the day the service and the backtest refit on the same date. This is the check that ties the verdicts to the code being served.\nMonitoring was replayed over the test year, and says so: there was no real deployment. Two findings came out of it. The naive alert threshold, \u0026ldquo;the model does worse than debiased RTE\u0026rdquo;, fires on 208 days out of 365 at national level: it describes a state known since the backtest and detects nothing. The threshold kept is self-referenced (1.15 times the model\u0026rsquo;s ratio on validation). And LightGBM\u0026rsquo;s raw coverage drops to 0.52 on the portfolio in January: the brief\u0026rsquo;s \u0026ldquo;90% that drifts to 70%\u0026rdquo; did happen, and the conformal repair catches up with a lag.\nWhat got in the way # The benchmark was the first question. \u0026ldquo;Beat RTE\u0026rdquo; looked settled. The +1.1 GW bias and the quarter-hour ambiguity turned it into two benchmarks and two readings, and the conclusion flipped between published and debiased RTE. Leaks come from the clock. A two-day lag counted in UTC steps read, on the day clocks go back, a value later than the segments\u0026rsquo; cutoff; a test found it before any computation. Later, a backfill of 26 October 2025 reused a registry model trained for 30 June 2026, eight months in the future. A tested rule now prevents it. A sanity check removed real values. The pre-registered range filter dropped 7,399 real steps from an electrically heated segment whose winter is six times its summer. It was replaced before any forecast, and the deviation is dated. Compute. The full backtest takes several hours on an RTX 5070 and 32 cores, including 4.5 hours for LightGBM on the panel, and the addendum adds about 9 GPU hours. Two LightGBM jobs launched together took five times longer: heavy jobs run one after the other. The Docker image weighs 6.9 GB compressed, almost all of it torch and CUDA, while the API alone needs neither. A separate image for the API is the next step. What I take away # The tech watch\u0026rsquo;s conclusion does not carry over as is. Against univariate specialists, Chronos-2 won everywhere; against a LightGBM fed with weather and calendar, it loses on 66 segments out of 81. What makes the difference is what each model receives as input. \u0026ldquo;Beating the benchmark\u0026rdquo; is checked against the corrected benchmark. A one-line bias correction tied with all my models at national level. On the portfolio, where no such correction exists, the gap is real: 2.7% against 5.6%. Between the specialist and the TSFM, the best answer was \u0026ldquo;both\u0026rdquo;. The blend beats the best possible router, and removes a decision to monitor. Holidays remain the TSFMs\u0026rsquo; weak spot, even with a calendar covariate. A model that has seen ten years of them does better. An honest replay states what it does not show: the alert thresholds were set after seeing the naive threshold saturate, and real monitoring would set them on validation and keep them for a year. Reproduce # git clone https://github.com/gwils28/flexgrid_dayahead cd flexgrid_dayahead uv sync --extra prod # Python 3.12; Chronos-2 runs on GPU (a CPU works, much more slowly) make data # downloads and freezes the sources (sha256 in data/MANIFEST.json) uv run flexgrid build # intermediate tables uv run flexgrid forecast national snaive_day snaive_week mstl ets_seasonal lgbm chronos2 chronos2_cov uv run flexgrid forecast segments snaive_day snaive_week mstl ets_seasonal lgbm chronos2 chronos2_cov uv run flexgrid score national segments \u0026amp;\u0026amp; uv run flexgrid tables uv run flexgrid router \u0026amp;\u0026amp; uv run flexgrid coldstart make figures report # figures for every block, then the Quarto report in docs/ uv run pytest The addendum\u0026rsquo;s models run the same way (list in configs/protocol.yaml). Raw forecasts are not versioned; the per-origin tables are, and every figure and verdict starts from them. Licences: Chronos-2 Apache-2.0; Timestral, code Apache-2.0 and weights under a modified MIT licence; TimesFM 3.0, non-commercial weights, which a real BRP could not serve without another licence.\n","date":"8 November 2026","externalUrl":null,"permalink":"/en/projects/flexgrid-prevision-j1/","section":"Projects","summary":"The TSFM tech watch applied to a fictional balance responsible party, on real public data: LightGBM, Chronos-2 and their blend against RTE’s day-ahead forecast, published and debiased. A 2.7% MAPE on the portfolio against 5.6% without a model, a router that does not beat the blend, a cold start handled by the naive forecast, Timestral and TimesFM 3.0 in an addendum, a Dagster service and its replayed monitoring.","title":"Time series foundation models: forecasting tomorrow's electricity consumption for 3,517 commercial buildings","type":"projects"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/tags/chronos/","section":"Tags","summary":"","title":"Chronos","type":"tags"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/tags/intervalles-de-pr%C3%A9vision/","section":"Tags","summary":"","title":"Intervalles De Prévision","type":"tags"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/tags/mase/","section":"Tags","summary":"","title":"MASE","type":"tags"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/tags/prediction-intervals/","section":"Tags","summary":"","title":"Prediction Intervals","type":"tags"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/tags/statsforecast/","section":"Tags","summary":"","title":"Statsforecast","type":"tags"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/categories/tech-watch/","section":"Categories","summary":"","title":"Tech Watch","type":"categories"},{"content":" A time series foundation model forecasts a series it has never seen, without any training. A classical statistical model is refitted on each series, at every forecast. Which one wins, at which horizon, on which days, and with intervals that can be trusted? I asked the question on a year of French electricity consumption, hour by hour. Reading time: about 30 minutes. Every formula is followed by its intuition, so you can skim the maths without losing the thread.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThis article distils two weeks of hands-on research (2 to 15 August 2026) into time series foundation models, TSFMs for short. The plan had four blocks: three recent resources to read at the source, a backtest harness pitting a TSFM used as-is against strong statistical models, an architecture decision for the \u0026ldquo;complexity router\u0026rdquo; of a May 2026 paper, and the founding paper of the MASE to reread. Its success criterion: say in one sentence at which horizon and in which regime the TSFM wins or loses, based on interval coverage and not only on the mean error. Everything is reproducible in the W25_tsfm_viability repository.\nIn short. The main results, with their scope:\nOn French load in 2019, Chronos-Bolt base beats the reference model at every horizon step and on every ordinary day type, without ever having seen the series. Against the strongest specialist (MSTL) the gap is smaller: 15% less error for Bolt base, 24% for Chronos-2. Chronos-Bolt\u0026rsquo;s \u0026ldquo;90%\u0026rdquo; interval is its 80% interval, by construction. A conformal repair, built from its own past errors, brings it to 0.902. Public holidays are where zero-shot loses: its 90% interval covers 37% of the hours there. Given the calendar as a covariate (exploratory), Chronos-2 halves its error on those days. A well-calibrated interval is not necessarily a useful one: the reference, which forecasts a flat line, covers 0.899 of the hours with an interval 45% of the load wide. On 50 European series, Chronos-2 wins on all 200 units (series × history length), most of all in cold start. A router can then only trade accuracy for cost, and the measured cost does not justify the trade. The routing decision drifts: 40 of 50 series change branch during the year. 1. The problem: one model per series, or one model for all # Time series forecasting classically relies on one model per series: pick a family (exponential smoothing, ARIMA, seasonal decomposition), fit it on the history, refit it as data arrives. I call this model the specialist: it only knows its own series, but it knows it well.\nA foundation model is pretrained once, on a large corpus of heterogeneous series (energy, traffic, sales, weather, synthetic data), to learn what they have in common: seasonalities, trends, moving levels. In use, it is given the recent past of a new series (its context) and forecasts what comes next without any fitting: this is zero-shot. The same model serves the whole portfolio.\nThe two models studied come from Amazon\u0026rsquo;s Chronos family:\nChronos-Bolt (small and base sizes) splits the context into patches of a few steps and directly forecasts the next 24 hours as nine quantiles, from 0.1 to 0.9. It sees nothing but the series. Chronos-2, more recent, forecasts 21 quantiles from 0.01 to 0.99 and accepts covariates: series known in the past, or known in advance, such as a calendar. A 0.9 quantile is the value below which the model expects reality to fall 90% of the time. Between the 0.05 and 0.95 quantiles lies a 90% prediction interval: if well calibrated, it contains reality 9 hours out of 10. That promise, as much as the mean error, is what the plan asked to check.\n2. What the sources say # I reread the plan\u0026rsquo;s three resources at the source (papers, model cards, published code, GIFT-Eval leaderboard results recomputed), last checked on 30 October 2026.\nClaim in the plan What the source says Verdict IBM\u0026rsquo;s three forecasting models (March 2026) emit quantiles natively True for FlowState r1.1 (9 quantiles), TTM-r3 (new multi-quantile head) and PatchTST-FM r1 (99 quantiles). Only TTM is new on this point holds The GIFT-Eval point-forecast leader has only 18.5M parameters On 12 April 2026, FlowState r1.1 was first by MASE only under the \u0026ldquo;zero-shot\u0026rdquo; and \u0026ldquo;replication code\u0026rdquo; filters; unfiltered, 13th. On 30 October 2026 it is 13th even filtered partly, and outdated The complexity router (arXiv:2605.24381) gives better accuracy at much lower cost than a universal TSFM The headline figure ranks series by their realised test MASE: an oracle. The deployed rule, in the authors\u0026rsquo; repository, scores 0.981 against 0.989 for the TSFM alone, with 33% fewer calls. The cost ratio (1,000 to 1) is assumed partly Specialists win on physically constrained systems The paper\u0026rsquo;s \u0026ldquo;energy\u0026rdquo; regime is ETTh1: the oil temperature of 7 transformers. No load series does not transfer The blog post classifies TSFM limits as fixable, unfixable and open Published 24 April 2026. Two groups only: true challenges (loss, context, high-frequency bias, scaling laws, covariates) and non-problems (interpretability, data leakage) partly The third point shaped the rest. The paper\u0026rsquo;s complexity router computes four indicators on each series\u0026rsquo; history (spectral entropy, coefficient of variation, seasonal autocorrelation, trend share); the series goes to the TSFM when at least two of the four thresholds are crossed, otherwise to a specialist.\nLeft, MASE on Traffic and M4 Daily: the router as deployed (0.981) does about as well as the TSFM alone (0.989); the 0.964 is only reached by ranking series by their test error. Right, the share of TSFM calls avoided: 70% for the oracle, 33% for the deployed rule. An oracle uses information that is not available at decision time, here the error each model will make on the test period. It bounds what a perfect router would gain, not what a real one gains. Hence a rule for block 3: evaluate the router out of sample from the start.\nTwo more checks. Contamination: Chronos-2\u0026rsquo;s training list contains neither ENTSO-E nor OPSD, the sources of my series; Chronos-Bolt\u0026rsquo;s is not itemised, and might contain Spanish load for 2015-2018 (same provider, another country). Nothing says Bolt\u0026rsquo;s score is optimistic, nothing rules it out. Licences: the Chronos models are Apache-2.0; IBM\u0026rsquo;s TTM-r3, under CC-BY-NC-SA-4.0, cannot go into a commercial deliverable.\n3. The series and its traps # The series is French national electricity consumption, hour by hour, published by ENTSO-E and packaged by Open Power System Data (version of 6 October 2020, checksum verified).\nDaily mean and min-max range. Winter peaks come from electric heating. In blue, the test year (365 origins); in red, the first lockdown window, treated separately. On top of the yearly cycle driven by heating, load superimposes two cycles: 24 hours (night, morning ramp, plateau, evening peak) and 168 hours (five working days, a lower weekend). A model that has only one of the two is systematically wrong.\nLeft, the mean daily profile by weekday (2017-2018, Paris time): working days look alike, Saturday and Sunday are clearly lower. Right, two weeks of January 2018: the 24-hour cycle and the weekend dip. Defects found before modelling # Left, 9 gaps of 1 to 13 hours, 42 hours out of 50,398. Right, one hour at 158 GW on 7 July 2020, above any physically possible value. One hour at 158 GW, for a national record of about 102 GW, and nine gaps. Values outside [20, 110] GW are set to missing. Gaps are filled for model inputs only (linearly up to 3 hours, otherwise with the previous week), and an imputed hour is never scored: a model should not be rewarded for recovering a value I made up.\nTwo naives rather than one # The plan asked for a season-24 SeasonalNaive as the baseline: tomorrow is a copy of today. It therefore forecasts Monday with Sunday, and I expected a strawman. I added the weekly naive (a copy of last week). The measurement proved me wrong.\nLeft, a Monday: the copy of Sunday (green) misses the level of a working day, the copy of the previous Monday (orange) is too high. Right, mean absolute error by weekday over 2017-2018: neither naive dominates. The daily naive clearly loses on Monday and Saturday, the days that follow a change of regime; on the other five days, yesterday looks more like today than last week does. Both stay in the bench.\n4. The protocol: a rolling-origin backtest # A rolling-origin backtest replays history: stand at a past date (the origin), give the models only the earlier data, forecast the next 24 hours, compare with reality, then move forward one day. The plan asked for at least 6 origins; I took one per day over 2019, i.e. 365, plus a separate stress window during the first lockdown (55 origins).\nAt each origin, the statistical models are refitted on their window, the naives copy a day or a week, and the Chronos models read 2,048 hours of context without fitting anything. The next 24 hours are scored. Model Fit Window Intervals SeasonalNaive(24), SeasonalNaive(168) none last day, last week from residuals AutoETS, season 24 refitted at every origin, AICc selection 672 h (4 weeks) model-based ETS(Z,Z,A), seasonality forced (declared deviation) refitted 672 h model-based MSTL [24, 168] + AutoETS trend refitted at every origin 1,344 h (8 weeks) model-based Chronos-Bolt small, base zero-shot 2,048 h context quantiles 0.1 to 0.9 Chronos-2 zero-shot, one series at a time 2,048 h context quantiles 0.01 to 0.99 ETS is the exponential smoothing family: a level, optionally a trend and a seasonality, updated with more weight on recent data; AutoETS picks the combination with an information criterion. MSTL decomposes the series into several seasonalities (24 and 168 hours) and a trend, forecasts each piece and recombines them: it is the only statistical model in the bench that knows both cycles.\nThe protocol, the verdict thresholds and the uncertainty rule were committed before the first forecast. Any value changed afterwards is a deviation, declared wherever it appears (there was one). Tests corrupt future values and check that the forecasts do not move: no leakage from the future.\nMeasuring the uncertainty of a gap # \u0026ldquo;Chronos makes 15% less error than MSTL\u0026rdquo;: on another year it might be 5% or 25%. The bootstrap gives a confidence interval: draw origins at random, with replacement, among the 365, recompute the ratio, 2,000 times, and keep the central 95%. Whole origins are drawn, because the 24 hours of one forecast are not independent, in blocks of 7 consecutive days (moving block bootstrap), because neighbouring days look alike.\nThe pre-registered verdict rule: the challenger wins if the interval of the MASE ratio (challenger / reference) lies entirely below 1 and the coverage of its 90% interval is in the [0.85, 0.95] band; it wins on points only if coverage falls outside the band; it loses if the ratio\u0026rsquo;s interval lies above 1; otherwise, no conclusion.\n5. Measuring a forecast: the MASE and its alternatives # The fourth block of the plan was a reading, Hyndman and Koehler (2006), Another look at measures of forecast accuracy, which introduced the MASE. I put it here because everything that follows depends on it.\nThe mean absolute error (MAE) is in megawatts: it cannot be averaged between Germany and Latvia. The MASE divides the error by that of a naive method on the same series\u0026rsquo; history:\n$$ \\text{MASE} = \\frac{\\dfrac{1}{H}\\sum_{h=1}^{H} \\lvert y_{T+h} - \\hat y_{T+h} \\rvert} {\\dfrac{1}{T-m}\\sum_{t=m+1}^{T} \\lvert y_t - y_{t-m} \\rvert} $$The numerator is the error over the \\(H = 24\\) forecast hours; the denominator, the mean error the copy of the value \\(m\\) hours earlier would have made over the history. With \\(m = 24\\), a MASE of 0.4 means \u0026ldquo;40% of the error of copying yesterday\u0026rdquo;. Rereading the paper, I noticed it defines the MASE with the one-step naive (\\(m = 1\\)); the seasonal version comes from the FPP3 textbook (section 5.8), which is the one to cite for \\(m = 24\\).\nWhy not the MAPE, nor the sMAPE # Left, the same series shifted towards zero, with the same forecast errors at every level. Right, the MAPE and the sMAPE grow by two orders of magnitude as the series approaches zero; the MASE stays at 1.29. The MAPE divides each error by the observed value: the same errors give a MAPE a hundred times larger near zero, and it is undefined at zero. It also assumes a meaningful zero, which degrees Celsius do not have. The so-called \u0026ldquo;symmetric\u0026rdquo; sMAPE divides by the mean of actual and forecast, and is not symmetric either.\nFor y = 100, an error of 60 costs 86% if the forecast is too low and 46% if it is too high: a model that over-forecasts looks better. French load never drops below 29 GW, so the MAPE would remain usable there. The argument for the MASE here is comparability across series. (The utilsforecast library computes an sMAPE equal to half the textbook one; it drives no decision.)\nSame model (Chronos-2, 12 weeks of context) on the 50 series of block 3. Left, the MAE spans a factor of 70, from 18 MW (Latvia) to 1,245 MW (Great Britain), and ranks countries by size. Right, the MASE runs from 0.25 to 0.87 and ranks series by how hard they are to forecast. The MAE ranks countries by size; the MASE ranks them by difficulty (Sardinia is the hardest, a German zone the easiest). That is what makes the block 3 router possible.\nThe choice of scale changes the meaning of \u0026ldquo;1\u0026rdquo; # Left, the in-sample error of the lag-m naive: 1.81 GW for m = 1, 3.08 GW for m = 24, 3.62 GW for m = 168. Right, the MASE of the same forecasts under the three scales: the ranking never changes, the reading of the threshold 1 does. With \\(m = 24\\), the daily naive scores 1.01: out of sample, it repeats what defines the scale. With \\(m = 1\\), the same forecasts score 1.72, because a 24-hour forecast is compared with the error of a one-hour forecast. A MASE only means something with its \\(m\\).\n6. Measuring an interval: coverage, width and score # The plan insisted: \u0026ldquo;if the TSFM wins on MASE but its actual 90% coverage is 70%, you must be able to say so\u0026rdquo;. Four measures serve for intervals:\ncoverage, the share of hours where reality falls inside the interval (0.90 targeted at 90%); the mean width, as a share of the load; the Winkler score, which combines both for an interval \\([l, u]\\) at \\(1-\\alpha = 90\\%\\): $$ W = (u - l) + \\frac{2}{\\alpha}\\,(l - y)\\,\\mathbb{1}_{y \u003c l} + \\frac{2}{\\alpha}\\,(y - u)\\,\\mathbb{1}_{y \u003e u} $$You pay the width, plus twenty times the miss (\\(2/\\alpha = 20\\)) when reality falls outside: a narrow interval that misses pays the penalties, a huge one pays its width;\nthe quantile loss (pinball loss), \\(\\rho_\\tau(y, q_\\tau) = (\\tau - \\mathbb{1}_{y \u003c q_\\tau})(y - q_\\tau)\\), which scores each quantile: for \\(\\tau = 0.9\\), being below reality costs 0.9 per unit of gap, being above costs 0.1, and the value that minimises this loss is exactly the 0.9 quantile. Averaged over 0.1 to 0.9, it scores the whole distribution. 7. Results on the year 2019 # The table printed by w25 backtest, identical from one run to the next (bit for bit on the same GPU, within 0.016 MW between CPU and GPU):\nModel MASE WAPE cov80 cov90 width 90 Winkler 90 quantile loss SeasonalNaive(24) 1.001 0.059 0.824 0.879 0.293 7.08 0.429 SeasonalNaive(168) 1.084 0.061 0.854 0.915 0.330 6.92 0.448 AutoETS 1.715 0.102 0.772 0.899 0.450 8.74 0.669 AutoETS + conformal 1.715 0.102 0.806 0.908 0.337 6.34 0.609 ETS(Z,Z,A), deviation 1.010 0.061 0.652 0.736 0.176 6.96 0.410 MSTL 0.479 0.028 0.689 0.779 0.090 3.49 0.195 Chronos-Bolt small 0.435 0.026 0.859 0.859 0.088 2.79 0.173 Chronos-Bolt base 0.406 0.024 0.858 0.858 0.082 2.65 0.162 Bolt base + conformal 0.406 0.024 0.795 0.902 0.108 2.72 0.163 Chronos-2 0.365 0.021 0.761 0.862 0.076 2.37 0.146 cov80 and cov90: coverage of the nominal 80% and 90% intervals. Width as a share of the load. The \u0026ldquo;conformal\u0026rdquo; rows are described in section 8.\nMASE on the x axis (lower is better), coverage of the 90% interval on the y axis. The grey band is the pre-registered [0.85, 0.95] band. Bars are 95% confidence intervals; diamonds, the conformal repair of the same model. The figure reads by corners: MSTL bottom left, accurate but under-covering; AutoETS top right, well covered but very inaccurate; the Chronos models top left, the only ones both accurate and close to the band.\nThe reference model forecasts a flat line # AutoETS\u0026rsquo;s MASE of 1.715, worse than the naive, made me look at what it was choosing: at 572 of the 608 origins, ETS(A,N,N), a level alone that forecasts the last value 24 times. Its selection criterion, the AICc, rests on the likelihood of one-step-ahead forecasts; on a smooth hourly series, the best forecast of the next hour is roughly the current hour, and the level-only model wins that contest. At 24 steps it knows nothing of the daily cycle.\nThe pre-registered verdict is still reported (\u0026ldquo;Bolt base makes four times less error\u0026rdquo;), but it says little. As a declared deviation, I added an ETS with forced seasonality, ETS(Z,Z,A), on par with the daily naive (1.010). Even against it, Bolt base makes 2.5 times less error. The real contender remains MSTL.\nA flat line can be \u0026ldquo;perfectly calibrated\u0026rdquo; # AutoETS covers 0.899 of the hours, almost exactly the nominal level, with an interval 45% of the load wide: from 30 to 60 GW on a summer day. Its Winkler score, 8.74, is the worst in the table. Coverage alone rewards a useless interval.\nMedian and 90% interval of Chronos-Bolt base and Chronos-2, mean forecast of MSTL, on an ordinary Tuesday, May Day and Christmas. Two days of context are shown before the origin. On a Tuesday in February, both TSFMs get the shape of the day right and their intervals enclose reality. The other two panels come back in section 9.\n8. Chronos-Bolt\u0026rsquo;s 90% interval # In the table, Chronos-Bolt has the same coverage at 80% and at 90%: 0.858.\nLeft, for each forecast quantile, the share of hours where reality falls below it: on the diagonal, the model is calibrated. For Bolt, the 0.05 and 0.1 quantiles give the same value. Right, Bolt has only 9 trained quantiles; asked for 0.05 and 0.95, it returns 0.1 and 0.9, hence its 80% interval. Trained on the quantiles 0.1 to 0.9, Chronos-Bolt returns 0.1 and 0.9 when asked for 0.05 and 0.95 (a test in the repository checks it on the real weights). Its \u0026ldquo;90% interval\u0026rdquo; is its 80% interval. I had spotted this by reading the code before running anything.\nYet it lands in the [0.85, 0.95] band because its 80% interval over-covers, and the confidence interval of that coverage (0.833 to 0.884) spills below the band. The rule says \u0026ldquo;wins\u0026rdquo;; I would not sign off a 90% interval on that basis. Chronos-2, trained from 0.01 to 0.99, has a real 90% interval (0.862), but under-covers at 80% (0.761).\nThe conformal repair # Instead of trusting a model\u0026rsquo;s quantiles, conformal prediction rebuilds them from its past errors. The version used here:\nkeep the model\u0026rsquo;s median as the point forecast; for each horizon step \\(h\\), collect the model\u0026rsquo;s absolute errors \\(\\lvert y - \\text{median} \\rvert\\) at the same step over the previous 56 origins, all observed at forecast time; for a 90% interval, add and subtract the 90% quantile of those errors to the median (with the method\u0026rsquo;s finite-sample correction). If future errors resemble past errors, reality falls inside the interval 90% of the time, whatever the model. For Bolt base, coverage goes from 0.858 to 0.902, with a slightly wider interval (0.108 instead of 0.082). I applied it to AutoETS too, so that the repair does not favour one side. Its limit: it calibrates on average over the previous 56 days, not on a rare regime.\n9. By horizon and by regime # Left, the mean scaled error by horizon step. Right, the coverage of the 90% interval by step; the pink area covers steps 13 to 18, 2 pm to 8 pm in Paris in summer. Dashed, Bolt base after conformal repair. At the first step, every model that reads the last hour does well: Bolt base equals MSTL, Chronos-2 makes 77% of its error. The gap opens from step 7, the morning ramp, where the naives and the seasonal ETS are most wrong. The TSFMs\u0026rsquo; coverage drops to 0.79-0.81 on steps 13 to 18, the afternoon and the evening ramp; MSTL\u0026rsquo;s collapses on the morning ramp (0.58 at step 7). The conformal repair, computed step by step, flattens the profile around 0.90.\nEach row is a regime, with its number of origins. The first two panels give the MASE ratio of the TSFM against the seasonal ETS and against MSTL (left of 1, the TSFM is more accurate), with the 95% confidence interval. The third gives the 90% coverage, with the pre-registered band in grey. This forest plot sums up the bench. Against the seasonal ETS, both TSFMs are left of 1 in every regime except holidays and bridge days. Against MSTL they are closer to 1, and a few intervals touch it (Sunday, spring and autumn for Bolt base). Coverage stays within the band or at its edge, except for one row: public holidays.\nThe May Day and Christmas panels above show why. A holiday looks like a Sunday, but nothing in the preceding hours tells a univariate TSFM. Both fell on a Wednesday: it forecasts an ordinary Wednesday, several gigawatts too high, and reality passes below its interval. Over the 11 holidays of 2019, Bolt base has a MASE of 1.49, four times its ordinary level and worse than the daily naive (1.42). Against the seasonal ETS the ratio is 1.12 [0.85, 1.55]: no better. Its 90% interval covers 37% of the hours (Chronos-2: 35%, MSTL: 26%).\nNo model in the bench receives a holiday flag. The difference is that a specialist can be given one, whereas a univariate zero-shot TSFM cannot be told; Chronos-2, however, can (section 11). The two bridge days of 2019 are not enough: below 10 origins, no verdict.\n10. Three robustness checks # Six origins decide nothing when the gap is small # Distribution of the MASE ratio over 2,000 draws of 6 origins among the 365. Against the seasonal ETS, 0.2% of draws reverse the ranking; against MSTL, 24.2%. Against the seasonal ETS (a factor of 2.5), almost every draw of 6 origins gives the right ranking. Against MSTL (a 15% gap), one in four reverses it. Six origins detect a large gap, not a close one. Here, 608 origins of eight models take about six minutes.\nThe lockdown # MASE as a 7-origin rolling mean, January to June 2020. In pink, the stress window after 17 March. Every model is hit for about a week after 17 March, then recovers. The ranking holds: Chronos-2 0.392, Bolt base 0.477, MSTL 0.506, seasonal ETS 0.631, daily naive 0.672. The daily naive\u0026rsquo;s error even falls by a third: with no difference between working days and weekends, copying yesterday becomes a good strategy.\nContext length (exploratory) # Left, the 2019 MASE by context length, with the MSTL and seasonal ETS levels dotted. Right, the 90% coverage. Explored after the results, so outside the protocol. With a single week of context, Bolt base (0.452) already beats MSTL fitted on 8 weeks (0.479). Its best context is 1,024 hours (0.392), not the protocol\u0026rsquo;s 2,048 (0.406); I did not change the protocol, since that choice would be made on the test year. The result foreshadows block 3: a TSFM needs much less history than a specialist.\n11. Telling the TSFM the calendar # Chronos-2 accepts known future covariates, holidays and bridge days among them. For each hour, two 0/1 flags (holiday, bridge day) are passed to the model over the context and the horizon. Same model, same context, no retraining. This part is exploratory: decided after the results, on the test year.\nChronos-2, 2019 MASE cov80 cov90 quantile loss holidays: MASE holidays: cov90 zero-shot, univariate 0.365 0.761 0.862 0.146 1.446 0.352 + holiday and bridge flags 0.331 0.808 0.899 0.130 0.745 0.769 Left, Chronos-2\u0026rsquo;s error by day type, without and with the flags: only the holiday row moves clearly. Centre, the 90% coverage. Right, May Day 2019: with the flags, the median comes down towards reality and the interval encloses it. On holidays the error is halved (ratio 0.52 [0.39, 0.67]) and coverage goes from 0.35 to 0.77. On other days it drops slightly (ratio 0.96): my hypothesis, untested, is that past holidays in the context stop looking like anomalies once flagged. Over the year, the 90% coverage reaches 0.899. \u0026ldquo;Zero-shot\u0026rdquo; therefore does not have to mean \u0026ldquo;univariate\u0026rdquo;. Found on 11 days of the test year, this result motivates a pre-registered test on another year; it does not replace it.\n12. The complexity router, rebuilt on 50 series # The third block asked for architecture reasoning, without code, to put the paper\u0026rsquo;s router into production. So that every figure would be measured, I first rebuilt the router.\nThe panel. 50 European countries and bidding zones from the OPSD package, with less than 0.5% missing hours over 2015-2019; 73 origins in 2019 (one every 5 days, to cover every weekday), a 24-hour horizon. Cold start, a new series with little history, is simulated by letting the models read only the last 1, 2, 4 or 12 weeks: 200 units (series × history length). The TSFM is Chronos-2 zero-shot; the specialist, MSTL [24, 168] refitted at every origin (a seasonal ETS below two weeks of history: a weekly cycle cannot be estimated on one week).\nLeft, each point is a unit: specialist MASE on the x axis, Chronos-2 on the y axis. Every point is below the diagonal. Right, the mean MASE by available history: 1.21 against 0.62 with one week, 0.60 against 0.48 with twelve. The TSFM wins on all 200 units, with gains from 5% to 70%, largest in cold start: with one week, the specialist has a single weekly cycle to fit, while the TSFM brings what it learned elsewhere. On this point the paper is confirmed.\nWhich features predict the gain # A router decides on features computed before the forecast (ex ante), never on the future error.\nThe TSFM\u0026rsquo;s gain over the specialist (in %) against six features, with Spearman\u0026rsquo;s rank correlation. Colour shows the history length, from lightest (1 week) to darkest (12 weeks). History length is the most related to the gain (rank correlation −0.62), followed by the 168-hour and 24-hour autocorrelations; the paper\u0026rsquo;s coefficient of variation and spectral entropy say little here. Several indicators also move with the history length itself. Intermittency and physical constraints, two features in the plan, cannot be tested: national load has neither.\nThe pre-registered policies # Policy mean MASE 90% coverage sent to the TSFM gap to \u0026ldquo;always TSFM\u0026rdquo; [95% CI] always the specialist 0.801 0.745 0% +0.266 [0.243, 0.296] always the TSFM 0.535 0.846 100% 0 paper\u0026rsquo;s rule, published thresholds 0.566 0.833 92% +0.030 [0.014, 0.056] learned stump, leave one country out 0.535 0.846 100% 0 oracle (winner of each unit) 0.535 0.846 100% 0 A decision stump is the simplest tree: one feature, one threshold. Its thresholds are learned on the other countries and applied to the one left out, so that no country scores a rule it helped build. Across the 28 folds it finds no threshold that improves on \u0026ldquo;always the TSFM\u0026rdquo;, and the oracle coincides with it. The paper\u0026rsquo;s rule, with thresholds tuned on Traffic and M4 Daily, sends 8% of the series to the specialist and loses 0.030 MASE out of sample.\nWhat the router can still buy: cost # If the TSFM wins everywhere, a router can only save TSFM calls, at the expense of accuracy. I added, outside the protocol, a stump that pays a penalty per series sent to the TSFM, with a varying penalty.\nMean MASE over the 200 units against the share of series sent to the TSFM. Dotted, random routing; in red, the oracle ranking; in blue, the cost-aware stump, out of sample; the black square is the paper\u0026rsquo;s rule. The stump stays within 0.015 MASE of the oracle between 25% and 100% of TSFM calls. Saving 22% of calls costs 5% of MASE, 44% costs 11%. The paper\u0026rsquo;s rule sits above the random-routing line: on this panel it routes worse than chance.\nIs the saving worth anything? The paper assumes a TSFM call costs a thousand specialist calls. Measured: Chronos-2 takes about 95 ms per series on 4 CPU threads (about 5 ms in a GPU batch), an MSTL refit 183 ms on one core. The assumption holds for a specialist that only predicts, not for one refitted at every forecast. The milliseconds come from a single machine; the ratio is what transfers.\n13. The routing decision drifts # The plan named a \u0026ldquo;senior trap\u0026rdquo;: a router\u0026rsquo;s decision can drift like a model\u0026rsquo;s. To measure it, I took a cost-aware stump tuned to send about two thirds of the series to the TSFM (the operating point of the paper\u0026rsquo;s deployed router), with a rule on the 24-hour autocorrelation, re-evaluated at every origin on the trailing four weeks.\nEach row is a series, each column an origin of 2019: violet for the TSFM, green for the specialist. Right, the share of series sent to the TSFM: from 12% to 78% depending on the period. 40 of 50 series change branch at least once, for 323 flips, and the share sent to the TSFM swings between 12% and 78%. The flips follow holidays and seasons: the German zones switch together at the turn of the year, the British and Danish series in June. Each flip changes overnight the model that produces a series\u0026rsquo; forecast. A hysteresis band (switch branch only when the indicator crosses the threshold by a given margin) cuts these back-and-forths:\nBand on the 24-hour autocorrelation flips in 2019 series that flip max for one series none 323 40 22 ± 0.01 218 38 16 ± 0.02 168 38 16 ± 0.05 78 35 6 Another lesson came from a mistake: my first version fitted the stump on indicators computed over 2,048 hours and applied it to indicators computed over 672 hours. Spectral entropy depends on the window, and the stump sent 4 to 20% of the series to the TSFM instead of about 56%. It is train/serve skew, router edition.\nThe architecture reasoning # Series enter the router, which picks one branch per series and per day. The TSFM and specialist branches are each other\u0026rsquo;s fallback. Weekly monitoring targets the router itself: route mix, flips, shadow forecast of the other branch. The requested deliverable fitted in 8 to 12 lines. With the measurements above:\nHistory length predicts the size of the TSFM\u0026rsquo;s gain, not the winner. No threshold can be justified on accuracy; the plan said that signals a poorly chosen routing feature, or here the absence of a route to choose. By default, the TSFM for every series, with the calendar as a covariate, rather than a branch switch on holidays. The specialist as the fallback: timeout or missing value on the TSFM side; the reverse if the fit fails or the history is under two days; the daily naive, flagged, if both fail. Both branches in daily batch; real time only for intraday re-forecasts, on the TSFM, which has nothing to refit. Every forecast carries its route and its model version. The router is monitored like a model: route mix, flips per series (alert above two a month), hysteresis, shadow forecast of the other branch on 5% of the series. One named owner decides the thresholds; every change is a release evaluated out of sample. No complexity router whose only gain would be a cost the measurement does not show. 14. What I take away # These observations hold for hourly load series, without temperature, against univariate statistical specialists.\nA zero-shot TSFM can beat a refitted specialist on a series it has never seen, and the gap depends mostly on the specialist: a factor of 4 against a flat line, 15% against MSTL. The reference deserves a look before being beaten: AutoETS\u0026rsquo;s one-step criterion does not match a 24-hour horizon. Coverage only reads alongside width: the flat line is the best-calibrated and least useful model in the table. An interval is checked at the source: Bolt\u0026rsquo;s 90% interval is its 80% one, and the conformal repair fixes it on average, not on rare regimes. The calendar is the weakness of univariate zero-shot, and a covariate fixes it without retraining. Cold start is where the gap is largest: half the error with one week of history. A router is only worth the gap it exploits, and here none favours the specialist. The routing decision drifts and needs the same monitoring as a model. Limits and open questions. A single series for block 2 and no temperature, a first-order driver of national load. The specialists are univariate: a specialist with a holiday flag is the next test, the one most likely to reverse some units. Origins are at midnight UTC, whereas a day-ahead market forecast is issued around noon the day before. The calendar effect on Chronos-2 remains to be confirmed on another year, with the protocol declared beforehand.\nA question to check your understanding. A vendor presents a TSFM whose 90% interval covers 91% of the hours on a six-month backtest. Is that enough to put it into production? At least three pieces of information are missing. The width or the Winkler score: a flat line covers 90% too (section 7). Coverage by regime and by horizon: a fine average can hide 37% on holidays or 79% in the afternoon (section 9). And the quantiles the model was trained on (section 8). Reproduce and go further # The code, tests and results are in the gwils28/W25_tsfm_viability repository (Python 3.11, versions pinned with uv); make all rebuilds everything, from the data to the report, GPU optional. The decisions taken before writing code are in SPEC.md, the deviations and corrections in IMPLEMENTATION_NOTES.md, including an independent review that fixed the bootstrap on rare regimes. The online report presents the figures in their original order.\nThe follow-up is a client case: the project page Time series foundation models: forecasting tomorrow\u0026rsquo;s electricity consumption for 3,517 commercial buildings asks the question again with weather and calendar as covariates, RTE\u0026rsquo;s day-ahead forecast as the benchmark, and a cost in euros. Against a LightGBM given those covariates, Chronos-2 no longer wins everywhere, and blending the two beats any router.\nSources\nSoni, Das and Guduguntla, Assessing the operational viability of foundation models for time series forecasting, arXiv, May 2026, and the authors\u0026rsquo; code. Ansari et al., Chronos: Learning the language of time series, 2024; the Chronos-2 technical report, 2025; the Chronos-Bolt base model card. TSFM.ai, IBM\u0026rsquo;s March 2026 time-series refresh, April 2026. Kafritsas, Time series foundation models: a deep dive into strengths and limitations, AI Horizon Forecast, April 2026. Hyndman and Koehler, Another look at measures of forecast accuracy, International Journal of Forecasting, 22(4), 2006; Hyndman and Athanasopoulos, Forecasting: Principles and Practice, 3rd edition, section 5.8. Open Power System Data, time series package, version of 6 October 2020. ","date":"6 November 2026","externalUrl":null,"permalink":"/en/posts/tsfm-zero-shot-specialiste/","section":"Posts \u0026 notes","summary":"When does a foundation model used without any training beat a model fitted on the series? What I observed over a year of French electricity load, with interval coverage and not only the mean error, then rebuilding a router between the two on 50 series.","title":"Time series foundation models: zero-shot against the specialist","type":"posts"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/categories/veille/","section":"Categories","summary":"","title":"Veille","type":"categories"},{"content":"","date":"6 November 2026","externalUrl":null,"permalink":"/en/tags/zero-shot/","section":"Tags","summary":"","title":"Zero-Shot","type":"tags"},{"content":" A coding agent working on its own for an hour needs a machine that doesn\u0026rsquo;t go to sleep, that can be broken without consequence, and that holds none of the production keys. A small VPS meets all three conditions, provided it is set up accordingly. Reading time: about 20 minutes. State of play on 31 October 2026: prices, versions and defaults have changed several times this year, so every figure is dated.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nIn my way of working with Claude Code, the agent runs in my laptop\u0026rsquo;s terminal. That\u0026rsquo;s enough as long as I stay at the keyboard. As soon as a task runs long, you want to follow it from a phone, or run several agents in parallel, the laptop becomes the constraint: it goes to sleep, it holds my keys and documents, and a misplaced rm -rf is expensive there.\nThis article gathers what I took away from looking into the question: the Claude Code documentation, vendor guides (Doppler, Tailscale), hosting price lists and community write-ups. It targets a data scientist or MLOps engineer based in France, hence European hosting. Sources are at the end.\nIn short. The setup that emerges combines three machines with distinct roles:\nA European CPU VPS dedicated to the agent, reachable only over a private network (Tailscale), used through tmux, disposable and backed up by snapshots. A separate production machine, deployed by CI, never by the agent. GPUs rented by the hour, started by a script when training needs one. Secrets are injected at runtime by a manager (Doppler or equivalent), read-only and time-limited. Expect €25 to €130 a month depending on use, agent subscription included, which makes up 70 to 90% of the bill.\nWhy run an agent on a VPS # Need What the VPS brings Long, persistent sessions The agent keeps going when the laptop sleeps; tmux keeps the state Asynchronous work Start a task, walk away, get a notification Mobile access SSH (Termius, Blink) over Tailscale, or Remote Control from the Claude app Isolation from the personal machine An rm -rf or a malicious dependency hits the VPS, not your keys or documents Autonomous mode A disposable environment, restorable from a snapshot Parallelism Several agents in tmux windows or worktrees, without overloading your machine Closeness to production Same Docker images, same architecture Two 2026 changes shift the picture:\nRemote Control (February 2026) lets you drive, from the mobile app or claude.ai/code, a Claude Code session running on the VPS. The session only makes outbound HTTPS requests: no port to open. Limits: one remote connection per session, a terminal that must stay open, and expiry after about 10 minutes without network. Auto mode by default (14 August 2026) replaces the permission prompt at every step with a classifier that reviews actions. In an Anthropic test with 1,053 testers, human review caught 13.6% of dangerous commands, auto mode 89%. permissions.deny and ask rules apply before the classifier. Since v2.1.302, headless runs (claude -p) start in manual mode: you have to pass --permission-mode auto explicitly. Auto mode reduces the need to run the agent with no permissions at all (\u0026ldquo;YOLO mode\u0026rdquo;), but it remains a probabilistic filter. It doesn\u0026rsquo;t replace isolation, which is what the rest is about.\nThe choices that shape the setup # Twelve ideas come up from one source to the next. I treat them as a coherent whole rather than as rules: each answers a specific risk, and you can relax one knowing what you give up.\nThe VPS is a disposable workstation. What matters lives in git, in the secrets manager and in external backups; the machine can be rebuilt in a quarter of an hour. The agent\u0026rsquo;s machine and production are separate. The agent can break its VPS, not production. No public port without a reason. Tailscale or WireGuard for access, Cloudflare Tunnel or a reverse proxy for what must be public. Autonomy is paid for in isolation. --dangerously-skip-permissions only in a container or VM with no valuable secret; elsewhere, auto mode and sandbox. Secrets are injected at runtime, scoped, read-only, time-limited. The agent never fills in its own secrets. The \u0026ldquo;lethal trifecta\u0026rdquo; is broken: private data, untrusted content and external communication are not brought together without a network guard. Persistence is handled at the terminal level: tmux or zellij, mosh for unstable networks, Remote Control as an option. Parallelise with worktrees, not copies: one agent per branch, a human who reviews the diffs. Notifications replace watching: Stop and Notification hooks to ntfy or Telegram. CPU runs all the time, GPU on demand. Renting a GPU around the clock to write code isn\u0026rsquo;t worth it. Deployment stays boring: Compose, a self-hosted PaaS or Kamal, with CI. No improvised docker run by the agent in production. Everything is dated. Prices, versions and defaults change every quarter. The overall architecture # flowchart TD T[\"Phone or laptopTermius · mosh · VS Code Remote-SSH · Remote Control\"] T --\u003e|Tailscale| A subgraph A[\"Agent VPS · EU · no public port\"] direction TB A1[\"non-root user · tmux · worktrees\"] A2[\"Claude Code: auto mode + sandbox\"] A3[\"Doppler: dev_agent config, read-only token\"] A4[\"Docker · uv · MLflow\"] A5[\"hooks → private ntfy · snapshots + restic → S3\"] A1 ~~~ A2 ~~~ A3 ~~~ A4 ~~~ A5 end A --\u003e|\"git push (token scoped to the repo)\"| G[\"GitHub + Actionstests · build · PR review · registry\"] G --\u003e|\"deploy via Kamal / Coolify / Compose\"| P[\"Production VPS · separateCaddy · FastAPI API · database · Uptime Kuma\"] A -.-\u003e|\"start script\"| GPU[\"Hourly GPU · Scaleway L4/L40Straining → MLflow → automatic shutdown\"] The key point of the diagram: no arrow goes from the agent\u0026rsquo;s VPS to production. Code goes through git and CI, which alone holds the deployment key.\nChoosing a VPS # 2026 was a year of price rises, attributed to the cost of RAM and disks. Hetzner raised its cloud prices twice (1 April and 15 June); OVHcloud took its VPS-1 from €4.49 to €6.49 before launching a \u0026ldquo;VPS 2027\u0026rdquo; range; Scaleway and netcup raised prices too.\nEntry-level comparison # Provider Plan Indicative price (Oct. 2026) Location Notes Hetzner CX23: 2 vCPU, 4 GB, 40 GB €5.49/month excl. VAT (excl. IPv4) Germany, Finland Best price per GB of RAM; CX/CAX ranges often out of stock Hetzner CAX11 (Arm): 2 vCPU, 4 GB €5.99/month Germany, Finland Check Arm support for Python and ML wheels OVHcloud VPS-1 2027: 2 vCores, 4 GB, 40 GB NVMe €3.81 to €4.49 excl. VAT depending on commitment France Daily backup included OVHcloud VPS-2: 6 vCores, 12 GB, 100 GB ≈ €11.99 excl. VAT France Good candidate for several agents plus MLflow Scaleway DEV1-M ≈ €14.74 excl. VAT, storage extra Paris, Amsterdam, Warsaw Hourly GPUs in the same account Hostinger KVM 1/2, Claude Code template preinstalled Promotions; renewal ×2 to ×3 Several, incl. EU Simplest to start; official MCP server; average performance DigitalOcean, Vultr, Linode 1 vCPU, 2 GB ≈ $12/month (DO) Frankfurt, Amsterdam, Paris Pricier for the same RAM, excellent API Contabo Large VPS Very low Germany Lots of RAM; uneven I/O and support according to the community Oracle Free Tier Arm Ampere €0 Frankfurt, Paris, Marseille Capacity hit-and-miss; ill-suited as a main workstation Sources disagree on some prices (CX23, the OVH 2027 range): treat them as orders of magnitude and check the official price list before ordering. The renewal price matters more than the promotional one.\nSizing # One agent, web development or light Python: 2 vCPU, 4 GB, 40 GB. Watch Claude Code\u0026rsquo;s memory with htop. Two to four agents in parallel: 4 to 8 vCPU, 8 to 16 GB, 80 to 160 GB NVMe. Data science (pandas, scikit-learn, LightGBM): at least 16 GB. RAM is almost always the limiting factor. With a PaaS on the same machine: Coolify or Dokploy need at least 2 GB more. Disk: Docker images, uv environments and datasets fill 40 GB in a few weeks. A separate volume or S3 storage avoids running out. Location and GDPR # A VPS with a European provider keeps data at rest in the EU, outside US extraterritorial laws. But the code and excerpts the agent reads go to the model\u0026rsquo;s API (Anthropic, OpenAI, Google), wherever the VPS sits: European hosting doesn\u0026rsquo;t settle that transfer. For real personal data, the options are to work on pseudonymised or synthetic samples, to check your plan\u0026rsquo;s data retention (Zero Data Retention options on Team and Enterprise), or to use a self-hosted model.\nGPUs on demand # Option Price (Oct. 2026) Use Scaleway L4 24 GB from €0.79/h Inference of quantised 7-8B LLMs, vision, light LoRA Scaleway L40S 48 GB from €1.46/h Heavier inference, image generation Scaleway H100 PCIe ≈ €2.73/h Serious training Modal per second One-off GPU jobs from Python GPU rented by the month L4 around the clock ≈ €577/month Rarely worth it solo Twenty hours of L4 a month cost about €16, against nearly €580 for a permanent instance. The agent\u0026rsquo;s VPS stays on CPU; the agent writes the scripts that start the GPU instance, run training, log to MLflow, then shut the instance down.\nSetup and security # Basic hardening # a non-root user with sudo; the agent runs as that user, never as root; key-only SSH: PasswordAuthentication no, PermitRootLogin no; UFW denying incoming connections by default; SSH is then only allowed on the tailscale0 interface; fail2ban while SSH is still exposed, unattended-upgrades for security patches; Tailscale or WireGuard for access; Cloudflare Tunnel, Caddy or Traefik on 80/443 for what must be public; a snapshot before every big autonomous session, and external backups (restic or borg to S3). The matching install sequence, for Ubuntu 24.04 (to adapt):\n# --- As root, once --- adduser dev \u0026amp;\u0026amp; usermod -aG sudo dev mkdir -p /home/dev/.ssh \u0026amp;\u0026amp; cp ~/.ssh/authorized_keys /home/dev/.ssh/ \u0026amp;\u0026amp; chown -R dev:dev /home/dev/.ssh sed -i \u0026#39;s/^#\\?PasswordAuthentication.*/PasswordAuthentication no/; s/^#\\?PermitRootLogin.*/PermitRootLogin no/\u0026#39; /etc/ssh/sshd_config systemctl restart ssh apt update \u0026amp;\u0026amp; apt -y upgrade apt -y install ufw fail2ban unattended-upgrades tmux mosh git curl jq bubblewrap socat dpkg-reconfigure -plow unattended-upgrades ufw default deny incoming \u0026amp;\u0026amp; ufw default allow outgoing \u0026amp;\u0026amp; ufw allow OpenSSH \u0026amp;\u0026amp; ufw --force enable # --- Private network --- curl -fsSL https://tailscale.com/install.sh | sh \u0026amp;\u0026amp; tailscale up --ssh # Once the connection over the Tailscale IP is confirmed: ufw allow in on tailscale0 \u0026amp;\u0026amp; ufw delete allow OpenSSH # --- As dev --- curl -fsSL https://claude.ai/install.sh | bash # Claude Code curl -LsSf https://astral.sh/uv/install.sh | sh # Python/uv # Doppler: dev_agent_claude config, read-only expiring token, # then: doppler configure set token \u0026lt;token\u0026gt; --scope . # In the project: /sandbox, deny rules (~/.ssh, push --force, infra/prod) tmux new -s project \u0026#34;doppler run -- claude\u0026#34; The order of the two Tailscale lines matters: public SSH access is only removed once the connection over the private network is confirmed to work, otherwise you lock yourself out.\nAuthenticating the agent # Claude Code installs with the official script, then claude and /login. If OAuth fails for lack of a browser on the server, an ssh -L 15735:localhost:15735 user@vps tunnel from the laptop works. An API key never goes in .bashrc; without a secrets manager, a chmod 600 file loaded by the service.\nSecrets # Doppler\u0026rsquo;s guide for Claude Code spells out a practice that carries over to other managers:\na configuration reserved for the agent, derived from the development environment (for example dev_agent_claude), with only the secrets the task needs; the agent never creates or edits its own secrets; a read-only, expiring service token, scoped to the directory (--scope .); launching through doppler run: secrets become environment variables and vanish when the process ends; Doppler\u0026rsquo;s MCP server, if used, with that same token and --read-only; never live credentials: development and test only. Alternatives are Infisical (open source), 1Password CLI (op run), sops with age, or the network secrets of Claude Code on the web.\nLimit. A secret in an environment variable can still be read by the agent (env). The manager shortens lifetime, narrows scope and prevents file leaks, but doesn\u0026rsquo;t stop a compromised agent from exfiltrating what it sees. Hence network filtering and low-privilege tokens: a GitHub token scoped to one repository, a test database. Prompt injection and exfiltration # In June 2025 Simon Willison named the \u0026ldquo;lethal trifecta\u0026rdquo;: private data, untrusted content, external communication. Meta\u0026rsquo;s \u0026ldquo;Rule of Two\u0026rdquo; (October 2025) follows from it: never more than two of the three in one session. A coding agent combines all three by default: the repository and its secrets, READMEs, issues and dependencies written by others, and curl or git push to talk to the outside.\nConcrete levers to break at least one branch:\nClaude Code\u0026rsquo;s native sandbox (Linux: sudo apt-get install bubblewrap socat, then /sandbox). It isolates files and network through a domain allow-list proxy. Two traps: if the sandbox fails to start, Claude Code carries on without it with a mere warning; and the dangerouslyDisableSandbox escape hatch exists. Check at startup, and forbid it if needed (documentation). @anthropic-ai/sandbox-runtime wraps the whole process, MCP and hooks included, with network and writes denied by default. A container or devcontainer with a network policy and seccomp; for kernel-level isolation, gVisor or a separate VM. Codex CLI: workspace-write mode cuts the network by default; --yolo means --dangerously-bypass-approvals-and-sandbox. MCP hygiene: don\u0026rsquo;t put an MCP that reads outside content (mail, tickets, web) in the same session as an MCP that holds secrets or write access. The isolation ladder # Level Implementation When 0 Agent directly on the VPS, manual mode Getting started 1 Auto mode, bubblewrap sandbox, deny rules Everyday use 2 Agent in a container with no production secrets, filtered network Autonomous mode, long tasks 3 Disposable VM or VPS per project, or microVM (E2B, Firecracker) Untrusted code, several clients 4 Managed cloud sandbox (Claude Code on the web, None or Trusted network) Asynchronous tasks from mobile Level 1 covers most everyday use; you move up a notch when you drop permissions or when the code comes from outside.\nDay-to-day work # Persistence and access # tmux is the standard: one session per project, tmux attach -t project. Dedicated configs (for example tmux-claude-code) enable allow-passthrough on and a high history-limit. zellij as a modern alternative, mosh to survive the phone switching networks. On mobile: Termius, Blink or Echo on the Tailscale address, or Remote Control (claude remote-control or /rc) for chat access from the Claude app. VS Code Remote-SSH or remote Cursor to review diffs and open the VPS\u0026rsquo;s notebooks without exposing JupyterLab. Parallelising # git worktree add ../project-feature-x feature-x gives each agent its own directory and branch, in its own tmux window, with separate dev ports and PRs reviewed by a human. Two agents in the same working directory trip over each other. Subagents parallelise within a task; worktrees, across tasks.\nContext and automation # CLAUDE.md or AGENTS.md: test commands, conventions, prohibitions (\u0026ldquo;never touch infra/prod\u0026rdquo;). In auto mode, this file also feeds the classifier: it is part of the security policy. Hooks: PreToolUse to block, Stop and Notification to alert through ntfy (claude-ntfy-hook), on a hard-to-guess topic or a self-hosted ntfy. MCP: the bare minimum, read-only. A Coolify or Dokploy MCP that drives production from the agent\u0026rsquo;s VPS cancels the separation. Headless: claude -p \u0026quot;…\u0026quot; --permission-mode auto from cron or a systemd timer, for a drift report or a dependency update. Claude Code on the web and routines: claude --cloud and /schedule, with a None, Trusted, Custom or Full network level (documentation). GitHub Actions: the Claude Code action or codex exec for PR review, with minimally scoped tokens. For a data and MLOps profile # Environments: uv and a base Docker image shared by development and production. Data: S3 object storage (Scaleway, OVHcloud); DVC or lakeFS if datasets change. Experiment tracking: MLflow on the VPS (SQLite or Postgres, artefacts on S3), reachable over Tailscale only. The agent logs its runs there, which leaves an auditable trail. Pipelines: Prefect, Dagster, or a Makefile or just. Notebooks: the agent writes tested modules; notebooks stay for exploration, as jupytext or Marimo, easier to edit and to review as diffs. Light serving: FastAPI or BentoML on CPU (scikit-learn, LightGBM, ONNX Runtime); quantised LLMs on hourly GPUs (vLLM). Deploying # Same VPS or a separate one? # The same VPS is defensible for a prototype or staging. Once there are real users or real data, separating machines costs €5 to €10 a month, little against the risk of an agent restarting containers or reading production secrets.\nSelf-hosted tools # Tool Licence Recent version Model For whom Docker Compose + Caddy free — Versioned files, automatic HTTPS Simplest; 1 to 3 ML services Coolify Apache-2.0 v4.4.6 (9 Oct. 2026) Web panel, Traefik, one-click apps A \u0026ldquo;Heroku at home\u0026rdquo; Dokploy Apache-2.0 + proprietary part v0.30.8 (29 Sept. 2026) Web panel, Swarm, Traefik Compose projects, clustering needs Kamal MIT v2.12.0 (18 June 2026) CLI + YAML, SSH deployment, no panel Minimal attack surface CapRover, Dokku Apache-2.0 / MIT — Swarm, Heroku-style Small machines A comparison counts, in 2026, 61 GitHub security advisories for Coolify (16 critical) and 56 for Dokploy (33 critical), against none for Kamal and CapRover. An admin panel therefore belongs behind Tailscale, and kept up to date.\nThe deployment chain # The agent pushes a branch. CI (GitHub Actions) tests, builds the image and pushes it to a registry. CI deploys (Kamal, a Coolify or Dokploy webhook, or docker compose pull \u0026amp;\u0026amp; up -d with a restricted deploy key); the agent doesn\u0026rsquo;t deploy from its VPS. A reverse proxy (Caddy, Traefik or Nginx) serves the application. Observability follows: Uptime Kuma, logs (Loki or docker logs), metrics (Prometheus and Grafana, or Netdata); on the ML side, latency, errors and input drift (Evidently). Production backups go to S3, and restoring them is tested. Alternatives to a VPS # Option Strengths Weaknesses Indicative cost When to choose it VPS dedicated to the agent Control, persistence, fixed price, EU Admin and security are on you €5 to €15/month Daily use, technical profile Local machine Zero latency, local GPU, free Sleep, weak isolation, no async €0 Interactive exploration, with sandbox Home server, mini-PC Lots of RAM, consumer GPU Electricity, residential IP, home network exposed €300 to €1,000 upfront Large RAM or GPU needs Claude Code on the web Nothing to administer, mobile, routines Depends on GitHub, constrained environment Included in the subscription Well-scoped async tasks Codespaces, Gitpod/Ona, Coder Standardised devcontainers, teams Hourly billing, long sessions expensive Variable Teams, onboarding E2B, Daytona microVMs, fast start, SDK Built for products, not a dev workstation ≈ $0.05/vCPU-h Building an agent Modal GPU per second, Python-native CPU sandbox 3 times pricier than functions ≈ $0.14/core-h One-off GPU jobs Fly.io Machines/Sprites Fast start, pay as you go Less EU/GDPR-oriented ≈ $0.04/CPU-h Ephemeral environments For an individual, a VPS remains the most flexible base; managed sandboxes complement it for async work, GPUs or a product.\nWhat it costs # Claude subscriptions in October 2026: Pro at $20/month, Max 5x at $100, Max 20x at $200, Claude Code included with shared limits (a 5-hour window and a weekly cap). Some sources describe a restructuring into \u0026ldquo;Max $100 / Ultra $200\u0026rdquo;: the official price list is the reference.\nScenario Agent Agent VPS Production GPU Indicative total Getting started Pro $20 ≈ €4 to €5.5 — — ≈ €25/month Regular solo data scientist Max 5x $100 8-12 GB, €10 to €15 €5 to €6 20 h of L4 ≈ €16 ≈ €120 to €130/month Heavy use, several agents Max 20x $200 (+ another agent) 16 GB, €20 to €30 ≈ €10 50 h of L4 ≈ €40 ≈ €250 to €300/month Excluding VAT, with $1 ≈ €0.9. The subscription makes up 70 to 90% of the bill: saving a few euros on the VPS matters little, forgetting a GPU that\u0026rsquo;s running matters a lot.\nCommon pitfalls # an agent running as root on the production machine, next to the panel, the database and the secrets; the production .env copied onto the agent\u0026rsquo;s VPS, or keys in .bashrc; a sandbox you believe is active when it disabled itself for lack of bubblewrap or socat; --dangerously-skip-permissions outside a disposable container, or blanket \u0026ldquo;allow\u0026rdquo; rules (Bash(*)); JupyterLab, MLflow, ttyd or the Coolify panel on the public IP, even behind a password; a public ntfy topic carrying code summaries; a GPU instance left running, for lack of a trap or an automatic shutdown timeout; two agents in the same directory instead of two worktrees; backups never tested by restoring them; a VPS picked on its promotional price without looking at renewal; \u0026ldquo;external content reader\u0026rdquo; and \u0026ldquo;secret holder\u0026rdquo; MCPs in the same session; a claude -p cron job assumed to run in auto mode, when it has started in manual mode since v2.1.302. My checklist # Choices\nEuropean provider, at least 4 GB of RAM (8 to 16 GB for data or several agents), renewal price checked. Separate production VPS; GPUs rented by the hour. Hardening\nNon-root user, key-based SSH, root disabled. UFW denying by default, SSH over Tailscale only, fail2ban, unattended-upgrades. No admin service on the public IP. Agent\nbubblewrap and socat installed, /sandbox active and checked. Auto mode (or manual for sensitive work), explicit deny rules, no-permission mode only in a disposable container. CLAUDE.md or AGENTS.md with commands, conventions and prohibitions. Minimal, read-only MCPs, no trifecta. Headless scripts with an explicit --permission-mode. Secrets\nSecrets manager, agent-specific config, read-only expiring token. No production secret on the agent\u0026rsquo;s VPS; minimally scoped tokens. Workflow\ntmux or zellij and mosh, mobile client over Tailscale. One worktree per agent, human review of PRs. Notifications on a private topic. Private MLflow, artefacts on S3, reproducible uv environments. Deployment and operations\nCI that tests and builds; deployment goes through CI. HTTPS reverse proxy, monitoring, logs. Snapshots before autonomous sessions; backups tested by restoring them. Quarterly review of prices, versions and defaults. Caveats # Dates. State as of 31 October 2026; prices, ranges and defaults changed several times this year. Fact or practice. The tmux configs, ntfy tools, Marimo and jupytext, the preference for Kamal and the views on Contabo and Oracle are community practice. The auto mode figures come from Anthropic. Secondary sources. Several prices come from aggregators that sometimes contradict each other; the Coolify and Dokploy security advisory counts come from a single comparison. Coverage. Codespaces, Gitpod/Ona, Gemini CLI, Cursor, OpenCode and Aider were not examined in detail; the isolation, secrets and network principles apply to them in the same way. Sources # Hostinger: Claude plugin Hostinger: installing Claude Code on a VPS Doppler: agents and Claude Code Claude Code: sandboxing Claude Code: sandbox environments Claude Code: secure deployment (Agent SDK) Claude Code on the web Claude Code: cloud environments Claude Code: routines Anthropic: auto mode Help Net Security: auto mode VentureBeat: Remote Control Hetzner: price adjustment stackvaluelab: Hetzner price increase 2026 zen-cloud: OVH price increase 2026 Scaleway: choosing the right GPU Qovery: GPU pricing comparison 2026 The lethal trifecta for coding agents AI coding CLI sandbox modes compared NetworkChuck: remote-claude-code Claude Code on a VPS guide (Medium) pethost: Coolify vs Dokploy vs CapRover vs Dokku haloy: self-hosted deployment tools compared Fly.io: AI sandbox pricing ssdnodes: Claude plans ","date":"3 November 2026","externalUrl":null,"permalink":"/en/posts/agents-code-vps/","section":"Posts \u0026 notes","summary":"A CPU VPS dedicated to the agent, reachable only over a private network, a separate production machine and GPUs rented by the hour: the state of play on 31 October 2026, from isolation to secrets and costs.","title":"A coding agent on a VPS: isolation, secrets and costs","type":"posts"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/tags/ai/","section":"Tags","summary":"","title":"AI","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/tags/ia/","section":"Tags","summary":"","title":"IA","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/categories/method/","section":"Categories","summary":"","title":"Method","type":"categories"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/categories/m%C3%A9thode/","section":"Categories","summary":"","title":"Méthode","type":"categories"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/tags/mlops/","section":"Tags","summary":"","title":"MLOps","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/tags/s%C3%A9curit%C3%A9/","section":"Tags","summary":"","title":"Sécurité","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/tags/tailscale/","section":"Tags","summary":"","title":"Tailscale","type":"tags"},{"content":"","date":"3 November 2026","externalUrl":null,"permalink":"/en/tags/vps/","section":"Tags","summary":"","title":"VPS","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/bsts/","section":"Tags","summary":"","title":"BSTS","type":"tags"},{"content":" Taking a counterfactual validated on synthetic panels to a client case: promotions with no start date, a control pool that cannibalises, zeros that are not zero sales, and a management team that wants to know what share of its promotion budget does nothing. StatusCompleted · 6 blocks, 7 notebooks, local pipeline Timeline22 June – 27 July 2026 Stack Python 3.13causal_tspandasstatsmodelsscikit-learnDoubleMLDagsterMLflowuvpytest Source code 2.4%\nof promotion jumps measurable\n1,592 campaigns out of 67,458 jumps in onpromotion, by a rule fixed before the results 0.83 to 0.90\n95% coverage on planted truth\nBSTS local level on the real series; 0.5 to 0.7 for OLS, synthetic control and DiD 64%\nof the promotion budget unspent\non the lower bound of the intervals, almost no campaign is worth its days Transparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of this page. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThis repository applies the post Causal impact in time series: the counterfactual and its placebos to a client brief. The causal_ts package from the tech watch (counterfactual, placebos, isolation, baselines) is reused as a pinned dependency; everything specific to promotions is new. The repository, its notebooks and its figures are in French.\nThe goal # NovaRetail is a fictional grocery chain that runs promotion after promotion without knowing their incrementality. Management\u0026rsquo;s suspicion: a large share of the sales \u0026ldquo;on promotion\u0026rdquo; would have happened anyway, with pull-forward (the dip after the promotion) and cannibalisation between families on top. The brief asked for a counterfactual per campaign, an uplift with an interval, an elasticity curve per family, placebo backtesting, an orchestrated stack and monitoring.\nThe data is the Kaggle Store Sales competition (Corporación Favorita, Ecuador): 54 stores, 33 families, 1,684 days, a count of items on promotion per day and family, the oil price, holidays, and the earthquake of 16 April 2016 as an exogenous shock.\nWhat I wanted to learn came down to three questions:\nwhat becomes of a design validated on clean panels when the treatment is not defined, zeros have several meanings and the controls cannibalise each other; how to score an estimator on real series, with no known truth, other than by looking at the width of its intervals; what a pipeline and monitoring can watch over a quantity nobody ever observes, a counterfactual. The approach # The work follows six blocks, one folder each, with a results README per block. The code lives in one package (src/novaretail), a CLI nr chains the steps and writes its summaries to data/results/*.json, and the seven notebooks call the package with no logic of their own. The brief and the framing decisions are in plan.md, and the protocol (credibility thresholds, reference model, planted truth) was fixed there before the first result. The working method is the one in Claude Code for data science: what changes in my working method.\n1 Data A dense panel, five kinds of structural zeros (store not open yet, family never sold, closure, family not listed yet, long zero run) set to NaN rather than imputed, a calendar with transferred holidays, paydays and the oil gradient, and a synthetic Favorita-like world with a known counterfactual to build the whole chain before the Kaggle files. 2 Identification The treatment is constructed: a campaign is a jump of onpromotion above its rolling median, for at least three days, preceded by 120 quiet days. The control pool is a registry of rules (declared substitution edges, no promotion in the window, not the same city), never a choice by fit. Then donor isolation by placebo in space. 3 Estimation The causal_ts local-level counterfactual per campaign, effects in units read path by path and split into during, after and net; synthetic control, augmented synthetic control, DiD and OLS on the same interface; Double ML for the response to intensity; the earthquake as a test bench. 4 Backtesting Truth planted in the real series: effects of 0, 5, 15 and 30%, step, ramp and pulse shapes, with or without pull-forward and cannibalisation. Each estimator is scored by its coverage, its false positives at zero effect and its power, not by its width. 5 Pipeline Local Dagster assets (panel, campaigns, estimates, backtest, monitoring, calendar), MLflow as a registry of estimates and their checks, a sha256 manifest of the raw files instead of DVC. No Docker, no cloud. 6 Monitoring and calendar Four signals (pre-period fit, rolling placebo, isolation, donor-target stability), a margin reconciliation on a simulated ledger, and a calendar reallocation at constant budget on the lower bound of the intervals. What the files corrected in the brief # The brief described a different dataset and a different question from what the files allow. Each gap was settled on the data, then written down.\nThe brief says What the data says \u0026ldquo;Favorita omits the zeros, decide whether to rebuild them\u0026rdquo; The file is dense (one row per store, family and day, except Christmas Day). The job is the opposite: sorting zeros that are there. 31% of rows are zero, and 80% of those zeros are structural. \u0026ldquo;Price elasticity curve per family\u0026rdquo; No price in Favorita; onpromotion counts items. The deliverable becomes a response to promotion intensity by Double ML, and is named as such. \u0026ldquo;Treatment: onpromotion \u0026gt; 0\u0026rdquo; Most families are promoted permanently at a low level. A campaign is a jump above the background, and the share of promoted days measurable by this design is the first result. \u0026ldquo;Non-promoted families of the same store as controls\u0026rdquo; That is the case the tech watch showed to be biased (+26% with a cannibalised donor, placebo in time green). The pool is a registry, then an isolation. \u0026ldquo;Credible interval\u0026rdquo; Frequentist prediction intervals, scored by their coverage measured on truth planted in the real panel. \u0026ldquo;Earthquake: perfect exogenous shock\u0026rdquo; A test bench for the method, not a promotion. The donors sold more themselves; the effect is a lower bound. \u0026ldquo;Predicted uplift against realised margin\u0026rdquo; No margin in Favorita. The reconciliation is shown on a simulated ledger with an injected drift. The results # Every number comes from data/results/*.json, produced by the CLI on the Kaggle files.\nHow many promotions are measurable # Out of 67,458 jumps in onpromotion, the detection rule keeps 1,592. The rest is too short (the noise of an item count), falls in a pre-period already taken by another campaign, or overlaps a structural zero. The kept campaigns last 3 to 20 days (deciles), 6 at the median. On a stratified sample of 301 estimated campaigns, 170 pass the credibility protocol (pre-period R², one-step coverage, Ljung-Box, placebo in time, donor isolation), 114 fail and 17 have too few donors.\nLeft, the fate of each jump in onpromotion: 1,592 measurable campaigns, 63,302 jumps too short. Right, measurable campaigns per family: ten families out of 33 have none. What a campaign is worth # Over the 170 credible campaigns, the median lift during the campaign is +1.2%, with deciles at −8% and +13%. 7% of campaigns have a positive lower bound during the campaign, 5% on the net (campaign plus 14 days); 1 to 2% have a negative upper bound. The net sum over the 170 campaigns is −0.6% of the counterfactual. Pull-forward is not measurable at this level: the median share of units taken back after the campaign is −14%, a slight surplus rather than a dip, with intervals that contain zero almost everywhere.\nOne measurable campaign, in the three classic panels. The orange band is the campaign window; the 14 days after it are there to see the pull-forward dip. Intervals from 3,000 jointly simulated paths and 40 parameter draws. What the design is worth # This is the part that matters. On quiet windows of real series, with the real donor registry, a known effect is written into the series, and each estimator is scored on the cumulative effect.\nCoverage of the 95% interval on a +15% step planted in about thirty real windows. Nobody reaches 0.95; the local level is the least far off. Estimator 95% coverage (15% step) False positives at zero effect Power at +5% / +15% BSTS local level 0.83 17% 20% / 40% Augmented synthetic control 0.90 38% 43% / 23% OLS and static BSTS 0.60 41% 37% / 60% Synthetic control 0.80 41% 47% / 27% Difference in differences 0.73 34% 20% / 30% Three readings:\nNobody reaches 95% on real series. The tech watch measured 0.92 to 0.94 for the local level on synthetic panels; here 0.83 to 0.90 depending on the specification. The iid-error estimators sit at 0.5 to 0.7 and fire on nothing four times out of ten. The 17% false positive rate is the reference for the placebos. With eight fake dates, 1.4 firings are expected on nothing; the protocol threshold (at most 2 out of 8) therefore rejects a quarter of campaigns by construction, and that is intended. Power is low. A 5% lift over seven days is detected two times out of ten, barely above the false positive rate; 15%, four times out of ten. This design cannot measure a small promotion on a single store-family series. Share of intervals excluding zero by planted lift, from 0% (false positives) to 30%. The iid estimators detect more because they fire more, including on nothing. The earthquake, as a test bench # The stores of the stricken provinces (Manabí, Esmeraldas) against twenty stores of distant provinces: +39% in units over eight weeks [+32%; +46%], or 572,000 units, with an effect that grows from +10% in the first week to +45 to +55% through the eighth. The placebo in space shows that 5 donors out of 20 have an own effect that excludes zero: emergency relief purchases made the whole country sell more. The effect on the stricken provinces is therefore a lower bound, and the interference is visible because we know where to look.\nLeft, the placebo in time over four fake dates: one fires. Right, the placebo in space: each donor in turn as the target; in red, those whose own effect excludes zero. The response to intensity, and why it does not feed the calendar # Double ML gives a pooled θ of 0.15 to 0.59 depending on what the nuisances contain (lags, months, trend), with tiny standard errors. Holdout 0.36 against 0.34 in training, DoubleML 0.38 as a cross-check, a monotone dose-response by bins. The concordance with the lifts of the 170 measured campaigns is nil (correlation 0.02). The instability is the result: a promotion\u0026rsquo;s date is not \u0026ldquo;as good as random\u0026rdquo; given calendar controls, and DML does not fix that. PRODUCE\u0026rsquo;s θ (5) comes from an onpromotion that is about 200 one day a week and 0 or 1 on the others: a recording regime, not a response.\nMonitoring and calendar # The rolling placebo on never-treated windows exceeds 10% false positives over two thirds of the period: the design is on permanent alert, and that is the right information. With 17% false positives per donor and twelve donors, isolation removes two by noise on 114 campaigns; donor by donor coefficients change sign 223 times between two campaigns of the same target. Signals 3 and 4 are only readable once signal 2 is back under 10%.\nMonthly gap between realised margin and the margin implied by the estimates. Left, a clean ledger: one false alarm in 44 months. Right, a 2% per month drift injected in April 2016: first alarm in July, three months later, as the rule requires. The calendar reallocates campaign days per store, at constant budget, towards the families with the highest value per day on the lower bound. Over 2,327 days and 53 stores, 48 days move (2%) and 1,500 are left unspent (64%): no measured family in the store has a positive value on the lower bound. Ten families out of 33 never have a measurable campaign.\nHistorical and reallocated campaign days per family, all stores. Unmeasured families are unchanged; almost everything else is withdrawn. What got in the way # The treatment was the problem, not the estimator. onpromotion is a count with a permanent background, a weekly spike on PRODUCE and recording regimes that change in 2016. Half the project\u0026rsquo;s time went into deciding what a zero is and what a campaign is. The choice between BSTS and synthetic control, for its part, is settled by coverage. The brief described a different file. The 2017 version of the Favorita dataset omits zero sales; the 2021 version is dense. Three more kinds of structural zeros than planned showed up at load time: 595 of the 1,729 store-family series start more than 30 days after the store opens, PRODUCE has whole quarters at zero in 2014 and 2015, and long zero runs are not intermittency. Simulated paths sometimes diverge. On the log1p scale, a path that runs away weighs millions of units once transformed back. Divergent paths are dropped, counted and checked; a campaign with more than 5% of them is not credible by protocol. Real series cost ten points of coverage. Zeros, extreme values, regime changes: the difference between a clean panel and a file. The RMSE of some estimators comes from a few windows where a donor jumps and a model without a stochastic level extrapolates that jump. Study sizes. 30 windows per backtesting cell, or ±0.1 on each coverage; 300 campaigns estimated out of 1,592 (13 minutes), a single draw of the stratification. The orders of magnitude are stable from one run to the next, the second decimal is not. What I take away # Management\u0026rsquo;s suspicion holds, but the evidence is weak: the net sum of the 170 credible campaigns is zero to −0.6%, and the same design detects a 5% lift planted in these series only once in five. \u0026ldquo;No measurable effect\u0026rdquo; and \u0026ldquo;no effect\u0026rdquo; are two different sentences. A counterfactual is a forecast, and its causal content rests on the design: which campaigns, which donors, which placebos. That design is scored on known truth before producing a number, and the score on real series is lower than on synthetic panels. The control pool is declared, not fitted: the best fit is often the closest substitute. The DML θ are not elasticities, and their sensitivity to the specification says the identification assumption does not hold on this panel. They are reported for that reason. What the method cannot see, nothing here can see: a donor-target relationship that breaks on the campaign date. Only a reconciliation with realised margin or a real holdout settles it. The recommendation to NovaRetail is to measure fewer campaigns and anchor a few of them by experiment, with randomly drawn control stores. Reproduce # git clone https://github.com/gwils28/novaretail-promo-incrementality cd novaretail-promo-incrementality make install # uv sync (Python 3.13), causal_ts as a pinned git dependency make test-fast # unit tests on the synthetic world # drop store-sales-time-series-forecasting.zip (Kaggle) in data/raw/, then: uv run nr ingest \u0026amp;\u0026amp; uv run nr campaigns uv run nr estimate --param-draws 40 # registry, counterfactual, placebos, isolation, verdict uv run nr backtest --n 30 # planted truth, scores per estimator (2 min) uv run nr quake \u0026amp;\u0026amp; uv run nr dml # earthquake (15 s), response to intensity (3 min) uv run nr monitor \u0026amp;\u0026amp; uv run nr calendar # signals, simulated reconciliation, reallocation make notebooks # re-executes the seven notebooks (10 min) Without the Kaggle files, uv run nr ingest --source synthetic runs the same chain on the synthetic world. Data: Corporación Favorita, Kaggle competition Store Sales - Time Series Forecasting.\n","date":"29 October 2026","externalUrl":null,"permalink":"/en/projects/novaretail-incrementalite-promos/","section":"Projects","summary":"The BSTS counterfactual from the tech watch applied to the promotions of a fictional grocery chain on the Favorita data: campaigns detected rather than assumed, a donor registry, coverage measured on planted truth (0.83 to 0.90 against 0.5 to 0.7 for iid estimators), the earthquake as a test bench, Double ML, a Dagster pipeline and monitoring.","title":"Causal impact on time series: measuring what a grocery chain's promotions really bring in","type":"projects"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/causal-inference/","section":"Tags","summary":"","title":"Causal Inference","type":"tags"},{"content":" In late September I described my working method with Claude Code in five rules. One month later, three things have moved: Anthropic\u0026rsquo;s doctrine on what to write to the agent, the default permission mode, and the verification tools. Here is what I keep, what I change, and the files that go with it. Reading time: about 27 minutes. This article complements the first version without replacing it: the principles are explained in detail there, here I only cover what changes. Every feature is introduced the first time it appears.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThe first version held in five rules: context is the scarce resource, CLAUDE.md is short, I explore and plan before coding, nothing is finished without proof, and I remain responsible for every line. None of them has become false. But the way to hold them has changed, because the tool and its documentation have changed. I re-read the official Claude Code documentation (version 2.1.312, as of 27 October 2026), Anthropic\u0026rsquo;s post of 24 July on the new rules of context engineering for Claude 5 models, and the July field guide on finding your unknowns with Fable. This article draws the consequences for a data scientist who codes in the terminal and commits by hand.\nIn short. The five rules remain, and each takes a more precise form:\nContext is measured, not guessed: /context at the start of a task, and I know what survives a compaction. CLAUDE.md describes criteria, not prohibitions: judgment for style, mechanisms for what must never happen. Before planning, I surface my unknowns: blind spots, interview, throwaway prototype, references rather than descriptions. Verification is a ladder: in the prompt, then /goal, a Stop hook, a reviewer in a fresh context, and me. I remain the owner of the code, and it is written in the configuration: deny rules, hooks, manual commit. 1. What has changed since September # The table sums up the six changes that moved my practice. The following sections take them one by one.\nTopic In September As of 29 October 2026 Context philosophy Explicit rules, examples, a CLAUDE.md that gathers everything there is to know Anthropic removed more than 80% of Claude Code\u0026rsquo;s system prompt for the 5 models, with no measured loss. Doctrine: judgment over rules, interfaces over examples, progressive disclosure. /doctor proposes the cuts Permission mode Manual by default, allowlists, sandbox Auto mode is the starting mode of interactive sessions: a classifier reviews non-trivial actions. deny rules remain the only hard guarantee Memory CLAUDE.md and the # shortcut Auto memory on by default, path-scoped rules in .claude/rules/, audit with /doctor prompt-audit Verification Tests and human review /goal (an end condition evaluated at every turn), Stop hooks of type prompt or agent, /code-review in a fresh subagent, /verify Parallelism Subagents, experimental agent teams Subagents in the background by default, forks with /subtask, dynamic workflows written in JavaScript Models The 4 lineage The 5 lineage: Opus 5.5 by default, Sonnet 5.5, Haiku 5.5, Fable 5.1; a one-million-token context; effort levels from low to max The common thread fits in one sentence: the lever is no longer writing good prompts, but assembling the right context at the right moment, and delegating hard rules to deterministic mechanisms. Everything else follows.\n2. Context: measure, don\u0026rsquo;t guess # What comes in before my first question # What I type is only a small part of what the model sees. The rest is assembled by Claude Code, and each piece has its own cost and behaviour. I only knew this roughly; the documentation now details it, and that changes how instructions should be filed.\nElement When it loads What I take from it Claude Code system prompt Always About 4,000 tokens, heavily reduced for the 5 models CLAUDE.md (user, project) and rules without paths Always, at every request Permanent cost: every line is paid at every turn Auto memory (MEMORY.md) Always The first 200 lines or 25 KB, whichever comes first Skills The description only The body arrives only on invocation; the listing is budgeted at about 1% of the window MCP servers Tool names only Schemas stay deferred until use Rules with paths, subfolder CLAUDE.md At the first matching file A local convention costs nothing until its area is touched Hooks Never Zero cost, unless the hook returns content The practical consequence: what must be true at every session goes to the top of the hierarchy, what is only needed sometimes moves down to a skill or a path-scoped rule. That is the principle of progressive disclosure, and section 4 gives the placement table.\nWhat survives a compaction # When the window fills up, Claude Code summarises the history (/autocompact sets the threshold). The summary is useful, but lossy, and I did not know precisely what was lost. The figure shows what I retained.\nAfter a compaction, the system prompt, the root CLAUDE.md, the memory and the plan come back from disk. Recent files and skill bodies come back in part. Instructions given in conversation are lost. Click to enlarge. Two lines of this figure changed how I work.\nInstructions given in conversation are lost. \u0026ldquo;Don\u0026rsquo;t push to main\u0026rdquo;, said out loud at the start of a session, does not survive. A conversational limit is not a rule: if it matters, it goes into a deny rule or a hook. Skill bodies come back truncated, 5,000 tokens per skill and 25,000 in total, oldest evicted first. So the essentials of a SKILL.md go at the top of the file, and details in side files. I am adding a compaction instruction to my CLAUDE.md files, along the lines of: \u0026ldquo;on compaction, keep the list of modified files, the test commands and the modelling decisions taken\u0026rdquo;. And for a long useful session, /compact with a targeted instruction, or /rewind then \u0026ldquo;summarise from here\u0026rdquo; to compact only a part.\nThe cues I use # The thresholds below are not dogma. They are signals that trigger a precise action, and several are new.\nSignal What I do Start of a complex task /context: check that the expected CLAUDE.md is loaded, spot a greedy MCP server or skill Change of topic /clear. A clean session beats a long session full of corrections Two failed corrections on the same point /clear, then a first message that includes what I learned Side question with no follow-up /btw: the answer stays out of the history, zero context growth Broad exploration of a repository An Explore subagent, or a strictly scoped search Context beyond 50 to 60% on a 200,000-token model, or visible noise on one million Write the state to a file, commit, /clear, restart by reading that file The agent ignores a CLAUDE.md rule The file is too long or ambiguous: prune, clarify, or convert to a hook The one-million-token context of the 5 models does not change these reflexes. Size is not what protects, relevance is: degradation with noise is still real, it just arrives later.\n3. From rulebook to judgment # Anthropic\u0026rsquo;s post of 24 July lays out six \u0026ldquo;before / now\u0026rdquo; shifts for Claude 5 models. They have been built into /doctor, the diagnostic command. Here is each one, with what it changes in my files.\nBefore Now What I change Give rules Let the agent judge \u0026ldquo;Never write comments\u0026rdquo;, \u0026ldquo;functions under 30 lines\u0026rdquo; become criteria: \u0026ldquo;code that looks like the surrounding code\u0026rdquo;, \u0026ldquo;reproducibility first\u0026rdquo;, \u0026ldquo;no abstraction that was not asked for\u0026rdquo; Give examples Design interfaces A script whose signature speaks, backtest.py --horizon 14 --quantiles 0.1,0.5,0.9, says more than a paragraph of examples, and does not constrain exploration Put everything at the top Progressive disclosure Short CLAUDE.md; the verification procedure in a skill; infrastructure conventions in a rule limited to infra/** Repeat yourself Describe once Not the same instruction in CLAUDE.md, in the skill and in the prompt. A tool\u0026rsquo;s usage goes in its description, once Memory in CLAUDE.md Auto memory Claude records my corrections and preferences itself; CLAUDE.md keeps what is shared with a team Simple specs Rich references A spec can be a test suite, a function from another repository to port, an HTML mockup, a rubric that a verifying agent will apply The stated reason: absolute rules create conflicts between the system prompt, the skills and the day\u0026rsquo;s request, and the arbitration costs quality; examples constrain exploration more than they steer it.\nGuardrails and style rules are not the same thing # This is the nuance I do not want to lose. \u0026ldquo;Let the agent judge\u0026rdquo; concerns preferences and conventions. It does not remove guardrails. The right split is: judgment for style and technical decisions; deterministic mechanisms for what must never happen, such as touching data/raw/, pushing to main, running terraform apply in production. Rules that cost more than they protect are removed; the others move into the infrastructure. Section 7 shows mine.\nAuditing your setup # Three commands do what I used to do by hand.\n/doctor: for a versioned CLAUDE.md, proposes cuts on what can be derived from the code (tree, dependencies, architecture overview) and keeps the pitfalls, the reasons and the conventions that differ from the defaults. /doctor prompt-audit: looks for instructions written for older models, references to files or commands that no longer exist, and contradictions between CLAUDE.md, rules, skills and subagents. Nothing is changed until you apply. /skill-doctor: the context cost and actual usage of every skill. My CLAUDE.md files run from 29 to 66 lines depending on the repository. The longest is this blog\u0026rsquo;s, because it describes overridden templates and content conventions that nothing in the code explains. The test I apply line by line has not changed: if I remove it, will the agent get it wrong? And I added a non-regression test for rules, suggested by the documentation: a harmless, checkable instruction at the end of the file. If it stops being followed, the instruction budget is saturated.\n4. Where each instruction goes # The question \u0026ldquo;where do I write this?\u0026rdquo; comes up on every project. Claude Code loads ten layers, in a fixed order: organisation CLAUDE.md, personal CLAUDE.md, personal rules, project CLAUDE.md, project rules without paths, rules with paths, a git-ignored CLAUDE.local.md, subfolder CLAUDE.md, auto memory, skills. The files are concatenated, not ranked: two contradictory instructions give an arbitrary choice. Hence a placement table, which I keep up to date.\nIf it is… Then it goes in… A fact true at every session, not derivable from the code: build command, required environment variable, pitfall The project\u0026rsquo;s CLAUDE.md, under 200 lines A convention valid for only part of the tree A rule with paths: in .claude/rules/ A multi-step procedure, knowledge used occasionally A skill, 50 to 300 lines; beyond that, split into reference.md, examples.md, scripts/ A workflow with side effects that I want to launch myself A skill with disable-model-invocation: true: zero cost until I call it A correction I already made in conversation Auto memory captures it; if it is a team rule, I add it explicitly to CLAUDE.md Something that must happen every time, without exception A hook, or a deny / ask rule in settings.json Access to data or an external tool An MCP server, or more often a CLI: gcloud, bq, gh, mlflow A setup to reproduce on a second repository A plugin Auto memory: using it without being ruled by it # In September I described memory as short notes the agent writes when I give it a lasting instruction. It is now on by default, and structured in four types: user (my role, my preferences), feedback (corrections, validated approaches), project (work in progress, decisions), reference (where to find information outside the repository). It lives per repository, in ~/.claude/projects/\u0026lt;project\u0026gt;/memory/, shared between worktrees, never synced to the cloud.\nThis blog\u0026rsquo;s memory holds about ten notes today. Three of them were written after a correction on my part, and that is their value:\narticles aim for 30 minutes on Hugo\u0026rsquo;s reading-time counter; a 40-minute article was judged too long and brought back to 30, by condensing the prose and never the figures; the research duration announced in an introduction comes from the commit dates of the associated repository, not from the initial plan: \u0026ldquo;one day of research\u0026rdquo; became \u0026ldquo;three weeks\u0026rdquo; after reading the git log; never a co-author line in a commit. I re-read these notes from time to time with /memory, and I delete the ones that are no longer true. At a client with confidentiality constraints, memory can be disabled per project. One point that surprised me: subagents do not inherit the main memory, except forks. If a rule must reach them, it is repeated in the delegation prompt.\nSkills: a procedure, not a manual # Commands and skills have merged: a .claude/commands/x.md file and a .claude/skills/x/SKILL.md folder both create /x. I only write skills now. Two real examples, from my repositories.\nThe first prepares a commit without making it. It exists because I want to remain the sole contributor of these repositories: the agent checks that the staged diff covers one topic, runs the fast checks, and proposes a message in the Conventional Commits format. Its description says when to call it, and that is what the agent reads:\n--- name: prepare-commit description: Prepare a commit for the maintainer to run: check the staged diff covers one topic, run checks, and propose a Conventional Commit message. Use when the user asks to commit, prepare a commit, or write a commit/PR message. --- Claude never runs `git commit` or `git push` in this repo: the maintainer must be the sole GitHub contributor. 1. Run `git status` and `git diff --staged` […] 2. If the staged diff mixes topics, stop and propose how to split it. 3. Run `make check-fast` and report. Do not propose a commit while it fails. The second frames a database migration, and starts with the failing test. The rest is the conventions the agent cannot guess: money as Numeric(14, 2), half-open timestamptz UTC intervals, never a Float.\nSizing: under 500 lines for SKILL.md, the essentials at the top because of truncation on compaction, scripts in scripts/ where they are executed without ever entering the context.\n5. Surfacing the unknowns before planning # This is the clearest addition to my loop. The July field guide on Fable describes the quality of the agent\u0026rsquo;s work as limited first by how the user clarifies their unknowns: what they know and say, what they know they don\u0026rsquo;t know, what they will only recognise on seeing it, and what they have not thought of. In data science the fourth category is the most expensive: it is sales censored by stock-outs, the variable computed after the fact, the calendar that changes mid-history.\nSeptember\u0026rsquo;s loop had five beats. It has six.\n1 Explore In plan mode, the agent reads and answers without changing anything. I scope the exploration (\u0026ldquo;read forecast/cv/ and features/base.py, explain how rolling origins are computed\u0026rdquo;) or delegate it to a subagent if it is broad. 2 Surface the unknowns Before writing a line: a \u0026ldquo;blind spots\u0026rdquo; pass (\u0026ldquo;what considerations have I probably not thought of for this capacity-constrained pricing?\u0026rdquo;); then, for any non-trivial feature, an interview, one question at a time, prioritising those whose answer would change the architecture, ending in a SPEC.md. The next session, a clean one, executes the spec. 3 Plan I ask for a plan and annotate it in the editor (Ctrl+G), then \u0026ldquo;address all my notes, don\u0026rsquo;t implement yet\u0026rdquo;. The plan survives compaction. For a change that fits in one sentence, I skip this step. 4 Code Asking explicitly for verification: \u0026ldquo;implement from the plan, write the rolling_origin tests, run make test-fast and fix\u0026rdquo;. And an implementation notes file, where the agent records its deviations and conservative choices. 5 Verify The ladder of section 6. 6 Commit, myself After reviewing the diff. Before signing a change I did not write, I ask for an explainer and get quizzed: \u0026ldquo;what happens if the series has fewer than 14 points?\u0026rdquo;. If I cannot answer, I am not ready to commit. Two practices from the guide seemed counter-intuitive, then useful. The throwaway prototype before the real code: a dashboard mockup or a fifty-line notebook reveals the criteria one can only formulate on seeing them. And references rather than descriptions: pointing at existing code, even in another language, a test suite or an HTML mockup. Code is the language the agent reads best; a mockup is a better spec than a paragraph.\n6. Verification: a ladder, not a step # \u0026ldquo;Nothing is finished without proof\u0026rdquo; remains the rule. What has changed is that the documentation now offers an official gradation of the means to hold it, and several rungs are new.\nFive levels, from the least to the most demanding. The first three close the loop without me; the last two are where I decide. Click to enlarge. In the prompt # Every request contains the means to prove it is fulfilled: \u0026ldquo;write the function, here are three test cases, run the tests afterwards\u0026rdquo;. That was already the case in September. It remains the most profitable rung, because it costs nothing to set up.\n/goal: an end condition evaluated at every turn # New. /goal \u0026quot;make test exits 0 and MASE on backtest \u0026lt;= 0.82, or stop after 20 turns\u0026quot; hands a small model the job of judging the condition after every turn; the agent continues until \u0026ldquo;reached\u0026rdquo; or \u0026ldquo;impossible\u0026rdquo;. The rules that work: one measurable condition, a named check, a bound on turns. It is the tool for unattended work, coupled with auto mode. I have not used it on a real task yet; the first documented session (condition, turns, evidence) is the first item on my November list.\nThe Stop hook: a deterministic gate # This one I have had from the start, and it bears the rule\u0026rsquo;s name. On my mentoring repository, which holds several Python projects, it re-runs pytest in every project whose src/ or tests/ changed since the last commit, and refuses to hand back control while a test fails:\n#!/usr/bin/env bash # Stop: \u0026#34;nothing is finished without proof\u0026#34;. For every project whose src/ or # tests/ changed since the last commit, run pytest; if a test fails, Claude # cannot stop. input=$(cat) [[ $(jq -r \u0026#39;.stop_hook_active // false\u0026#39; \u0026lt;\u0026lt;\u0026lt;\u0026#34;$input\u0026#34;) == \u0026#34;true\u0026#34; ]] \u0026amp;\u0026amp; exit 0 # … detection of modified projects via git status --porcelain … for p in \u0026#34;${!projets[@]}\u0026#34;; do sortie=$(cd \u0026#34;$p\u0026#34; \u0026amp;\u0026amp; uv run --quiet pytest -q 2\u0026gt;\u0026amp;1); code=$? # 5 = no test collected: nothing to prove, not a failure (( code != 0 \u0026amp;\u0026amp; code != 5 )) \u0026amp;\u0026amp; echecs+=$\u0026#39;\\n\u0026#39;\u0026#34;### $p\u0026#34;$\u0026#39;\\n\u0026#39;\u0026#34;$(tail -n 25 \u0026lt;\u0026lt;\u0026lt;\u0026#34;$sortie\u0026#34;)\u0026#34; done [[ -n \u0026#34;$echecs\u0026#34; ]] \u0026amp;\u0026amp; jq -n --arg r \u0026#34;Tests fail: fix before finishing.$echecs\u0026#34; \\ \u0026#39;{decision: \u0026#34;block\u0026#34;, reason: $r}\u0026#39; The detail that matters: pytest\u0026rsquo;s return code 5 (no test collected) is not a failure, otherwise a freshly initialised project could never hand back control. And the hook reads stop_hook_active so as not to block itself. Stop hooks now come in three types: command, prompt (a yes / no question put to a model, \u0026ldquo;are all tasks complete?\u0026rdquo;) and agent (a multi-turn subagent, experimental). I stay with command: a passing test suite is proof; a model\u0026rsquo;s \u0026ldquo;yes\u0026rdquo; is not.\nThe reviewer in a fresh context # An agent that has just written code is badly placed to criticise it. /code-review launches a fresh subagent that only sees the diff; /code-review ultra launches several in the cloud. My home-made reviewer is a file in .claude/agents/, read-only, which reports by decreasing severity: correctness (units, timezone-naive dates, half-open intervals, look-ahead in anything date-based), numbers without provenance, respect of layers, missing tests, and several topics mixed in one diff. Its last line is the most important: \u0026ldquo;if nothing is wrong, say so plainly; no style nits that ruff already enforces\u0026rdquo;.\nA reviewer ordered to find gaps will find some, even on sound work. I ask it to report only what touches correctness or the stated requirements, and to treat the rest as optional. Otherwise one drifts into over-engineering: abstraction layers, defensive code, tests of impossible cases.\nMe # /verify builds and launches the application to confirm that a change works in the real product, not only in the tests. Then I re-read the diff in VS Code and ask for the evidence, test output and command run, rather than assertions of success. It also works for sessions I did not watch.\nTest first, in two sessions # The agent still writes the implementation first by default, and the wording that works has not changed: \u0026ldquo;write a test that fails for X, implement nothing, run it and show me the failure\u0026rdquo;, then \u0026ldquo;implement the minimum to make it pass, without touching the test\u0026rdquo;. The documentation now recommends a structural variant that I am keeping for the functions that matter: two sessions. Session A writes the tests; session B implements, without having seen the discussion about the test. The implementation cannot \u0026ldquo;bleed\u0026rdquo; into the test logic. With two terminals and --worktree, it is five minutes of setup.\n7. Guardrails: auto mode and what must never happen # Permission modes # Since version 2.1.283, interactive sessions start in auto mode: everything runs, under the control of a background classifier (Sonnet 5 by default) that reviews non-trivial actions. By default it blocks curl | bash, sending sensitive data outside, production deployments and migrations, git push --force, git reset --hard, terraform destroy, granting IAM permissions, displaying secrets. After three consecutive blocks, it pauses and asks again.\nMode What runs without asking When I use it default (manual) Reads only First contact with a client repository, real data acceptEdits Reads, edits, file commands inside the working directory Local refactors I follow on screen plan Reads; no edit until the plan is approved Exploration and planning auto Everything, under the classifier\u0026rsquo;s control Everyday work, /goal, unattended runs dontAsk Only what is pre-approved; everything else is refused claude -p batches in a loop bypassPermissions Everything, with no control An isolated container without internet, nowhere else What stays true whatever the mode: deny rules apply everywhere, even in bypass; protected paths (.git, .claude, .mcp.json, .bashrc) are never auto-approved; and rm on the root, the home or the working directory can be allowed by no rule. The documentation says it itself: auto mode reduces prompts, it does not guarantee safety. Hence a sandbox (/sandbox) or a devcontainer on top, for defence in depth.\nMy configuration # On my application repositories, settings.json carries the same deny list, and it says what I do not delegate:\n\u0026#34;permissions\u0026#34;: { \u0026#34;deny\u0026#34;: [ \u0026#34;Bash(git commit:*)\u0026#34;, \u0026#34;Bash(git push:*)\u0026#34;, \u0026#34;Bash(git rebase:*)\u0026#34;, \u0026#34;Bash(git reset --hard:*)\u0026#34;, \u0026#34;Bash(gh pr merge:*)\u0026#34;, \u0026#34;Read(./.env)\u0026#34;, \u0026#34;Read(./.env.*)\u0026#34;, \u0026#34;Edit(./data/**)\u0026#34;, \u0026#34;Write(./data/**)\u0026#34; ] } The commit is mine, merging a pull request too, secrets are never read, data is never modified by the agent. Three hooks complete the list, because the agent can also modify files through the shell:\na PreToolUse on Bash, which inspects only git invocations and refuses co-author trailers, --no-verify and any change to core.hooksPath; a PreToolUse on edits, which refuses data/raw/ with a readable reason (\u0026ldquo;raw data is downloaded and verified by SHA-256 by the project\u0026rsquo;s code; write to data/processed/\u0026rdquo;); a PostToolUse that runs ruff format on every modified Python file, from the uv project that contains it. This blog, on the other hand, has neither settings.json nor hooks. The stakes are low, the proof is the build, and when I explicitly ask for a commit and a push, the agent does them, without a co-author line. The level of guardrail follows the stakes, not a general rule: at a client, I start in manual mode, enable the blocking of reads outside the working directory, and add ask rules as checkpoints on git push, terraform apply, dvc push and deployments.\nUnattended # claude -p \u0026quot;…\u0026quot; runs without an interface, in CI for instance. Two changes since September: non-interactive mode now starts in manual (version 2.1.302), so the wanted mode must be stated; and --bare ignores local hooks, skills, MCP servers and CLAUDE.md, to make the run reproducible. My read-only pull request review:\nclaude -p \u0026#34;…\u0026#34; --output-format json --allowedTools \u0026#34;Read,Grep,Glob\u0026#34; --max-turns 10 \\ --permission-mode dontAsk --bare 8. Delegating: subagents, forks, workflows # September\u0026rsquo;s section 7 table still holds, with three more mechanisms. I give it as a decision table.\nNeed Mechanism Why that one The agent gets a convention wrong twice CLAUDE.md Permanent fact, not derivable from the code The same prompt retyped a third time Skill On demand, with scripts executed without entering the context A side task floods the conversation Subagent Isolated context; only the summary comes back The same processing on dozens of items, or results to cross-check Dynamic workflow A script decides, not the agent turn by turn; the context only keeps the final result Something must happen every time Hook Deterministic A second repository needs the same setup Plugin Bundles skills, hooks, subagents and MCP; testable with claude plugin eval What has changed for subagents # They run in the background by default, /tasks shows what is running, and a subagent only sees its own system prompt, the task, the CLAUDE.md hierarchy and the git state. Not the history, not the memory, not the output style; the Explore and Plan subagents even skip CLAUDE.md. Forks are new: /subtask creates a subagent that inherits the whole conversation and reuses the prompt cache, often cheaper than a blank subagent when context matters. A blank Explore is only worthwhile if the clean window is the goal. And the agent rarely delegates on its own: I ask (\u0026ldquo;use a subagent to…\u0026rdquo;) or write it in CLAUDE.md.\nThe useful fields in a subagent definition: model (haiku for cheap exploration, opus or fable for review), effort, memory: project, maxTurns, isolation: worktree. My read-only statistician from September has not changed by a line.\nDynamic workflows # A workflow is a JavaScript script that the agent writes and a runtime executes in the background: agent(prompt, {schema}), pipeline(), parallel(). Intermediate results stay in the script\u0026rsquo;s variables; the main context only receives the conclusion. It is triggered by \u0026ldquo;use a workflow\u0026rdquo; or the ultracode keyword, and /workflows lets you follow, pause and save a script to replay it.\nThe cases that concern me: an audit of temporal leaks across a whole repository, with adversarial verification (one agent finds, another tries to refute); the migration of dozens of notebooks into modules. I have not used it on a real repository yet, and I will start on a small slice. Agent teams remain experimental; I do not use them.\nModels and effort # The 5 lineage replaces the 4 lineage. My main setting is Fable 5.1, for long sessions and ambiguous root causes; Opus 5.5, the default, is enough for planning and review; Haiku 5.5 suits read-only subagents and the /goal evaluator. The opusplan alias (Opus in plan mode, Sonnet in execution) is a good cost / quality compromise. For effort, I start low and go up if the answer is shallow; max stays reserved for architecture decisions and hard debugging.\n9. On the job: forecasting, MLOps, GCP # The .claude/ tree of a forecasting project # Here is how I lay out a forecasting repository today, applying the placement table of section 4:\n.claude/ ├── CLAUDE.md # \u0026lt; 200 lines: goal, commands, pitfalls, way of working, compaction ├── rules/ │ ├── temporal-safety.md # no paths: golden anti-leak rule, splits, horizons │ ├── infra.md # paths: [\u0026#34;infra/**\u0026#34;, \u0026#34;*.tf\u0026#34;] │ ├── notebooks.md # paths: [\u0026#34;notebooks/**\u0026#34;]: throwaway, no business logic │ └── serving.md # paths: [\u0026#34;src/serving/**\u0026#34;]: FastAPI, latency, schemas ├── skills/ │ ├── verify-forecast/ # tests + backtest + comparison to baseline │ ├── eda/ # reproducible EDA (scripts/profile.py) │ ├── new-experiment/ # disable-model-invocation: branch + config + MLflow run │ └── causal-check/ # DAG, sanity checks before any causal conclusion ├── agents/ │ ├── data-explorer.md # haiku, read-only, bq query --dry_run │ ├── forecast-reviewer.md # opus, read-only │ └── terraform-engineer.md # plan / validate, never apply ├── hooks/ │ └── protect-data.sh # PreToolUse: blocks data/raw/, models/, .env └── settings.json # deny / ask / allow permissions, hooks, sandbox What is not in CLAUDE.md: the tree (derivable), the dependency list (in pyproject.toml), the theory of MASE (the model knows it), the detailed verification steps (in the skill). The verify-forecast skill fits in four steps: the fast suite must be green, otherwise stop; run the reference backtest; compare MASE and pinball to the figures in the versioned baseline.json; report the deltas, the modified files and the commands run. And one clause that comes from practice: do not conclude \u0026ldquo;improvement\u0026rdquo; if the empirical coverage of the 0.9 quantile leaves [0.85, 0.95].\nWhat the agent really did # The new point on the MLOps side is the traceability of the sessions themselves. MLflow provides mlflow autolog claude, which installs hooks that trace every tool call as a span, with tokens and cost. For a data scientist this is a trace of truth: what the agent did, in order, with inputs and outputs, readable after the fact and queryable from the command line. It is the second item on my November list.\nFor the rest, September\u0026rsquo;s choices hold: BigQuery read-only (every query is a billed job), bq --dry_run in the allowlist and real execution under ask; Terraform in a generate, validate, plan, read, fix loop, with a rule limited to infra/** for the prohibitions (no 0.0.0.0/0, mandatory encryption, least privilege), apply under ask and destroy under deny. And still no Claude Code as a production orchestrator: dynamic workflows serve one-off bulk processing, Prefect or Airflow serve production.\nMockups as specs # Claude Design has been integrated into Claude Code since 16 September (/design); the standalone site closes on 14 December 2026, and projects are migrated before then. For my use, the point is the \u0026ldquo;rich references\u0026rdquo; rule: a forecasting dashboard mockup, with quantile bands and drift alerts, shown to the client before writing a line of front-end, then given to the agent as the spec. It complements Figma, it does not replace it.\n10. What I keep, what I add # To stay honest about the real state of my practice on 29 October, here is where I stand.\nIn place In progress To come Files CLAUDE.md of 29 to 66 lines; skills; statistician and reviewer subagents; memory per repository Compaction instruction in every CLAUDE.md; a pass of /doctor and prompt-audit Path-scoped rules (infra/**, notebooks/**) on the next forecasting project Guardrails deny on commit, push, rebase, merge, .env, data/; PreToolUse, PostToolUse, Stop hooks Sandbox for everyday work in auto mode Manual mode and blocking of reads outside the directory at a client Verification In the prompt; test Stop hook; reviewer in a fresh context; test first Systematic /code-review before commit; implementation notes and quiz One complete documented /goal session; two-session tests on a function that matters Delegation Read-only subagents Forks with /subtask First dynamic workflow on a slice: temporal leak audit Traceability Manual commits, README \u0026ldquo;what I correct relative to the plan\u0026rdquo; MLflow tracing of sessions on one project What I do not delegate has not moved: the choice of the question, the protocol fixed before the results, the interpretation, irreversible operations, publication. The difference is that these rules no longer rest on my vigilance alone: three of them are in a settings.json.\nThe signals that make me change course gained three lines.\nSignal What I do The agent ignores a rule despite insistence Prune CLAUDE.md; convert to a hook if it is an \u0026ldquo;always / never\u0026rdquo; Quality drops during a session /context; document and /clear A subagent comes back with a wall of text Tighten its description and the requested output format I retype the same procedure A skill Auto mode blocks repeatedly Read the rule named in the reason; adjust a precise allow rule, or report the false positive I cannot explain a change Do not commit; ask for an explainer and a quiz; re-read the diff Conclusion # In September I concluded that Claude Code had made my way of working more demanding. One month later, the demand has moved. It bears less on what I write to the agent, since the 5 models do without most rules, and more on what I put around it: a context that is measured and filed in layers, mechanisms that do not depend on its judgment, and verification in several levels of which I am the last.\nIf I had to keep one sentence from this revision, it would be this one: a limit said in conversation is not a rule. What must never happen goes into a file the agent cannot ignore. The rest, it judges better than my lists of prohibitions.\nThe tools change every week; ten minutes a month on the changelog and a /doctor are enough to keep up. What does not change: in the end, the code bears my name.\nFurther reading\nBest practices for Claude Code, the canonical reference, updated in place. T. Shihipar, The new rules of context engineering for Claude 5 generation models, Anthropic, 24 July 2026: the current doctrine, to re-read before any CLAUDE.md overhaul. T. Shihipar, A field guide to Claude Fable 5: finding your unknowns, Anthropic, 6 July 2026: interviews, prototypes, implementation notes, quizzes. A harness for every task: dynamic workflows in Claude Code, Anthropic, June 2026. Documentation: How Claude remembers your project, Explore the context window, Choose a permission mode, Automate actions with hooks, Keep Claude working toward a goal. MLflow × Claude Code: session tracing. ","date":"29 October 2026","externalUrl":null,"permalink":"/en/posts/claude-code-methode-revisee/","section":"Posts \u0026 notes","summary":"The lever is no longer the prompt but the context: what goes in, when, and what is handed to mechanisms that do not depend on the model. My method, revised as of 29 October 2026.","title":"Claude Code for data science: what changes in my working method","type":"posts"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/context-engineering/","section":"Tags","summary":"","title":"Context Engineering","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/tags/contrefactuel/","section":"Tags","summary":"","title":"Contrefactuel","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/counterfactual/","section":"Tags","summary":"","title":"Counterfactual","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/data-science/","section":"Tags","summary":"","title":"Data Science","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/double-ml/","section":"Tags","summary":"","title":"Double ML","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/tags/inf%C3%A9rence-causale/","section":"Tags","summary":"","title":"Inférence Causale","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/ml-engineering/","section":"Tags","summary":"","title":"ML Engineering","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/placebo-tests/","section":"Tags","summary":"","title":"Placebo Tests","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/en/tags/retail/","section":"Tags","summary":"","title":"Retail","type":"tags"},{"content":"","date":"29 October 2026","externalUrl":null,"permalink":"/tags/tests-placebo/","section":"Tags","summary":"","title":"Tests Placebo","type":"tags"},{"content":" What was the effect of a promotion, a price change, an announcement, on a series that could not be randomised? The usual answer is a counterfactual: a forecast of what the series would have done without the intervention. Building one by hand, what I mostly learned is what that forecast cannot say on its own, and what placebo tests can check about it. Reading time: about 30 minutes. Every formula is followed by its intuition, so you can skim the maths without losing the thread.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nUpdate, 29 October 2026. This research has since been applied to a fictional client case, the promotions of a grocery chain on the Favorita panel: Causal impact on time series: measuring what a grocery chain\u0026rsquo;s promotions really bring in. The project page shows what becomes of the design on real series, with campaigns to detect and a control pool to declare. This article distils three weeks of hands-on research into causal inference for time series, announced at the end of the article on retraining frequency. The starting plan had four blocks: two recent papers to read at the source, a counterfactual to build and validate by placebo, an architecture decision for a continuous incrementality measurement, and the founding paper by Brodersen and colleagues to reread. Its success criterion: a real effect whose interval excludes zero, a placebo effect whose interval contains zero. I met it, then looked for what it actually guarantees, on a panel where the truth is known. Everything is reproducible in the time_series_causal_statistics repository.\nIn short. The main results, with their scope:\nThe plan\u0026rsquo;s criterion is met on a known truth and on Volkswagen: −20.9% [−33.9%, −1.7%] on VW\u0026rsquo;s share price after the EPA notice, twelve fake dates all clean. The placebo-in-time only validates the pre-intervention model. On 100 panels where a control is contaminated by the intervention, it is green as often as on clean panels. Contamination needs the placebo-in-space; a relationship that breaks at the intervention date is seen by nothing. A placebo that \u0026ldquo;finds\u0026rdquo; an effect is partly a property of the placebo itself: with a shortened pre period, it fires on 11 to 17% of clean panels, for a nominal 5%. Static estimators (OLS on the controls, convex synthetic control, difference-in-differences) are as accurate as the local level on the point estimate, and six to seven times too confident: 80% false positives at zero effect. The Python port of Google\u0026rsquo;s CausalImpact gives the same point estimates with intervals half as wide. That is its default prior on the level standard deviation, not the data: on 30 panels with a known truth, it covers 77% of the time, against 90% by hand. 1. The problem: one unit, two worlds # The causal effect of an intervention on a unit at time \\(t\\) is the difference between what is observed, \\(y_t\\), and what would have been observed without the intervention, \\(y^0_t\\). The second quantity no longer exists once the intervention has happened: a counterfactual analysis is therefore a forecast of \\(y^0_t\\) over the post period, and the effect is the gap between the observed series and that forecast.\nOn the synthetic panel the counterfactual y0 is known (green): its difference from the observed series (blue) is the true effect, here a 10% lift planted over 90 days. In practice only the blue curve exists. The whole point is to make that forecast credible and honest about its uncertainty. But a good forecast is not a causal model. The counterfactual is built from control series: comparable units, untouched, whose relationship with the treated unit, learned before the date, is extrapolated after it. Three assumptions carry the result, and none of them is visible in the quality of the pre-intervention fit:\nAssumption What it means for one treated series What tests it the intervention date is \u0026ldquo;as good as random\u0026rdquo; given the past no anticipation, no simultaneous event specific to the treated unit placebo-in-time: a fake date inside the pre period must give an effect whose interval contains zero the controls are not affected, directly or by spillover no interference: the promotion on A does not cannibalise B, used as a control placebo-in-space: each control treated as if it were the unit of interest, at the real date the pre-period relationship persists the coefficients linking controls and target still hold after the date stability diagnostics on the pre period, and coverage on a panel with a known truth The protocol: fit a structural model on the pre period only, forecast the counterfactual with a 95% interval, then rerun the same analysis with a fake date placed inside the pre period. I applied it on a semi-synthetic panel with a known truth, then on the Volkswagen dieselgate panel.\n2. What the recent papers say # Time-Aware Synthetic Control (Rho, Illick, Narasipura, Abadie, Hsu and Misra, arXiv 2601.03099, January 2026). Classical synthetic control treats the pre-period time indices as interchangeable; TASC encodes time through a linear Gaussian state-space model learned by EM. It wins when observation noise is high and loses to robust synthetic control when it is low. Two gaps with the plan: Alberto Abadie is the fourth of six authors, not the lead signatory; and the \u0026ldquo;narrower intervals\u0026rdquo; than CausalImpact are a claim about width, with no coverage reported anywhere. Part 9 measures what a narrow interval can cost.\nWhen are time series predictions causal? (Carlson and Shephard, arXiv 2603.20394, March 2026). The conceptual paper. Its central theorem fits in one sentence: under a matching assignment condition, a difference of predictions is a causal effect. Without that condition, it is a difference of predictions. The table in part 1 is my translation of that condition for a single treated series.\nBrodersen, Gallusser, Koehler, Remy and Scott (2015), Inferring causal impact using Bayesian structural time-series models, Annals of Applied Statistics. A state space assembled block by block (local level, trend, seasonality, regression on the controls), priors on the variances and coefficients, inferred by Gibbs sampling; the predictive distribution is defined jointly over the post period so that serial structure propagates into the cumulative effect. Three things served me: their simulation study grades the method by two curves, power and coverage; their third analysis applies the model to the untreated regions and demands a null result, which is the placebo-in-space; and their three named assumptions: covariates unaffected by the intervention, a pre-period relationship that persists, a model structure that stays intact. The plan quoted only the first.\n3. The model: a local level and a regression on the controls # I built the model by hand with statsmodels, to know what is inside, and kept Google\u0026rsquo;s port as a cross-check (part 9). It is Brodersen\u0026rsquo;s without its Bayesian layer:\n$$ y_t = \\mu_t + \\beta' x_t + \\varepsilon_t, \\qquad \\mu_{t+1} = \\mu_t + \\eta_t, \\qquad \\varepsilon_t \\sim \\mathcal N(0, \\sigma^2_\\varepsilon),\\ \\eta_t \\sim \\mathcal N(0, \\sigma^2_\\eta). $$The intuition. The regression \\(\\beta' x_t\\) explains the part the treated unit shares with its controls; the level \\(\\mu_t\\), a random walk, absorbs what the target does on its own. Both variances are estimated by maximum likelihood on the pre period only; the counterfactual is the Kalman-filter forecast over the post period, simulated as joint paths for the cumulative effect\u0026rsquo;s interval. Three models on the same panel:\nstatic local level local trend one-step RMSE (pre period) 3.10 2.94 2.98 one-step R² 0.57 0.61 0.61 Ljung-Box on the residuals (p) 0.000 0.94 0.90 estimated cumulative effect (truth 831) 681 [617, 748] 569 [207, 926] 530 [120, 947] The residuals of the static regression are autocorrelated: a slow component remains, which the local level absorbs. But the real difference is in the forecast: the static model assumes the unexplained part is independent noise every day, the local level assumes it can wander.\nSame panel, same pre period. The static regression (left) gives a narrow interval that misses the truth; the local level (centre) a wide interval that contains it; the local trend (right) a wider one still. The forecast variance of a static regression is constant. A local level adds a variance linear in the horizon \\(h\\), hence an interval in \\(\\sqrt h\\); a local linear trend adds a random slope and the variance grows like \\(h^3\\). For the cumulative effect these powers go up by two.\nWidth of the 95% interval on one day as a function of the horizon: flat for the static regression, square root of h for the local level, faster still for the local trend. On this panel the static interval misses the truth and the local level\u0026rsquo;s contains it. Which is right is a matter of coverage, which part 5 measures over 100 panels.\nA second source of uncertainty. The forecast at the maximum-likelihood estimate ignores that the parameters are themselves estimates. CausalImpact integrates them out by MCMC; the cheap frequentist counterpart is to draw them from the asymptotic normal of the MLE and mix the forecast over those draws. With 365 pre-period days, the cumulative interval barely moves: [207, 926] becomes [203, 914].\n4. The protocol on a known truth # The panel: one treated unit and five controls, 365 days before the intervention and 90 after. All units load on two latent factors (random walks) and a weekly pattern, with their own weights, plus a small random-walk level of their own and observation noise. The controls are noisy proxies of what drives the target, not copies. The planted effect is a 10% lift over the post period, 831 cumulative units, and the counterfactual \\(y^0\\) is kept so every estimator can be graded.\nrelative effect, 95% interval cumulative effect tail probability real intervention (truth 10%, 831) +6.6% [+2.3%, +11.3%] 569 [207, 926] 0.000 placebo at day 243, data truncated at the real date +0.3% [−5.3%, +6.7%] 38 [−655, 742] 0.461 Top: the analysis at the real date, the cumulative interval excludes zero and contains the truth. Bottom: the same analysis with a fake date two thirds into the pre period, on data that stops at the real date; the cumulative effect stays around zero. Both criteria are met: the real interval excludes zero and contains the truth, the placebo interval contains zero. The placebo is computed on data truncated at the real date, so that no treated data enters the analysis.\nOne panel proves the protocol runs, not that the method is good. Two extensions make the placebo more informative:\nThe placebo-in-time as a sweep. One fake date is one draw. Eight fake dates spread over the pre period give the distribution of \u0026ldquo;effects\u0026rdquo; the design produces when nothing happens: that is the yardstick for the real effect. Here, all eight intervals contain zero. The placebo-in-space as a ranking. Each control is treated in turn as the unit of interest, at the real date. The real unit ranks first of six by absolute effect. Left: eight fake dates with 90-day post windows; the intervals contain zero, the real effect (orange) sits outside the cloud. Right: each unit treated as if it were the target; the real unit is the only one whose interval excludes zero. 5. Is the 95% interval a 95% interval? # Two curves grade the method: the share of intervals that exclude zero by true effect (power, and at zero effect the false-positive rate), and the share that contain the truth (coverage, which must stay near 95%). I drew them over 100 panels per effect size, for the maximum-likelihood interval, for the interval mixed over parameter draws, and for the placebo-in-time on the same panels. Since the effect touches neither the pre period nor the controls, coverage, width and bias are the same at every effect size; only detection moves.\nestimator coverage false positives at 0% detection at 1% 2% 5% 10% mean width local level, MLE interval 0.92 0.08 0.10 0.15 0.51 0.94 874 local level, 40 parameter draws 0.93 0.07 0.07 0.13 0.47 0.92 912 placebo-in-time on the same panels (243-day pre period, truth 0) 0.89 0.11 1,321 Left: the share of intervals excluding zero by true effect. Right: the share containing the truth; a little under 95% for the MLE interval, one point more with parameter draws, 89% for the placebo. Detection is modest. A 1% lift is invisible, 5% is seen one time in two, 10% almost always, with 90 post days. That is the order of magnitude of Brodersen\u0026rsquo;s simulation.\nThe placebo fires more often than 5%. The placebo windows have a shorter pre period (243 days instead of 365) and a less well calibrated interval: 11% false positives. One placebo in nine therefore finds an effect on a perfectly valid design. Reading the sweep against its own clean-panel rate is fairer than one date against a 5% threshold.\nThe length of the pre period matters, because a variance needs more points than a mean. Coverage over 50 panels per cell, at a 5% effect:\npre period (days) 60 120 240 365 730 local level, MLE 0.72 0.84 0.92 0.88 0.92 local level, parameter draws 0.88 0.90 0.92 0.92 0.92 OLS on the controls 0.18 0.20 0.22 0.10 0.16 The MLE interval under-covers on short pre periods; the parameter draws correct it. OLS stays around 0.2 at every length. The parameter draws earn their keep on short pre periods, where the estimated level variance can collapse to zero and hand back an overconfident static model. A prior corrects that too, and that is one honest reason CausalImpact puts one on the level standard deviation. Part 9 shows the other side of it.\n6. Nine estimators on the same panels # Each estimator fits on the pre period and returns the cumulative effect with the interval a practitioner would compute for it (iid errors for the simple ones). Same 100 panels, effects of 0 and 5%.\nestimator coverage false positives at 0% detection at 5% width RMSE pre-period mean 0.10 0.90 0.88 262 954 trend only (local trend, no controls) 0.96 0.04 0.15 2,224 559 difference-in-differences (mean of controls) 0.12 0.88 0.90 148 400 OLS on the controls, iid interval 0.19 0.81 0.93 124 274 convex synthetic control, iid interval 0.17 0.83 0.91 142 335 static regression in state-space form 0.21 0.79 0.93 135 274 local level + regression (MLE) 0.94 0.06 0.57 886 227 local level + regression, parameter draws 0.95 0.05 0.53 938 227 local trend + regression 0.92 0.08 0.44 1,065 280 In red, estimators whose coverage is below 90% (left) or whose false-positive rate is above 10% (right). Only the models with a stochastic level, and the trend alone, pass both. OLS on the controls and the static state-space regression are the same estimator. Point estimate nearly as accurate as the local level\u0026rsquo;s (RMSE 274 against 227), interval six to seven times too narrow: 80% false positives at zero effect. Four promotions out of five that did nothing would be declared effective. That is the measured version of the \u0026ldquo;overoptimistic inferences\u0026rdquo; Brodersen cites, after Bertrand, Duflo and Mullainathan. The convex synthetic control is nearly unbiased but inherits the same iid interval. The classical method knows this: it reports permutation ranks rather than intervals. Difference-in-differences is worse on both counts: the mean of the controls is a poor synthetic control when loadings differ across units. Trend extrapolation without controls covers, with an interval two and a half times wider, and detects a 5% lift one time in seven. The local level is the only estimator with near-nominal coverage and false positives near 5%. The local trend pays for its extra component in width and power. That table is, of these three weeks, the strongest argument for the placebo: the placebo-in-time of an OLS design would fire four times out of five too.\n7. When the design breaks: who catches what # The real deliverable is not the effect but the ability to say why a measurement is invalid. I planted four violations one at a time, with a 5% lift, 100 panels per scenario:\ncannibalised control: the treated effect is subtracted, one for one, from controls c0 and c1 (a promotion on A that eats the sales of B and C, used as controls); halo on the controls: the same amount is added to c0 and c1; own trend: the treated unit drifts by +0.1 per day on its own from day one (about 36% a year on a level of 100); relation break: the treated unit\u0026rsquo;s loadings on the common factors change by +30% at the intervention date. Four checks on each: the placebo-in-time, the placebo-in-space, a Chow test on differences between the two halves of the pre period, and the bias against the truth, which one never knows in practice. The \u0026ldquo;clean\u0026rdquo; row gives each check\u0026rsquo;s false-alarm rate.\nscenario coverage relative bias placebo-in-time fires placebo-in-space flags c0 / c1 flags a clean control Chow rejects clean 0.89 −0.5% 0.17 0.10 / 0.08 0.24 0.15 cannibalised control 0.86 +26% 0.17 0.39 / 0.36 0.34 0.15 halo on the controls 0.82 −27% 0.17 0.38 / 0.45 0.31 0.15 own trend 0.69 +71% 0.39 0.10 / 0.08 0.24 0.15 relation break 0.56 −11% 0.17 0.10 / 0.08 0.24 0.15 Left: each check\u0026rsquo;s firing rate by scenario. Right: the relative bias of the cumulative effect and the coverage. The relation break is biased and under-covered without any check rising above its false-alarm rate. Cannibalised control and halo. The bias is large (+26% and −27%), and yet the placebo-in-time is exactly as often green as on clean panels: the contamination starts at the intervention, nothing is wrong before. The placebo-in-space flags each contaminated control about four times more often than a clean one: power, but far from certainty. It also flags clean controls more often: a contaminated donor contaminates the placebos that use it. Own trend. The placebo-in-time fires in 39% of panels, more than twice its clean rate: a unit that drifts on its own produces \u0026ldquo;effects\u0026rdquo; at any date. The bias is +71%, because the local level forecasts flat and the drift continues. Relation break. Biased, under-covered, and no check fires above its clean rate. A relationship that stops holding at the intervention date is indistinguishable, in the data, from an effect of the intervention: the assumption is untestable by construction. The only defences are outside the data: a reason to believe the relationship persists, and the business reconciliation of part 10. Isolating a contaminated control # When the placebo-in-space flags several controls, one may be the culprit and the others its victims. The procedure removes the worst one (the smallest tail probability), reruns the check on the rest, and repeats until nothing is flagged.\nEach control\u0026rsquo;s own effect at the real date, with its interval. The controls removed by the isolation are marked; c0 and c1 are the two controls contaminated by construction. On this panel the isolation removes c1, then c0, then a clean control, c2. With all controls the effect is estimated 20% too high; with the two kept ones it is covered but very imprecise. Removing controls trades bias for variance, and with five donors the trade is a poor one. Hence the rule of part 10: a pool of 10 to 20 donors chosen before the campaign among products that cannot interact with the promoted one. The check is a guard, not a selection procedure.\n8. A real case: Volkswagen, 18 September 2015 # I took the public panel that the CausalImpact port uses as a test fixture: weekly closing prices of Volkswagen, BMW and Allianz, 325 weeks from January 2011 to March 2017. The intervention is dated: the US EPA\u0026rsquo;s notice of violation against Volkswagen, on Friday 18 September 2015. Here no truth is available, and every claim has to be defended by a placebo or a sensitivity analysis.\nVW, BMW and Allianz, with the date of the EPA notice. VW\u0026rsquo;s price loses 28% in one week. The first thing to notice # The week of Sunday 20 September 2015 already contains the crash: VW loses 28%. The pre period therefore ends on 13 September (246 weeks) and the post period starts on 20 September (79 weeks). But BMW falls in the same week (−8%) while Allianz gains 1%: the whole sector was repriced before recovering in October. If BMW reacted to the same event, it is not an untreated control.\nIs the pre-period relationship stable? # A rolling 52-week regression shows coefficients that wander: three share prices co-move through the market factor, but the loadings are not a law. A Chow test between the two halves of the pre period, computed on weekly differences, does not reject stability (p = 0.42). Stable enough to try, not stable enough to skip the placebo.\nThe effect, with BMW and Allianz as controls # value average effect per week −33.5 [−65.1, −2.1] cumulative effect over 79 weeks −2,650 [−5,140, −169] relative effect −20.9% [−33.9%, −1.7%] tail probability 0.018 one-step fit on the pre period R² 0.99, coverage 0.92, Ljung-Box p = 1.0 coefficients BMW 0.63, Allianz 0.30 The three usual panels: observed and counterfactual, pointwise effect, cumulative effect. The interval widens fast: 79 weeks is a long horizon for a random-walk level. The interval excludes zero, but not by much, and it widens fast: 79 weeks is a long horizon for a random-walk level. VW traded about a fifth below its counterfactual over the following eighteen months. The one-step R² of 0.99 says nothing: one-step forecasts of a random walk are always excellent. More telling: the residual spread after the date is 3.4 times the one before.\nThe placebo-in-time, one year earlier # I pretend the intervention happened on 14 September 2014, on data that stops on 13 September 2015. The estimated effect is indeed indistinguishable from zero: +5.4% [−8.2%, +23.1%]. Then the sweep: twelve fake dates between February 2012 and March 2015, 26-week post windows.\nTwelve fake dates, 26-week post windows, 95% intervals. All contain zero; the real effect over 79 weeks (orange) sits outside the cloud. All twelve intervals contain zero, the placebo estimates scatter between −7.6% and +9.0%, and the real effect sits well outside that cloud. This is the (b) half of the criterion, the one that makes the (a) half reportable. This placebo validates the pre-intervention model; it says nothing about BMW, which was hit by nothing in the placebo windows.\nWere the controls themselves treated? # Brodersen\u0026rsquo;s third analysis: each control as if it were the treated unit, with the other control as its donor, at the real date.\nown effect over 79 weeks over the 14 weeks after the notice BMW (counterfactual from Allianz) −11.3% [−27.3%, +13.8%] −1.5% [−10.5%, +9.6%] Allianz (counterfactual from BMW) +14.7% [−3.7%, +41.6%] +9.4% [+1.0%, +19.3%] BMW forecast from Allianz, and Allianz forecast from BMW, over the 14 weeks after the notice. Allianz \u0026lsquo;rises\u0026rsquo; 9.4% relative to BMW: it is BMW that fell. With two controls this check is a mirror: \u0026ldquo;BMW fell relative to Allianz\u0026rdquo; and \u0026ldquo;Allianz rose relative to BMW\u0026rdquo; are the same fact. BMW lost 8% then recovered 12% within three weeks, Allianz did not move: BMW is a doubtful control, at least in the short run. Rather than pick a verdict, I reported the effect under each control set.\nSensitivity: how much of the answer is a modelling choice? # specification relative effect 95% interval excludes zero BMW + Allianz, local level (reference) −20.9% [−33.9, −1.7] yes Allianz only −27.2% [−41.8, −2.8] yes BMW only −16.8% [−31.5, +5.9] no no control, local linear trend −22.9% [−45.0, +27.2] no BMW + Allianz, static regression −30.1% [−31.0, −29.1] yes BMW + Allianz, local linear trend −19.7% [−35.1, +4.4] no BMW + Allianz, log prices −4.3% [−8.4, +0.2] no BMW + Allianz, pre period from 2013 −21.5% [−34.9, −1.2] yes BMW + Allianz, pre period from 2014 −21.4% [−34.6, −1.5] yes Nine specifications. In grey, those whose interval contains zero. The sign never changes; the interval is what moves. Every specification in prices with a control puts the estimate between about −15% and −30%, and the sign never flips. What moves is the interval: the static regression is narrow with a confidence part 6 showed to be unearned, the local linear trend too uncertain at 79 weeks to conclude, BMW alone not significant. The defensible report is a range from −17% to −27%, with the control-set question flagged.\nDoes the model use time order at all? # A cheap check, after TASC: shuffle the pre-period rows and refit. A static regression gives the same answer; a model with a stochastic level does not, because a shuffled random walk is no longer one and the level variance collapses.\nOver 20 permutations of the pre period, the local level\u0026rsquo;s cumulative effect converges to the static answer, with an interval ten times narrower than the original. Over 20 permutations, the local level\u0026rsquo;s cumulative effect converges to the static answer and its narrow interval (width 518 against 4,971 in the original order). The local level\u0026rsquo;s wider interval is the price of serial dependence; the static regression\u0026rsquo;s narrow interval is what one gets by pretending the weeks are exchangeable.\n9. Comparison with Google\u0026rsquo;s CausalImpact # The hand-built model is a frequentist cousin of CausalImpact: same state space, maximum likelihood instead of priors and MCMC. I ran tfcausalimpact, the TensorFlow Probability port, on the Volkswagen panel and on synthetic panels.\ncase method cumulative effect, 95% interval width VW, BMW + Allianz hand-built, MLE −2,650 [−5,140, −169] 4,971 tfcausalimpact, default prior on the level sd (0.01) −3,355 [−4,427, −2,233] 2,193 tfcausalimpact, prior at 0.1 −2,878 [−5,212, −428] 4,784 Cumulative effect and 95% interval under four variants, for both control sets. The point estimates sit inside each other\u0026rsquo;s intervals; the widths do not. Same data, same state space, point estimates each inside the other\u0026rsquo;s interval, and an interval half as wide. CausalImpact\u0026rsquo;s default prior on the level standard deviation is 0.01, in units of the standardised series; maximum likelihood gives 0.085 here, eight times larger. A tight prior keeps the level nearly deterministic, which forecasts 79 weeks ahead with a nearly constant variance. Loosening it to 0.1 opens the interval up to the hand-built one. The narrow interval was the prior, not the data. This is not a flaw of either implementation, but the interval width on a real case is a modelling choice, to be reported as such. On a known truth one can ask who covers:\n30 panels, 5% lift coverage detection mean width RMSE seconds per fit hand-built, MLE 0.90 0.53 887 219 0.1 tfcausalimpact, default prior 0.77 0.80 494 262 15.7 Each segment is a 95% interval recentred on the truth: a segment that misses zero missed the truth. The tfcausalimpact intervals are shorter and miss more often. The default prior buys an interval 45% narrower and a higher detection rate at the price of coverage: 77% against 90%. It is a reasonable default for the daily marketing series the method was built for, and a poor default for a series with a large own random walk, such as a share price. A placebo sweep would reveal it: its false-positive rate would sit well above 5%.\nNeither \u0026ldquo;narrow\u0026rdquo; nor \u0026ldquo;Bayesian\u0026rdquo; is a synonym for \u0026ldquo;calibrated\u0026rdquo;. Coverage on a known truth, or a placebo sweep on real data, are the only evidence I found, and that brings me back to TASC\u0026rsquo;s \u0026ldquo;narrower\u0026rdquo; intervals. One practical point, finally: 0.1 seconds per fit against 12 to 16, which matters for sweeps and Monte Carlo studies.\n10. Measuring promotion incrementality continuously # The plan\u0026rsquo;s third block was an architecture decision, with no code: a promotion engine runs campaigns on products in stores, and for each campaign one wants the causal effect on sales by counterfactual, because the business will not accept a holdout for every promotion. Three points decide whether the number is worth anything.\nSelection and integrity of the controls. With part of the lift cannibalised from a product used as a control, the effect is overestimated by a quarter and the placebo-in-time stays green in 83% of cases. Before the campaign, candidate controls therefore come from a registry of eligible donors built from the product graph: same store cluster and seasonality as the target, but a different category and no substitution edge, whatever the quality of the fit, because the best-fitting control is often the closest substitute. The pool holds 10 to 20 donors, to afford losing some. After the campaign, the placebo-in-space with leave-one-out isolation flags the donors that moved: a negative own effect is a cannibalisation, a positive one a halo, and the effect is reported on the kept pool.\nMonitoring the causal assumption over time. The counterfactual is unobservable. Four substitutes, each tied to a failure mode the panel reproduces:\nSignal What it guards Rule pre-period fit of the deployed model (one-step R², Ljung-Box, coverage of the one-step interval), per campaign the model is the right size for the series one-step coverage outside [0.90, 0.98] or Ljung-Box p \u0026lt; 0.01 blocks the report rolling placebo-in-time on never-treated products, same model, same lengths, every week the calibration of the design where the truth is zero share of placebo intervals excluding zero over the last 50 runs above 10% (the panel measured 11% on short pre periods) raises an alert; the clean-panel rate is the baseline placebo-in-space on every campaign interference any isolation round beyond the first on a target triggers a review stability of the donor-target relationship: Chow on differences, drift of the coefficients between successive campaigns of the same target the relationship that is extrapolated p \u0026lt; 0.01 or a sign change of a coefficient blocks the donor What this monitoring cannot see, and the dashboard says so: a relationship that breaks at the campaign date. Only the third point catches it.\nThe business guard-rail. The estimated effect is a forecast residual. It is tied to money through a quantity observed later: the realised incremental margin, reconciled every month against the sum of estimated counterfactual and incremental margins. The reconciliation gap is a time series: its level is noisy, its drift is the alarm. A gap that grows in one direction for three months means the counterfactuals are systematically off, and that is what a relation break that nothing else sees looks like from outside. Two more anchors: an occasional real holdout on a small share of campaigns, compared with the counterfactual effect; and the sum of effects within a category, which cannot exceed its observed growth for long. As long as the gap has not been closed for a product family, the effect that goes into a decision is the lower bound of the interval.\n11. What I take away # These observations hold for the setups studied here: a semi-synthetic panel with five controls and 90 post days, and a panel of three share prices.\nThe counterfactual is a forecast, and its uncertainty is the model\u0026rsquo;s belief about what the unexplained part does over time. On these panels only a level that can wander gives a 95% interval that covers at 95%. Static estimators are accurate and overconfident: 80% false positives at zero effect. That is the strongest argument for not skipping the placebo. The placebo-in-time validates the pre-intervention model, and nothing that starts at the intervention date. It is blind to a contaminated control and to a relation break. The placebo-in-space is the check for interference, with modest power and a necessary leave-one-out isolation, because a contaminated donor flags its neighbours. A relationship that breaks at the intervention date is seen by no data-only check. Control selection before, business reconciliation after: that is the only treatment I found. A placebo that fires does not necessarily invalidate the design. I now read a sweep against its clean rate, not one date against 5%. On Volkswagen, the effect of the EPA notice is −17% to −27% over 79 weeks, depending on the control set, placebo clean at twelve fake dates, BMW flagged as a doubtful control. The interval width on a real case is a modelling choice. CausalImpact\u0026rsquo;s default prior gives an interval half as wide, which covers 77% instead of 90%. Open questions. I did not implement TASC, nor measure its coverage. A ramp and a pulse remain to be studied in place of the step-shaped effect, and the placebo-in-space was measured for a single contamination size. The monitoring of part 10 is a line of reasoning, not a production system.\nA question to check your understanding. A promotion on product A is measured by counterfactual with, as controls, three products from the same aisle chosen for the quality of their pre-period fit. The effect is +12% [+4%, +20%], and the placebo-in-time at three fake dates is clean. What is that number worth? The placebo-in-time only tests the pre period, where the controls were contaminated by nothing (part 7). If the promotion took sales from the same aisle, the counterfactual is pulled down and the effect overestimated without any placebo-in-time seeing it. What is needed is the placebo-in-space on each of the three controls, and a reason for having chosen them other than their fit. Reproduce and go further # The code, the five notebooks and the 22 tests are in the gwils28/time_series_causal_statistics repository (Python 3.13, versions locked with uv). Every table in this article comes out of a cts synth, cts vw, cts study all or cts tfci command, with fixed seeds; the Monte Carlo studies take about fifteen minutes and the cross-check about ten.\nSources\nBrodersen, Gallusser, Koehler, Remy and Scott, Inferring causal impact using Bayesian structural time-series models, The Annals of Applied Statistics, 9(1), 2015, and the CausalImpact implementation. Rho, Illick, Narasipura, Abadie, Hsu and Misra, Time-Aware Synthetic Control, arXiv, January 2026. Carlson and Shephard, When are time series predictions causal? The potential system and dynamic causal effects, arXiv, March 2026. tfcausalimpact, Python port of CausalImpact on TensorFlow Probability, and its test fixture for the Volkswagen panel. ","date":"27 October 2026","externalUrl":null,"permalink":"/en/posts/impact-causal-series-temporelles/","section":"Posts \u0026 notes","summary":"A counterfactual is a forecast. What I observed building one by hand, measuring its coverage on 100 panels with a known truth, planting four violations of the design, and applying it to Volkswagen’s share price after dieselgate.","title":"Causal impact on time series: the counterfactual and its placebos","type":"posts"},{"content":"","date":"27 October 2026","externalUrl":null,"permalink":"/en/tags/causalimpact/","section":"Tags","summary":"","title":"CausalImpact","type":"tags"},{"content":"","date":"27 October 2026","externalUrl":null,"permalink":"/tags/espace-d%C3%A9tats/","section":"Tags","summary":"","title":"Espace D'états","type":"tags"},{"content":"","date":"27 October 2026","externalUrl":null,"permalink":"/en/tags/state-space/","section":"Tags","summary":"","title":"State Space","type":"tags"},{"content":"","date":"17 October 2026","externalUrl":null,"permalink":"/tags/r%C3%A9entra%C3%AEnement/","section":"Tags","summary":"","title":"Réentraînement","type":"tags"},{"content":"","date":"17 October 2026","externalUrl":null,"permalink":"/en/tags/retraining/","section":"Tags","summary":"","title":"Retraining","type":"tags"},{"content":" How often should a forecasting model be retrained? The question looks operational. Digging into it, I found that it depends on what the service is expected to deliver: a low error, a recent model, forecasts that don\u0026rsquo;t change without reason. These three expectations don\u0026rsquo;t lead to the same frequency. Reading time: about 31 minutes. Every formula is followed by its intuition, so you can skim the maths without losing the thread.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThis article distils my active research for week 44 (October 2026). I started from a detailed work plan, with its scripts, its numbers and its conclusions. I first re-ran every script verbatim: each one reproduces its numbers to the printed digit. Then I checked every reading of those numbers. Several don\u0026rsquo;t hold, and that is where I learned the most. I also went back to the primary sources (weight licences on the Hub, paper bodies rather than abstracts) and tested everything on real data, under a protocol written before the results. It is all reproducible in the W44_retraining_frequency repository.\nIn short. The main results, with their scope:\nOn a simulated bench, a perfect break detector scores a MASE of 1.37, against 0.98 for a fixed weekly retrain. The gap comes mostly from the window the model is refitted on after detection. With a window that restarts at the break, plus a weekly retrain, the same detection does better than continuous retraining, on 5 seeds out of 5, with 1% of its refits. Retraining less often reduces how often published forecasts are revised, but makes each revision larger. From continuous to monthly, the mean jump grows 224-fold in the analytical model. The freshness budget depends on the served quantile by a factor of 2 in simulation, not 7 as the plan stated. On real data, the pre-registered decision rule is inconclusive. The data closing lag changes three measurements: the backtest error (underestimated by about 6%), the baseline and the trigger. A before/after dashboard shows 197% of the real effect for a retrain that does nothing, because of regression to the mean. The trigger threshold also makes it possible to estimate the effect at the threshold, with a regression discontinuity design. 1. The problem: choosing a retraining frequency # At each origin \\(t_0\\), a forecast service publishes forecasts for the next \\(H\\) steps. The model behind them was fitted on a history window of length \\(w\\), at some point in time. The operational question is: every how many steps \\(k\\) should it be refitted? \\(k = 1\\) is continuous retraining; with hourly data, \\(k = 24\\) is daily, \\(k = 168\\) weekly and \\(k = 720\\) monthly.\nWhat we expect from the service can be described along three axes:\nAxis The expectation What it implies for the frequency Accuracy the average error stays low retrain often (a recent model tracks the level better) Freshness the served model is never more than X hours old a bound on \\(k\\), which may depend on what is served Stability the published forecast doesn\u0026rsquo;t change without reason depends on the measure chosen: size or frequency of revisions On top of that comes cost: every refit uses compute, which matters across a park of hundreds of thousands of series.\nOne last parameter appears throughout the article: the closing lag \\(d\\). Data can\u0026rsquo;t be used as soon as it exists. Electricity readings, sales and invoices get consolidated, corrected and validated before they become available. Here, \\(d = 48\\) hours: a model \u0026ldquo;retrained now\u0026rdquo;, at time \\(t\\), is fitted on a window that ends at \\(t - 48\\).\nNotation Meaning Value on the bench \\(k\\) cadence: steps between two refits 1 to 720 hours \\(w\\) length of the fitting window 240 hours \\(d\\) data closing lag 48 hours \\(H\\) forecast horizon 24 hours \\(\\tau\\) served quantile level 0.50 · 0.80 · 0.95 This parameter \\(d\\) comes into play at three points of the analysis: the backtest, the baseline and the trigger.\n2. What recent work says # I read five recent works and set one aside. For each, I looked for the numbers in the body of the paper, not in the abstract, which sometimes contains none.\nMarco Zanotti\u0026rsquo;s two papers set the frame. The first (arXiv 2505.00356) studies ten global models on the M5 and VN1 retail data, from continuous retraining to none at all. The second (arXiv 2506.05776) introduces a stability measure for probabilistic forecasts, the Scaled Multi-Quantile Change (SMQC). What the paper bodies say:\nfor point forecasts, never retraining degrades the error by less than 5% on these datasets; monthly retraining cuts compute time by about 75%; probabilistic forecasts are more sensitive: the optimal frequency is about 2 weeks on M5, against 3 to 4 weeks for point forecasts; stability follows no simple rule: on VN1, retraining less often makes NBEATSx and CatBoost more stable but the MLP and linear regression less stable. On M5, almost nothing moves. Probabilistic forecasts therefore seem to call for somewhat more frequent retraining than point forecasts, with a moderate effect. This comes back in part 6.\nHarrison Katz (arXiv 2604.06438) treats retraining as a costly decision. The monitored quantity is a \u0026ldquo;learning debt\u0026rdquo;: the Kullback-Leibler divergence between a continuously updated shadow posterior and the frozen posterior of the deployed model. You retrain when the expected regret of waiting exceeds the cost of the refit. The policy does better than a tuned calendar in 72 scenario cells out of 72, and better than a tuned CUSUM in 58 out of 72, with a mean relative objective of 0.975 against CUSUM. The gap with a well-tuned CUSUM is therefore small; the main contribution is a threshold expressed in units of cost. The paper itself calls it a \u0026ldquo;proof-of-concept\u0026rdquo; on simulation, \u0026ldquo;not a universal replacement for drift detection\u0026rdquo;.\nPILOT (Kwak, Jung, Park, arXiv 2609.39789) learns a retraining trigger. The abstract gives no gain figure. I kept one design choice: the trigger\u0026rsquo;s training label is a future quantity (the coming rise in error), while the deployed trigger only reads errors that are already closed. This separation keeps the trigger from using data that isn\u0026rsquo;t available at decision time. Part 4 shows the effect of that kind of leak on a CUSUM.\nTimestral (Mistral, 15 October 2026) is a family of time series foundation models (5M, 28M, 123M and 209M parameters) that runs on CPU, at 140 ms for the largest on 8 threads. It is mostly relevant to serving; I come back to it in part 8, in particular for its licence.\nSet aside: DriftGuard (arXiv 2601.08928). It claims \u0026ldquo;97.8% detection recall within 4.2 days\u0026rdquo; and \u0026ldquo;up to 417 return on investment\u0026rdquo;, with no unit. Without a unit I couldn\u0026rsquo;t interpret the second number, and I didn\u0026rsquo;t use the paper as a source.\n3. The bench # To compare retraining policies you need a setting where the truth is known. So I simulated a park of 120 hourly series over 2,400 steps, served at horizon 24 by a deliberately simple model: the window mean plus a 24-hour daily profile, fitted on \\(w = 240\\) hours.\nThe series undergo two drifts of different kinds:\nsudden level breaks, drawn from a Poisson process (3.7 per series on average, 448 in total); a slow drift of the seasonal amplitude, sinusoidal. Ten policies are compared: never; a calendar with \\(k \\in \\{1, 24, 48, 168, 720\\}\\); a two-sided CUSUM on the realised error; Katz\u0026rsquo;s KL debt at two thresholds; and a perfect detector, which knows the exact date of every break and retrains on it. We measure accuracy (MASE), cost (refits per series) and stability (part 5).\nPolicy MASE Refits / series never 1.6003 0 monthly calendar (\\(k = 720\\)) 1.2520 2 weekly calendar (\\(k = 168\\)) 0.9829 12 daily calendar (\\(k = 24\\)) 0.9042 85 continuous (\\(k = 1\\)) 0.8915 2,063 CUSUM on realised error 0.9189 92.5 KL debt (threshold 2) 0.9546 30.3 perfect detector 1.3685 3.7 Effect of the closing lag on the backtest # First check: what happens if the backtest ignores \\(d\\), i.e. fits the model on data that would not yet be closed in production?\nIn blue, the usable window. The break falls in the not-yet-closed zone: the served level (10.03) doesn\u0026rsquo;t reflect the new level (12.05). A backtest without the lag (d = 0) would have served 10.24. Without the lag, MASE is underestimated by about 6% for every policy: +6.3% daily, +5.6% weekly, +6.2% for CUSUM. Since the gap is roughly the same everywhere, the ranking of policies doesn\u0026rsquo;t change: the whole curve shifts. A uniform gap is hard to spot, since it doesn\u0026rsquo;t affect comparisons. It does change the error level being reported.\n4. The case of the perfect break detector # This result surprised me. A detector that knows every break, with no false alarm and no delay, and retrains 3.7 times per series at each break, scores a MASE of 1.3685. A fixed weekly retrain scores 0.9829. The perfect detector comes second to last, ahead only of the policy that never retrains.\nThe work plan offered an explanation: the slow seasonal drift is not a break, so a break detector doesn\u0026rsquo;t correct it. I tested that explanation; the checks below show that it accounts for only a small part of the gap.\nThree checks # (a) Without seasonal drift, the gap remains. With level breaks only (same breaks, no slow drift), weekly scores 0.9425 and the perfect detector 1.2620. Only one kind of drift is left, the one the detector spots without error, and the gap is still 34%.\n(b) The refit window doesn\u0026rsquo;t contain the break. The detector fires at the break time \\(t\\). But because of \\(d\\), the fitting window ends at \\(t - 48\\): it contains no point after the break. The refit is fitted on the old regime. Delaying the refit improves the score, slowly:\nRefit delay after the break 0 48 120 168 288 (\\(= d + w\\)) MASE 1.3685 1.3699 1.2071 1.1384 1.1110 48 hours after the break, the window still holds only one post-break point out of 240. A 240-step mean is barely affected.\nIn orange, the perfect detector with the plan\u0026rsquo;s refit: the served level stays close to the old one. In grey, the weekly retrain follows in steps. In blue, the detector with a window that restarts at the break. (c) Changing what follows the detection. I kept the same detection and changed the refit: after a break at \\(c\\), refit at \\(c + d + 24\\), \\(c + d + 72\\) and \\(c + d + 240\\), on a window that starts at the break instead of covering the old regime.\nPolicy MASE Refits / series weekly 0.9829 12.0 continuous 0.8915 2,063 perfect detector, plan\u0026rsquo;s refit 1.3685 3.7 perfect detector + restarted window 0.9489 10.2 perfect detector + restarted window + weekly 0.8789 22.2 The last row does better than continuous retraining, with 1% of its refits. Since this is a single draw, I re-ran the experiment on five seeds and compared the policies per seed (each sees the same series):\nRight: the restarted window + weekly does better than continuous on all 5 seeds. Left: the restarted window alone does better than weekly on 4 seeds out of 5, a gap consistent with a tie. On this bench, then, the issue is not the detection but what follows it. Two choices are involved: when to refit, and on which window. With a fixed window, detection adds little. On its own, it doesn\u0026rsquo;t handle the slow drift. Combined with a window that restarts at the break and a weekly retrain, it gives the best score on the bench.\nLimit. This result assumes a perfect detector. With a real CUSUM restarting the window at its estimate of the break date, MASE gets worse (1.0445 vs 0.9179 without the restart), and instability is multiplied by 7: false alarms shrink the window to a few noisy cycles. Restarting the window therefore seems useful mainly when the break is known from another source (a price change, a store opening, a dated incident). I did not tune the CUSUM variant afterwards, so as not to pick a variant after seeing its result. CUSUM and unclosed errors # In the plan, CUSUM stood out for its targeting: 40.9% of its refits came within 72 hours of a break, against 12.2% for a calendar, 3.4 times more. But that CUSUM read the error of the forecast for \\(t\\) at time \\(t\\), while the bench assumes observations only close at \\(t + 48\\). The trigger was therefore using data that the model itself could not use.\nWith errors closed at the same lag, MASE barely changes (0.9179 vs 0.9189), but targeting drops to 20.0%: 1.6 times the calendar, not 3.4. About half the advantage came from that leak. This is the separation PILOT makes: the deployed trigger only reads closed errors.\nThe indicator itself deserves a remark. \u0026ldquo;Share of refits within 72 hours of a break\u0026rdquo; also counts refits that cannot correct anything: with \\(d = 48\\), a refit less than 48 hours after a break doesn\u0026rsquo;t see it. That is partly why the perfect detector reaches 100% on this indicator while having one of the worst MASE scores. On this bench, detection recall says nothing about the accuracy achieved.\n5. Stability of published forecasts # When a user checks tomorrow\u0026rsquo;s forecast on Monday, then again on Tuesday, they expect it to change only if something has changed. Every refit revises the published forecast. Churn is the average size of those revisions per time step.\nOne might expect that retraining less often makes the service more stable. That holds for the frequency of revisions, not for their size.\nEach point is a policy from the bench. Clause A: no jump above 10% of the scale (below the horizontal line). Clause B: a visible revision on at most 1% of hours (left of the vertical line). No point lies in the lower-left quadrant. Monthly retraining publishes jumps whose 95th percentile is 293% of the scale, on 0.1% of origins. Continuous retraining publishes jumps of 1.3%, on 12.6% of origins. The two usual ways of stating a stability requirement, \u0026ldquo;no revision above X%\u0026rdquo; and \u0026ldquo;at most N revisions a month\u0026rdquo;, rank the policies in opposite orders. On this bench, no cadence meets both at once.\nMean churn, often monitored, has a limitation: it is exactly zero for a model that is never retrained. It doesn\u0026rsquo;t tell a stable model from a frozen one.\nAn analytical model: the local level # To understand where this opposition comes from, I took the simplest possible model, where everything can be computed by hand: a local level. The series is a level that follows a random walk, plus noise:\n$$ y_t = L_t + \\varepsilon_t, \\qquad L_t = L_{t-1} + u_t, \\qquad u_t \\sim \\mathcal N(0, q), \\quad \\varepsilon_t \\sim \\mathcal N(0, \\sigma^2). $$The service publishes the mean of the last \\(w\\) observations, re-estimated every \\(k\\) steps. The served model has mean age \\(\\bar a = (k-1)/2\\). Its mean squared error decomposes exactly:\n$$ \\text{MSE}(k, w) = \\underbrace{\\sigma^2\\left(1 + \\tfrac{1}{w}\\right)}_{\\text{noise}} + q\\Big[\\underbrace{(\\bar a + 1)}_{\\text{model age}} + \\underbrace{\\tfrac{(w-1)(2w-1)}{6w}}_{\\text{window lag}}\\Big]. $$The last term is the centroid lag: a mean over \\(w\\) steps estimates the level as it was \\((w-1)/2\\) steps ago, not the current level. Setting the derivative in \\(w\\) to zero gives the optimal window:\n$$ w^\\star = \\sqrt{\\frac{3\\,(\\sigma^2 + q/6)}{q}} \\approx 27.4 \\quad (\\sigma = 1,\\ q = 0.004). $$The intuition. The two parameters act on two additive terms. The window sets its own lag, the cadence sets the model\u0026rsquo;s age. Retraining more often doesn\u0026rsquo;t reduce the lag of a window that is too long. A trend illustrates this: on \\(y_t = 0.01\\,t + \\text{noise}\\) with \\(w = 240\\), going from monthly to continuous removes 75% of the bias. The remaining 25% comes entirely from the window, and no cadence reduces it. This bias is a matter of model specification (a trend term, a weighting) rather than retraining frequency.\nFor stability, the variance of the jump \\(R\\) between two re-estimations \\(k\\) steps apart (with \\(a = \\min(k, w)\\) and \\(b = \\max(k, w)\\)) splits into two terms:\n$$ \\operatorname{Var}(R) = \\underbrace{\\frac{2a\\,\\sigma^2}{w^2}}_{\\text{estimation fluctuation}} + \\underbrace{\\frac{q}{w^2}\\left[a^2(b - a + 1) + \\frac{(a-1)\\,a\\,(2a-1)}{3}\\right]}_{\\text{real level movement}}. $$The first term is noise: the window slid, a few noisy observations came in and out, and the forecast moves although the level didn\u0026rsquo;t change. The second is information: the level really moved. The noise share is the ratio of the first term to the total variance, i.e. the share of published revisions that doesn\u0026rsquo;t reflect a change in the level.\nI checked these formulas against a 300,000-step Monte Carlo simulation on eight configurations: the MSE agrees within 0.6%, the jump variance within 1.8%, a gap smaller than the Monte Carlo error itself measured over five seeds.\nCadence \\(k\\) Mean jump Churn per step Jump vs \\(k = 1\\) Churn vs \\(k = 1\\) Noise share 1 0.0057 0.00572 ×1 ×1 67.6% 24 0.0802 0.00334 ×14 ×0.58 8.2% 168 0.4830 0.00288 ×84 ×0.50 1.6% 720 1.2787 0.00178 ×224 ×0.31 0.3% From continuous to monthly, the published jump grows 224-fold while churn per step only falls 3.2-fold. In this model, retraining less often does little to reduce total instability: it is published less often, but in larger jumps.\nNoise share of the revision variance against the cadence, for the bench\u0026rsquo;s window (w = 240) and the optimal window (w* = 27). At continuous cadence on a 240-step window, two thirds of the revision variance comes from estimation noise. With the optimal window \\(w^\\star = 27\\), it is 95%. The shorter the window, the higher that share. In other words, at continuous cadence, most published revisions reflect fluctuations of the estimate rather than a change in the level.\nA correction to the plan. The plan stated that churn per step reaches a constant floor, beyond which the cadence would have no effect. That only holds for \\(k \\ll w\\). At \\(k = w\\), churn is 82% of that \u0026ldquo;floor\u0026rdquo;, and beyond it keeps falling as \\(\\sqrt{q/k}\\): at \\(k = 720\\) it is already below it. The cadence therefore still affects churn, weakly (×0.47 from \\(k = 1\\) to \\(k = 240\\)) but measurably. 6. Model freshness and the served quantile # A probabilistic service doesn\u0026rsquo;t publish one value but quantiles: the median (\\(\\tau = 0.50\\)) to plan, the 0.95 quantile to size stock or capacity. Each one is scored with the pinball loss:\n$$ \\rho_\\tau(y, \\hat q) = \\begin{cases} \\tau\\,(y - \\hat q) \u0026 \\text{if } y \\ge \\hat q,\\\\ (1 - \\tau)\\,(\\hat q - y) \u0026 \\text{otherwise.}\\end{cases} $$The intuition. At the 0.95 quantile, a forecast that is too low costs 19 times more than one that is too high by the same amount. The loss is minimal when 95% of observations fall below the forecast.\nThe question for this part: at what age does a model stop doing better than a naive baseline? I call \\(k_{\\max}\\) the largest cadence at which the model still does better than the seasonal naive (\u0026ldquo;tomorrow will be like today at the same hour\u0026rdquo;). That is the freshness budget that could go into a service contract.\nThe starting result # On 200 simulated series, with gaussian quantiles (mean plus \\(z_\\tau\\) times the in-window residual standard deviation), the plan found:\n\\(\\tau = 0.50\\) \\(\\tau = 0.80\\) \\(\\tau = 0.95\\) \\(k_{\\max}\\) 168 h 120 h 24 h degradation from \\(k = 1\\) to \\(k = 720\\) +41% +52% +97% The 0.95 quantile would thus need a model seven times more recent than the median, which would argue for a separate freshness budget per quantile. While checking this result, I found two effects that explain most of it.\nCoverage of gaussian quantiles # A well-calibrated 0.95 quantile is exceeded 5% of the time. This one covers 90.9% of observations when the model is recent, and 83.5% when it is a month old.\nGaussian (orange): 91% coverage for a recent model, 84% at one month. Conformal on closed errors (blue): stable around 92%, whatever the age. The standard deviation is measured inside the window, on in-sample residuals. It doesn\u0026rsquo;t account for out-of-sample error growing with the model\u0026rsquo;s age. The \u0026ldquo;+97% at the 0.95 quantile\u0026rdquo; therefore mostly reflects a calibration failure, more than a freshness cost.\nAn alternative: place the quantiles at the empirical quantile of the model\u0026rsquo;s own closed errors, per series and per lead time, over the last 240 origins, recomputed every 24 hours. This is a form of conformal prediction. The degradation from \\(k = 1\\) to \\(k = 720\\) then becomes +19%, +19% and +21% for the three quantiles, nearly the same. Coverage at the 0.95 quantile stays between 0.915 and 0.923 at every cadence. It doesn\u0026rsquo;t reach 0.95: the conformal guarantee assumes exchangeable errors, which level breaks violate.\nThe data the baseline reads # The lag-24 naive reads the 24 most recent hours, which the model may not see. The lag-72 naive, on a closed window, reads what the model can read. At origin \\(t_0\\), the plan\u0026rsquo;s seasonal naive copies \\(y_{t_0-23}, \\dots, y_{t_0}\\). The model, on the other hand, only has access to data closed at \\(t_0 - 48\\). The comparison gives the baseline 48 hours of data the model doesn\u0026rsquo;t have. The comparable baseline under \\(d = 48\\) is the lag-72 naive (same hour, three days earlier), and it is clearly worse.\nThis is the reverse of the backtest in part 3: here, ignoring the lag favours the baseline.\nResults after correction # Quantiles Baseline \\(k_{\\max}\\) 0.50 0.80 0.95 Ratio 0.50 / 0.95 gaussian lag-24 naive (plan) 168 120 24 7.0 gaussian closed naive 168 168 72 2.3 conformal lag-24 naive (plan) 168 168 36 4.7 conformal closed naive 336 336 168 2.0 With both corrections, the freshness budget still depends on the served quantile, but by a factor of 2, stable over five seeds. This result points in the same direction and has the same order of magnitude as Zanotti\u0026rsquo;s: probabilistic forecasts call for somewhat more frequent retraining than point forecasts (2 weeks against 3 to 4 on M5), not seven times more.\nStability also depends on how the quantiles are computed. With gaussian quantiles, only continuous retraining meets \u0026ldquo;jump p95 ≤ 10%\u0026rdquo; at the 0.95 quantile. With conformal quantiles, every cadence meets it (2.9% to 8.4%), but recalibration publishes a visible revision on 19% to 45% of origins. The opposition between size and frequency of revisions from part 5 shows up again.\n7. Real data: Monash Electricity # The previous results come from simulated data. To compare them with real data, I re-ran the analysis on Monash Electricity Hourly: 321 hourly electricity consumption series, 2012 to 2014, under a CC-BY-4.0 licence.\nThe protocol # Before running a single computation, I wrote and committed a protocol: the series kept, the evaluation period, the baselines, the metrics and the decision rule. Seven series were excluded for more than 10% zeros (one of them entirely zero). That leaves 314 series and 8,760 hourly origins in 2014, with a pinball loss scaled series by series.\nThe pre-registered decision rule: with conformal quantiles, compare \\(k_{\\max}\\) at the median and at the 0.95 quantile against the lag-24 naive. The plan\u0026rsquo;s hypothesis (the freshness budget depends on the quantile) is supported if the ratio is at least 4 over the year and in at least 3 quarters out of 4, and contradicted if it is 1 or less over the year.\nComparison with the baselines # \\(\\tau = 0.50\\) \\(\\tau = 0.80\\) \\(\\tau = 0.95\\) coverage 0.95 lag-24 naive (reads unclosed data) 0.5001 0.4269 0.1941 0.945 lag-72 naive (closed) 0.6410 0.5303 0.2343 0.938 lag-168 naive (closed, weekly) 0.5586 0.4654 0.2190 0.932 gaussian model, \\(k = 1\\) 0.5461 0.4289 0.2054 0.909 gaussian model, \\(k = 720\\) 0.6294 0.4900 0.2442 0.895 conformal model, \\(k = 1\\) 0.5932 0.4713 0.2175 0.909 conformal model, \\(k = 720\\) 0.6334 0.4877 0.2208 0.909 The model is worse than the lag-24 naive, which reads unclosed data (lower dashed line), and better than the closed lag-72 naive (upper dashed line). Applying the decision rule # Applied as written, the pre-registered rule is inconclusive. Against the lag-24 naive, the model never does better at the median, at any cadence, in any quarter: \\(k_{\\max} = 0\\) and the ratio is undefined. The hypothesis is neither supported nor contradicted.\nI added the comparison with the closed naive afterwards, because part 6 showed that the lag-24 naive reads unclosed data. That is a deviation from the protocol, and I report it as such. Against this baseline, with conformal quantiles, the model stays better up to 720 hours at every quantile: the ratio is 1.0 and the hypothesis is contradicted. I report both results; only the first follows the pre-registered protocol.\nWhat these data show:\nAgainst a baseline that reads the most recent data, no cadence does better. On this electricity consumption, the lag-24 naive is strong enough that its 48 extra hours of data are enough to beat the model, whatever the cadence. Ignoring the lag for the baseline leads to concluding that the model adds nothing; ignoring it for the model makes it look better than it is (+12.8% at the 0.95 quantile, daily). Against the closed baseline, the quantile dependence is weak with calibrated quantiles, and unstable without. With gaussian quantiles I got 720 / 720 / 168, then 336 / 336 / 168 just by shifting the refit calendar by 240 hours. A value that moves by a factor of 2 with that shift alone seems too fragile to me to serve as a contractual threshold. The second quarter is an exception. From April to June 2014, the model is worse than the closed naive at every cadence with conformal quantiles. A single freshness budget, measured over the year, hides this variation. Gaussian on the left, conformal on the right. The second quarter drops to zero. Neither method is calibrated: about 91% coverage for a 0.95 quantile, at every cadence. Conformal keeps coverage stable as the model ages, but not at its nominal level: over a year of consumption data, errors are not exchangeable. The weekly cycle matters for a sixth of the series. The median series has a flat average week, but 52 series (17%) have weekends at least 10% lower. On those series, the weekly naive does clearly better than the lag-72 naive. The model, with its 24-hour profile, doesn\u0026rsquo;t capture that cycle either. For those series, adding a weekly profile would probably help more than changing the cadence. 8. The forecast service in production # The previous parts are about the model. This one is about the system that serves it.\nThe Open Inference Protocol and time # The ecosystem\u0026rsquo;s standard inference protocol, the Open Inference Protocol V2 (KServe, Seldon, MLServer, Triton), describes requests and responses made of tensors. I re-read it field by field: it has no field for time. No as_of, no horizon, no quantile levels. There is a model_version, but no training date and no feature version. The only extension point, parameters, is described by the specification as \u0026ldquo;Currently no parameters are defined\u0026rdquo;. What goes in there is therefore server-specific.\nA practical consequence: two identical requests sent on two different dates are the same request as far as the protocol is concerned. A cache keyed on (series, horizon) returns the same forecast a month later, and after a retrain. While writing the corresponding test, I found that my own cache key, in the contract, was missing the training date. With a key that is too wide, the cache can keep serving an old forecast without any signal, which can hide, for example, a retrain that never happened.\nOne solution is to put this information above the protocol, in a forecast gateway that adds as_of, horizon, quantiles, training date and versions, and builds the cache key from all of them. The same gateway can refuse to serve a model older than the freshness budget of the requested quantile, and fall back to the closed naive.\nOn the tooling side, two observations when checking installation on Python 3.13. mlserver, the protocol\u0026rsquo;s reference implementation, hasn\u0026rsquo;t had a stable release since June 2025 and declares Requires-Python \u0026lt;3.13. Yet pip install mlserver does not fail: it installs version 1.3.5, from July 2023, the last one without a declared bound, without any warning. And uv installs 1.7.1 without enforcing the upper bound. In both cases nothing signals the problem; it is therefore worth checking the version actually installed and the declared bounds, not just whether the installation succeeded. KServe, which is maintained, implements the same protocol.\nServing cost of a foundation model on CPU # Timestral makes a CPU-served foundation model conceivable. From the published latencies, refreshing 500,000 series daily with the 209M uses €163 of CPU a month. But if everything must be recomputed in a 30-minute nightly window, it takes 39 8-vCPU machines, which cost €7,862 a month if they stay allocated.\n500,000 series: the cost depends on peak capacity (×48 for the 209M), not on CPU-seconds consumed. That is a factor of 48. The cost then depends on the compute window more than on the cadence, unless the platform can release machines between windows. And at €1 of decision loss per series per month, with a refit every 6 hours, a TSFM only needs to do 0.13% better than the classic model to cover its compute cost. In this scenario compute weighs little; accuracy on the data at hand and the licence weigh more.\nThe Timestral weights licence # Timestral\u0026rsquo;s code is under Apache-2.0, but the weights are not. The LICENSE file in the four Hub repositories is a \u0026ldquo;Modified MIT License\u0026rdquo; containing an important clause: no rights of use if the consolidated monthly revenue of the company, or of the user\u0026rsquo;s employer, exceeds 20 million dollars, derivatives and fine-tunes included. For a consultant, that means the client\u0026rsquo;s revenue. Chronos-2\u0026rsquo;s weights, on the other hand, really are under Apache-2.0: this restriction is specific to Timestral.\nThe weights licence is therefore found on the Hub, not on the code repository\u0026rsquo;s badge, and how it applies depends on the client.\nA checkable service contract # I gathered these elements in a contract file, forecast_service.yaml, checked by a deterministic deployment gate before anything goes to production. Six rules: the licence is valid for this client, the compute window holds, the cadence stays within the freshness budget of every quantile, a single stability requirement is chosen, the retraining effect is measurable, and the closing lag applies to the backtest, the baseline and the trigger.\nPASS G1 weights licence ALLOW under the cap (Modified MIT), re-check monthly PASS G2 compute window 1 workers needed, 2 provisioned PASS G3 freshness budget cadence 24 vs tightest budget 168 FAIL G4 stability clause jump p95 0.540 vs clause 0.1 PASS G5 measurable effect holdout 5%, rdd logging True PASS G6 closing lag d = 48, missing: none BLOCKED (fail_closed) The candidate (Timestral-28M, 20,000 series, daily refit) is blocked: a daily refit publishes jumps of 54% of the scale at the 95th percentile for the 0.95 quantile, above the 10% threshold. With the frequency requirement instead (4.2% of origins revised, against a 15% cap), it passes. The choice between the two is a business one; the deployment gate simply makes it explicit.\n9. Measuring the effect of a retrain # Last question: once the policy is in place, how do you know whether retraining improved the forecasts? A common answer is a dashboard comparing the error before and after each refit.\nThat dashboard is biased by construction. The refit fires when the recent error exceeds a threshold. A high error comes partly from a real problem and partly from chance. After the refit, chance doesn\u0026rsquo;t necessarily repeat, and the error drops even if the refit changed nothing. This is regression to the mean. The measurement is exact; the selection rule produces the gap.\nTo put a number on it, I simulated 400,000 series, each with a bias linked to the model\u0026rsquo;s age, and a refit that removes 30% of that bias. True effect on the triggered series, measured by randomisation: −0.1053.\nDesign Estimate Reading before/after dashboard −0.3119 about three times the real effect placebo: same dashboard, refit with no effect −0.2072 197% of the real effect, with no change at all A/B among triggered series −0.0993, CI [−0.1141, −0.0844] the interval contains the real effect A/B, placebo +0.0066, CI [−0.0102, +0.0234] zero is in the interval About 66% of the improvement shown by the dashboard comes from regression to the mean. The gap also depends on the threshold:\nPlacebo effect, as a percentage of the real effect, against the share of the park triggered. The stricter the threshold (to the right), the larger the placebo effect. The stricter the threshold, the larger the placebo effect: 85% of the real effect when 64% of the park triggers, 284% when only 1.3% does. A strict threshold selects the series whose recent error owes most to chance, and whose error therefore drops the most on its own.\nThe holdout group and its cost # The classic approach is a holdout group: leave a small random share of triggered series without a refit, and compare. With a 5% holdout, the A/B estimates the effect within ±0.035. The plan put the cost of that holdout at 0.0057 of extra error \u0026ldquo;on the park\u0026rdquo;. That is actually the cost per triggered series. Since only 20% of series trigger, the cost per park series is 0.0012, five times less.\nRegression discontinuity at the threshold # The plan concluded that refitting 100% of triggered series makes the effect \u0026ldquo;incalculable by construction\u0026rdquo;. That is true for the average effect. But a deterministic threshold rule matches the setting of a regression discontinuity design (RDD): series just below and just above the threshold are comparable, and only the refit tells them apart. A local linear regression on each side measures the jump at the threshold.\nPost-refit error against the error that triggered (or not) the refit. The jump at the threshold (−0.063, standard error 0.003) is the refit\u0026rsquo;s effect at that point. With no holdout at all, the RDD recovers −0.059 to −0.064 for a true effect of −0.0625 at the threshold, and the corresponding placebo is close to zero. It needs no holdout group, only a log of the variable that triggers the refit.\nIt has a limit: it measures the effect at the threshold, not the average effect on triggered series (−0.1053), because series far above the threshold carry more bias and gain more from the refit. The two designs answer two different questions:\n\u0026ldquo;should we lower the threshold a little?\u0026rdquo; is a question for the RDD; \u0026ldquo;what does the retraining policy buy on average?\u0026rdquo; needs the holdout group. In the contract, I therefore required one or the other: a permanent holdout group, or logging the trigger variable.\n10. What I take away # These observations hold for the setups studied here: a simulated bench, a simple analytical model, one year of electricity consumption. I present them as such.\nThe three expectations don\u0026rsquo;t lead to the same frequency. On the bench, continuous retraining gives the best accuracy but publishes revisions on 12.6% of origins; monthly retraining publishes rarely, but in jumps of 293% of the scale. On the bench, detection matters less than what follows it. The same perfect detector goes from a MASE of 1.37 to 0.88 depending on the refit window and the calendar that goes with it. With a real CUSUM, restarting the window makes things worse instead: the value of this approach depends on how reliable detection is. The closing lag changed three measurements: the backtest error (about 6%), the comparison with the baseline on real data, and CUSUM\u0026rsquo;s targeting (from 40.9% to 20.0%). Size and frequency of revisions move in opposite directions, and mean churn doesn\u0026rsquo;t tell a stable model from a frozen one. In the local level model, the window and the cadence act on two distinct terms of the error. I only checked this separation in that model; with breaks (part 4), the two choices interact. The freshness gap between quantiles came largely from calibration and the baseline. After correction: a factor of 2 in simulation, around 1 on real data. On the electricity data, the freshness budget varies strongly by quarter, from 720 hours to zero. A before/after comparison overestimates the effect of a threshold-triggered retrain, all the more so as the threshold is strict. A holdout group or an RDD at the threshold can estimate the effect, each for a different question. Open questions. I did not test a real detector more precise than CUSUM, nor a restarted window with a minimum length. Transferring Timestral\u0026rsquo;s published latencies to an hourly park is not measured. And I did not look into why the second quarter of 2014 is an exception.\nA question to check your understanding. A service retrains a model when its error crosses a threshold. The dashboard shows −30% error after each refit. The threshold is raised to save compute, and the displayed gain moves to −45%. How should this be read? A stricter threshold selects series whose recent error owes more to chance, and regression to the mean grows (part 9). The displayed gain therefore doesn\u0026rsquo;t support a conclusion. Estimating the real effect requires a holdout group, or an RDD at the threshold if the trigger variable is logged, and an \u0026ldquo;after\u0026rdquo; error measured on closed data. Next. Part 9 was about a selection effect tied to threshold triggering. The same mechanism shows up in other interventions, for instance a promotion launched after a demand dip. My next research week will be about causal inference for time series: synthetic control, effects of a dated intervention, placebos in space and in time.\nReproduce and go further # The code, the seven notebooks and the 55 tests are in the gwils28/W44_retraining_frequency repository (Python 3.13, versions pinned with uv). Every table in this article comes from a uv run w44 … command, with fixed seeds. The real dataset is checked against its sha256 before use, and the protocol for it was committed before the results.\nSources\nZanotti, On the retraining frequency of global models in retail demand forecasting, Machine Learning with Applications (DOI 10.1016/j.mlwa.2025.100769). Zanotti, Analyzing the retraining frequency of global forecasting models: towards more stable forecasting systems, arXiv, 2026. Katz, Cost-sensitive retraining via posterior learning debt, arXiv, 2026. Kwak, Jung, Park, Pseudo-Label-Triggered Retraining from Forecast Errors for Online Time Series Forecasting, arXiv, 2026. Mistral AI, Timestral, announcement of 15 October 2026, and the weights licence on the Hub (mistralai/Timestral-*). Open Inference Protocol V2, specification. Data: Godahewa et al., Monash Electricity Hourly, Zenodo, CC-BY-4.0. ","date":"17 October 2026","externalUrl":null,"permalink":"/en/posts/frequence-reentrainement/","section":"Posts \u0026 notes","summary":"Accuracy, freshness and stability don’t call for the same retraining frequency. What I observed on a ten-policy bench, an analytical model, the Monash electricity data and a causal simulation.","title":"Retraining frequency in forecasting: accuracy, freshness and stability","type":"posts"},{"content":"","date":"17 October 2026","externalUrl":null,"permalink":"/en/tags/serving/","section":"Tags","summary":"","title":"Serving","type":"tags"},{"content":"","date":"17 October 2026","externalUrl":null,"permalink":"/tags/stabilit%C3%A9/","section":"Tags","summary":"","title":"Stabilité","type":"tags"},{"content":"","date":"17 October 2026","externalUrl":null,"permalink":"/en/tags/stability/","section":"Tags","summary":"","title":"Stability","type":"tags"},{"content":"","date":"8 October 2026","externalUrl":null,"permalink":"/en/tags/%C3%A9co2mix/","section":"Tags","summary":"","title":"ÉCO2mix","type":"tags"},{"content":"","date":"8 October 2026","externalUrl":null,"permalink":"/en/tags/hierarchical-reconciliation/","section":"Tags","summary":"","title":"Hierarchical Reconciliation","type":"tags"},{"content":" Understand hierarchical reconciliation well enough to know when it helps, when it hurts, and why. Not just call it: rewrite it, then put it to the test on real data. Stack Python 3.12NumPystatsforecastHierarchicalForecastBayesReconPyuvpytest Source code 1.07e-5\ngap to the library\nhome-made MinT(shrink) vs HierarchicalForecast +2.3%\nregional MASE\nMinT shrink vs bottom-up, over 119 éCO2mix origins in 2024 356\nbacktested origins\n2023 and 2024, MSTL and AutoETS, against naive and RTE Transparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of this page. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThis repository is the lab behind the post Hierarchical forecast reconciliation: promises and limits. The post tells what I learned; this page shows how the work is organised and how to reproduce it. The repository and its figures are in French.\nThe goal # Forecasts produced series by series don\u0026rsquo;t add up: the regions don\u0026rsquo;t sum to the national total, and someone has to arbitrate. Reconciliation (MinT first) promises to remove that arbitration. I wanted to find out three things:\nwrite the projection P = S(S'W⁻¹S)⁻¹S'W⁻¹ by hand, and match the library to the digit; find where MinT breaks: bias, quantiles, the numerical wall; check on real data whether reconciliation improves accuracy, or only coherence. The approach # The work follows six blocks, one folder each. The code lives in a single package (src/w37_reconciliation), the teaching notebooks call it, and each block has its README (results) and its TUTORIEL (how to rerun it).\n1 Research Four 2026 papers checked, a three-point summary and a note on what is overrated: reconciliation sold as an accuracy gain. 2 MinT from scratch Schäfer–Strimmer shrinkage, oblique projection, three assertions. Gap to HierarchicalForecast: 1.07e-5. It does not come from λ, as the plan assumed, but from the covariance estimator (centring and dividing by T−1). 3 System design A materialised batch job with a versioned YAML contract, a hash of the hierarchy and a quality gate that blocks publication. 4 Fundamentals MinT spreads the bias of a single forecast to every leaf. Applied to quantiles, it gives a vector that is coherent and wrong: 0.764 coverage instead of 0.900. Only joint sampling is calibrated. 5 éCO2mix mini-project 12 regions → France, RTE data 2013-2024, rolling-origin backtest, then a benchmark against naive forecasts and RTE\u0026rsquo;s own forecast, and three confirmation studies. Details below. 6 BayesReconPy On intermittent sales (M5), Gaussian MinT produces negative sales; Bayesian conditioning produces none, and is the most accurate. The éCO2mix mini-project # The business goal: deliver electricity demand forecasts where the 12 regions add up exactly to the France total.\nThe protocol, fixed before the results in configs/eco2mix.yaml:\nhourly data; SeasonalNaive(168) and MSTL as base models; a 52-week training window, strictly before each origin; a 24 h horizon; five reconciliations, compared by MASE level by level. The main result # The result comes in two steps. On the 4 origins of the main protocol, MinT shrink seemed to win (−2.2% regional MASE). Over 119 origins covering all of 2024, it does worse than bottom-up, on all 13 series.\nAggregate gain of MinT shrink over bottom-up (regions) by number of origins used, in 2024 and 2023 (labels in French). The grey band holds 90% of 200 random orderings: with 4 origins, anything goes. The cause is shown by a controlled experiment. MinT estimates its W matrix on in-sample residuals. MSTL\u0026rsquo;s are smoothing leftovers, correlated at 0.11 across regions, while the real errors are correlated at 0.56. Estimated on the errors of past origins instead, W brings MinT back level with bottom-up (−0.1%, not significant).\nThe benchmark: where the gain comes from # A ranking between reconciliations says nothing about what they are worth against doing nothing. So I put the five reconciliations into a full benchmark: four naive baselines (last value, 4-week average, same hour last week or the day before), MSTL alone, and RTE\u0026rsquo;s official day-ahead forecast. The latter is not a like-for-like competitor: it uses the weather forecast. As published, it also underestimates demand by about 1.1 GW throughout; a debiased version removes its mean error over the previous 52 weeks, with no leakage.\nMean MASE over the 119 origins of 2024, from simplest to most elaborate (labels in French). Reconciliations form a plateau; only the debiased RTE forecast, which has the weather, goes below it. The staircase reads in three steps:\nThe statistical model does almost everything. MSTL divides the seasonal naive error by 2.6 at France level (61% gain). Reconciliations form a plateau. Their France MASE ranges from 0.352 to 0.370, and bottom-up is its low point, at both levels. Reconciliation moves the error by a few percent; the base model had moved it by 60%. Only the weather goes below the plateau. The debiased RTE forecast is 13.6% better than bottom-up in 2024, and 12% in 2023. The ranking does not depend on the metric: it is the same in MASE, MAE (MW), RMSE and MAPE. It does depend on the hour being forecast, and that is the newest result.\nGain over bottom-up at each hour of the forecast day (2024, labels in French). At France level, the direct forecast and MinT win the first 6 hours, then lose for the rest of the day. At France level, the direct MSTL forecast of the total and MinT shrink beat bottom-up by 7.5% and 8.6% over hours 1 to 6: right after the origin, the last known value of the total is very informative, and MinT gives it weight. Further out, the total\u0026rsquo;s forecast drifts, while the sum of 12 regional forecasts cancels part of their errors. A day has 6 favourable hours and 18 unfavourable ones: bottom-up wins on average. For an intraday forecast, the conclusion could flip.\nThree confirmation studies # Added after the results, these studies change one thing at a time and keep the rest of the protocol:\nAnother year. Over 118 origins in 2023, the staircase is the same and MinT shrink still loses to bottom-up (−0.8% regional, −1.5% France), but the gaps are three times smaller and none is significant after Holm. Another base model. The W explanation makes a testable prediction: with a model whose in-sample residuals are genuine one-step errors, MinT\u0026rsquo;s handicap should vanish. AutoETS is a filter that only looks at the past. It is much less accurate than MSTL (France MASE 0.64 vs 0.37), but MinT\u0026rsquo;s handicap vanishes: −0.0% regional. The cause, replayed. Estimating W on the errors of past origins also erases the handicap in 2023 (−1.0% → −0.3% regional). From 3 to 99 past origins, MinT stays within ±3% of bottom-up: there is no magic window, it just needs real errors. The direct measurement of the mechanism fits in one table:\nBase model Correlation seen by W (in-sample) Real correlation (out of sample) MinT shrink vs BU (regions) MSTL 0.11 to 0.13 0.56 −2.8% AutoETS 0.49 to 0.57 0.66 −0.0% When W sees correlations close to reality, MinT loses nothing. When it sees nearly independent regions while they actually err together, MinT spreads its corrections badly. With a good W, it still does no better than bottom-up.\nAggregate gain over bottom-up in each study (labels in French). The 4 November Tuesdays are the exception; with MSTL, bottom-up is always ahead or level, and with AutoETS every method tightens around it. The verdict: over 356 origins and two base models, bottom-up is never beaten significantly. What weakens is the size of the effect: in 2023 or with AutoETS, choosing between BU, WLS and OLS makes almost no difference.\nWhat got in the way # Data before models. Nouvelle-Aquitaine stops at the end of 2024. The source also has a daylight-saving defect every year: duplicates in March, a missing hour in October. A missing row raises no error, it silently skews the total. 4 origins rank nothing. Drawing origins at random, MinT\u0026rsquo;s gain over bottom-up ranges from −10.8% to +4.3% with 4 origins. It takes 22 in 2024, and 96 in 2023 where the gap is smaller, for every draw to give the same sign. Hence paired tests with Holm correction before concluding. An average of ratios can lie. Against bottom-up, the debiased RTE forecast is 13.6% better in aggregate gain, but 51.6% worse in average per-origin gain: on calm days, bottom-up\u0026rsquo;s error is tiny and the ratio blows up. To compare very different methods, compare averages of errors, not averages of ratios. The naive λ doesn\u0026rsquo;t scale. The T × m × m tensor weighs 31 GB at m = 5,000. Going through the T × T Gram matrix brings that down to 0.6 GB, with the same result. Check a package before recommending it. BayesReconPy 0.5.0 no longer installs as is (PuLP 4), and its licence is contradictory. What I take away # Coherent doesn\u0026rsquo;t mean calibrated, nor more accurate: MinT guarantees coherence, the rest is a bonus. On a flat, highly correlated hierarchy, bottom-up is the most accurate choice and the easiest to defend: it is never beaten significantly, over two years and two base models. The lever is the base model and the information it gets: the weather is worth more than any reconciliation. A hidden assumption (in-sample W) can overturn a result: test it, don\u0026rsquo;t assume it. Here, three independent experiments point to the same cause. A conclusion is confirmed by changing one thing at a time: the year, the model, the way W is estimated. Reproduce # git clone https://github.com/gwils28/W37_hierarchical_reconciliation cd W37_hierarchical_reconciliation uv sync \u0026amp;\u0026amp; uv run pytest # 100 tests: 89 unit, 11 end-to-end make eco2mix # downloads the RTE data, backtest, report make eco2mix-robustness # 119 origins in 2024 (≈ 40 min) make eco2mix-confirmation # 2023, then AutoETS on 2024 (≈ 25 min) Data: Open Data Réseaux Énergies (ODRÉ) — RTE, Licence Ouverte v2.0.\n","date":"8 October 2026","externalUrl":null,"permalink":"/en/projects/w37-reconciliation-hierarchique/","section":"Projects","summary":"MinT rewritten from scratch, its assumptions tested one by one, and an éCO2mix mini-project (12 regions → France) where bottom-up beats the reference method, over 356 origins, two years and two base models.","title":"Hierarchical reconciliation: MinT from scratch, tested on electricity demand in 12 French regions","type":"projects"},{"content":" Know where the household\u0026rsquo;s money goes, month after month, what its assets are worth and when its plans will be funded, without handing its accounts to an online service. Everything runs on a machine at home. StatusIn development · phases 1 to 4 shipped, phase 5 under way TimelineSince October 2026 RoleSolo: specifications, architecture, development, operations Stack Python 3.13FastAPISQLAlchemy 2PostgreSQL 17NumPyReact 19TypeScriptTailwindDocker ComposeCaddyGitHub Actions Source code 137\nmerged pull requests\neach one blocked until CI is green 736\nautomated tests\nincluding 49 property tests on financial invariants 8\narchitecture decisions\nwritten down, with the rejected alternatives Transparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of this page. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nWork in progress. This page describes the repository as of 24 October 2026. The figures above are counted in the code, and will be updated with each phase shipped. There are fewer tests than on 15 October: the spending redesign removed bank import and everything that depended on it, tests included. Every screenshot shows fictional data. They come from a demo instance, filled with an invented household (Camille and Sam, ten months of spending, two years of net worth, nine goals). No amount, label or balance comes from real accounts. The problem # Consumer budgeting apps almost all ask for the same thing: access to your bank accounts, and hosting your financial history on their servers. Spreadsheets ask for nothing, but get abandoned after three months: entry is slow, and the totals break at the first copy and paste.\nI wanted a tool in between:\nprivate by design: the data stays on a household machine, with no telemetry and no third-party service; quick to keep up: one entry session a week, at the keyboard, with nothing to sort afterwards; exact to the cent: every total must be recomputable from the lines entered; looking ahead: tell whether the household\u0026rsquo;s plans will be funded on time, and where its net worth could go; ready for data science: simulation, forecasting and a natural-language assistant must be addable without rebuilding the foundations. It is also my first product: a project meant to last, not a throwaway experiment. So it is where I apply my way of working with Claude Code.\nWhere the project stands # Phase 0 · Foundations Shipped Technical groundwork Specifications, architecture and decisions written before the first line of code. Repository, tooling, continuous integration, Docker Compose stack, Claude Code configuration and a visual identity borrowed from this blog. Phases 1 and 2 · Budget Shipped, then redone The heart of the product Two-factor authentication, household and invitations, accounts, bank import and reconciliation, categorisation rules, transfers, budget plans by period with their review, recurring items, reports and an activity log. The whole accounts-and-import side was later replaced by the v1.0 redesign. Phase 3 · Net worth Shipped Assets and liabilities, month by month A monthly net-worth matrix: categories and lines entered by hand, debts, net worth, and four charts (history, allocation as a sunburst and a mosaic, contributions to the change). Phase 4 · Financial goals Shipped An amount to gather by a date Goals arranged in a pyramid (the emergency fund as its foundation, then long, medium and short term, under a golden capstone for treats), funded by envelopes taken from the net-worth lines, with monthly tracking and an effort plan for the month. v1.0 redesign · Simpler spending Shipped What I actually do, not what I had imagined No more accounts, import or transfers: every expense is a line entered by hand, in a grid built for the keyboard. One level of categories, fixed expenses to confirm, monthly targets with their matrix and a year view. The spending data starts afresh. Phase 5 · Analytics and data science Under way Where my trade comes in Shipped: the goals analysis (six views and points to look at, computed by rules) and the Monte Carlo engine of the net-worth projection. Next: the projection page, the chance of reaching each goal, retirement, then cash-flow forecasting once the spending has enough history. Phase 6 · Local generative AI Planned With nothing leaving the machine An assistant that answers natural-language questions about your own data, using a local language model (Ollama), restricted to a read-only analytics view. Why bank import went away # Import was where the project started: drop in the bank\u0026rsquo;s CSV export and let the rules sort the expenses. In practice, I already entered my expenses by hand every week, and the statement then arrived as a chore: rows to sort again, duplicates of what was already entered, a reconciliation to check. So v1.0 removed accounts, import, transfers, rules, recurring items, the budget month and the plans, with their data, to keep what is useful: quick entry and a readable budget. The deviation is logged in the repository\u0026rsquo;s README, like every deviation from the original plan.\nWhat works today # One household, three points of view # The owner creates the household and invites the other members with a single-use link; there is no public sign-up. Login with a password (argon2id) and a mandatory TOTP code.\nSpending belongs to the household: every member sees and edits every line. Net worth and goals, on the other hand, have a visibility, line by line:\nprivate: only its owner sees it; shared: the whole household sees it, but it still belongs to its owner; joint: the household\u0026rsquo;s common holding, which belongs to nobody. Net worth and goals can then be read at the level of the household, the joint holdings or my lines. The app is arranged in three spaces, Spending, Wealth and Goals; the overview, at the top, sums up all three.\nOctober for the whole household: net worth and its change, the progress of each tier of the pyramid, then the month\u0026rsquo;s figures. Fictional data. Quick entry, at the keyboard # The spending space\u0026rsquo;s main page is a grid: one month, its lines, every cell editable in place. An empty row always waits at the bottom; Tab moves to the next cell, Enter saves and opens a new row with the same date. Everything works without the mouse.\nAbove it, a quick line takes an expense typed in any order: bakery 12.50 yesterday, 45 train 13/10 or groceries 87.40 #home.\nthe date is recognised (\u0026ldquo;yesterday\u0026rdquo;, a weekday, dd/mm); the amount is the last number, the others stay in the label; #word forces the category that starts with that word; the rest is the label, and its category comes from the label memory: picking a label already used brings back the category of its latest line. The quick line has understood “cinema 24.50 friday #leisure” before it is even saved; below, the fixed expenses to confirm, greyed, then the month\u0026rsquo;s lines. Fictional data. A header recalls the last line entered and when, and a dot marks it in the grid, to pick up where you stopped.\nThe green dot marks the last line entered; the empty row waits for the next one, with the same date. Fictional data. A line can also be split across several categories: the last part takes the rest, so the sum always comes out exact.\nA hypermarket run: €88.51 of groceries, the rest (€36) for the home. Fictional data. Categories of three kinds # One level of categories, each of a kind:\nexpense: rent, groceries, leisure; savings: money put aside, never spending, but it reduces the month\u0026rsquo;s rest; income: salaries, benefits, refunds. A month\u0026rsquo;s income is the sum of its income lines, and the rest is income minus spending minus savings. One-off categories (rent, subscriptions), paid in one go, are never judged against the pace of the month.\nFixed expenses # Rent, the childminder, subscriptions, insurance: a fixed expense describes a line that comes back every month, quarter or year. Each occurrence due shows greyed at the top of the month\u0026rsquo;s grid. Enter confirms it and creates the line, editable first; it can also be skipped for the month. Nothing is ever created by itself.\nRent, the childminder, subscriptions: each fixed expense with its frequency, day and category. Fictional data. The overview lists what is still to confirm and the projected rest at the end of the month.\nThree October occurrences still to confirm, and the projected rest once they are paid. Fictional data. Monthly targets # Each expense category can get a monthly target, set from a month on and kept until changed; past months keep theirs. A savings category gets a minimum to reach.\nthe month so far, on the overview: one bar per category, ranked by share used, with a mark for the share of the month elapsed, so spending ahead of the calendar shows before it overruns; the matrix: one row per category, one column per month, each cell coloured by its gap to the target (green below, grey within ±5 %, burnt orange above); the year: the targets to date and for the whole year, the gap, and how many months were over, on or under target. On 24 October: transport and shopping have overrun their target; the vertical mark shows the share of the month elapsed, and rent, paid in one go, is not judged against the pace. Fictional data. Six months of budget: income received at the top, then each category against its target. Green: under; grey: within ±5 %; orange: over. Fictional data. 2026 to date: cumulated target, spent, gap, and how many months were over, on and under target. Fictional data. Reports over any period # A month, a quarter, a year or any period up to ten years, compared with the previous period of the same length. A running period stops at the current month: \u0026ldquo;this year\u0026rdquo; is compared with the same months last year, not with a whole year.\nSeptember against August: the totals, then each category with its average, its share and its change. Fictional data. A report also gives a monthly chart stacked by category, spending accumulated day by day, and the period\u0026rsquo;s money flows: income on the left; spending, savings and the unallocated rest on the right, both sides always adding up to the same total.\n2026 month by month: the six largest categories in colour, the others in grey, against the income line. Fictional data. 2026\u0026rsquo;s flows: salaries and benefits on the left; each category, savings and the unallocated rest on the right. Fictional data. Net worth, month by month # Net worth is tracked as in a spreadsheet: one line per holding, filed in a category, one column per month, and a value entered by hand once a month.\nMonthly tracking: net worth, gross assets and debts, then the lines by category and each one\u0026rsquo;s share. Fictional data. a month without a value carries the previous one forward, shown as carried; \u0026ldquo;Update this month\u0026rdquo; lists every line with its last value, to enter them all at once; each line has an owner and a visibility; a regulated savings account (Livret A, LDDS, LEP, PEL, Livret jeune) knows its ceiling. An Analysis page gives four charts, each category in its colour. Since ten hues cannot all stay apart for every colour-vision deficiency, every chart also keeps its labels and its data table.\nCategories stacked month by month, debts below zero, and net worth as a line. Fictional data. October\u0026rsquo;s allocation: categories then lines in the sunburst, one tile per category in the mosaic. Fictional data. From November 2025 to October 2026, each category\u0026rsquo;s contribution to net worth; the loan being repaid lifts it. Fictional data. Goals, in a pyramid # A goal is an amount to gather by a date: an emergency fund, a trip, a home deposit. Goals are arranged in a pyramid, drawn in stone:\nthe foundation is a Roman column: the emergency fund, filled first. The app proposes its amount, six months of fixed expenses by default; three tiers, set by the deadline and never chosen: the long term (beyond five years) at the base, the medium term in the middle, the short term (twelve months at most) at the top. A goal climbs a tier by itself as its deadline nears; a golden capstone for treats (boxing gloves, a record player, a fine bottle of rum), with no deadline, locked until the essentials are in order: the emergency fund full and at least three quarters of the goals ahead. The household\u0026rsquo;s pyramid: the emergency fund\u0026rsquo;s column at 96 %, the three tiers, the locked capstone; on the right, the month\u0026rsquo;s effort plan, the emergency fund served first. Fictional data. A goal needs nothing more to enter: it is funded by envelopes, the part of a net-worth line set aside for it. When the line rises, each goal receives its share of the rise; when it falls, the fall comes out of the free part first, and goals only lose once the line drops below what they hold, treats first and the emergency fund last.\nflowchart LR L[\"Net-worth lineEuro-fund life insurance\"] --\u003e|50 % of rises| A[\"EnvelopeTrip to Portugal\"] L --\u003e|30 % of rises| B[\"EnvelopeNew car\"] L --\u003e|20 % of rises| C[\"EnvelopeHome deposit\"] L -.-\u003e|falls first| F[\"Free part\"] Each goal has its status (reached, ahead, on track, behind), against a straight path from start to target, and its estimates: the monthly effort required, the actual effort over the last six months, the estimated date at that pace. The month\u0026rsquo;s effort plan shares out the household\u0026rsquo;s savings capacity: the emergency fund first, up to an adjustable share, then the other goals by their required effort, then the unlocked treats. It is a recommendation: nothing is moved.\nThe short term: two goals ahead, each with its gauge, its pace marker and its estimates. Fictional data. The capstone: two treats locked until the essentials are in order. Fictional data. Each goal also has its page: its estimates, its trajectory, its envelopes line by line and its month-by-month tracking.\nTrip to Portugal: three months ahead; the trajectory shows the values, the expected line and the projection at the current pace. Fictional data. Its envelope on the life insurance, then each month: reached, gained, expected and gap. Fictional data. The goals analysis # An Analysis tab looks at the goals together, at a reference month: the last month in which every line funding them has a value entered, so that a month not yet entered never passes for a month without gain.\nrequired effort against capacity over the next two years, over-committed months shaded; a timeline: each goal from its start to its deadline, with its estimated date at the actual pace and at the plan\u0026rsquo;s pace; horizon and risk: each goal placed by the months left and the volatility of its envelopes, with zones to watch (short and volatile, long and eroded by inflation) and the loss in one outcome in ten; where the money sits: a flow diagram from categories to lines, then to the envelopes and the free part; status month by month for each goal. Required efforts stacked by goal against the capacity: November 2026 goes over, while Sam\u0026rsquo;s emergency fund fills. Fictional data. Each goal from its start to its deadline: the estimated date at the actual pace (●) and at the plan\u0026rsquo;s pace (◇). Fictional data. The e-bike, six months from its deadline, is funded by volatile lines: it falls in the zone to watch. Fictional data. From categories to lines, then to the goals\u0026rsquo; envelopes and each line\u0026rsquo;s free part. Fictional data. Each goal\u0026rsquo;s status, month after month. Fictional data. Above them, points to look at, computed by some fifteen fixed rules: a goal behind for three months, an emergency fund too thin or held on volatile lines, a savings account close to its ceiling, money idle on a current account, goals that ask for more than the capacity (with the postponement that would be enough). Each point gives the facts, the figures, why it matters and options; never an instruction to buy or sell, nor a named product. Three at most are shown, and each member can snooze or dismiss one.\nThree points to look at: an emergency fund covering less than three months of fixed expenses, two near goals funded by volatile lines. Fictional data. Verified backups, logged actions # A full backup every night, checked before it is kept, with rotation (7 days, 4 weeks, 12 months) and a written restore procedure.\nAn activity log keeps sensitive actions, written in the same transaction as the action itself: sign-ins, invitations, deletions. The owner sees the whole household, a member their own entries.\nSign-ins (one with a wrong code), an invitation sent then cancelled, settings, a line deleted. Fictional data. How it is built # A simple architecture, designed for what comes next # Everything starts from a single docker compose up: a Caddy proxy with TLS on the local network, a FastAPI API, a PostgreSQL database and a backup service. The React interface is served from the same origin as the API, which allows cookie sessions with no CORS configuration.\nflowchart LR N[\"Browserlocal network\"] --\u003e|HTTPS| C[\"CaddyTLS + React interface\"] C --\u003e|/api| A[\"FastAPI APIkwak_api\"] A --\u003e K[\"kwak_coreall the business rules\"] A -.-\u003e|page to come| M[\"kwak_analyticsMonte Carlo, NumPy\"] M --\u003e K A --\u003e P[(\"PostgreSQL\")] S[\"Nightly backupchecked pg_dump\"] --\u003e P On the Python side, the code is split into packages with a strict rule: kwak_core holds all the business rules (amounts, the quick line, splits, monthly targets, fixed expenses, reports, net worth, envelopes and the effort plan, the analysis rules) and depends on nothing. It is tested without a database. The API and the kwak_analytics layer depend on it, never the other way round: a given figure has only one definition.\nThe contract between the API and the interface is the OpenAPI schema: the TypeScript client is generated from it, never written by hand. CI fails if someone forgets to regenerate it.\nMoney does not tolerate \u0026ldquo;roughly\u0026rdquo; # Amounts are exact decimals (Decimal in Python, NUMERIC in the database), never floats. And the rules that must always hold are written in the specifications, then checked by property tests (Hypothesis): instead of a few hand-picked examples, the test generates hundreds of cases and looks for the one that breaks the rule.\nInvariant What the test checks Quick line Every word typed ends up in exactly one field: date, amount, category or label Splits The parts of a split line add up exactly to its amount, each positive Savings Savings are never spending, and the month\u0026rsquo;s rest is the sum of its lines Monthly targets One target in force per category and month, and the yearly target is the sum of the months Fixed expenses A line fulfils at most one occurrence, and the projected rest deducts exactly those left Consistency A report, the matrix, the year and the month so far always give the same figures Flows Both sides of the flow diagram add up to the same total Net worth A month\u0026rsquo;s net worth equals its assets minus its debts, and the household is exactly the sum of its scopes Envelopes A line\u0026rsquo;s envelopes are never negative and never exceed its value, nor a goal its target Effort plan The plan never hands out more than the capacity, and the emergency fund is served first Capstone A locked treat receives nothing, neither from rises nor from the plan Analysis A goal\u0026rsquo;s weights on its lines sum to 1, and the postponement proposed is enough, one month less is not Projection With no volatility, every path matches the exact calculation; money is conserved every month API tests run against a real PostgreSQL started in a container, never against a mocked database.\nA Monte Carlo projection # The first piece of phase 5 is the engine that will tell where net worth could be in 1 to 40 years. The starting point is the current month of the matrix; each category has an expected return and a volatility, both adjustable, and the household\u0026rsquo;s monthly savings are added every month, shared across the categories.\n5,000 paths, month by month: each category grows by a lognormal return drawn from its own random stream, calibrated so that twelve months compound to exactly the chosen annual mean and volatility; a fixed seed: the same assumptions always give the same result, and two scenarios share the same draws, so their difference comes from the assumptions only; debts are repaid by their monthly payment and rate, and what they free joins the savings; negative savings draw on current accounts then savings accounts, never on the home or locked-in holdings; what is missing becomes a debt borrowed at 8 %. It is the one place where money goes through floating point, for speed: the inputs are Decimal, and the results come back to it, to the cent. The projection\u0026rsquo;s invariants (money conserved, ordered percentiles, identical paths with no volatility, simulated mean equal to the chosen return) are tested like the rest, and an engine version is pinned by a golden test.\nDecisions written down # Every structural choice gets a decision record (ADR), dated, with its rejected alternatives. A decision that changes gets a new record; the old one is never rewritten.\nDecision Rather than Self-hosting with Docker Compose Cloud hosting: needless cost and exposed financial data FastAPI, SQLAlchemy 2, Alembic Django: heavier, and the admin interface is of no use PostgreSQL, then DuckDB for analytics SQLite alone: concurrent writes, no vector extension Euro only Full multi-currency accounting, too costly for a single real need Server sessions and mandatory TOTP JWT tokens, hard to revoke Local language model (Ollama) A cloud API: the data must never leave the machine Deviations from the original plan are logged in the README too, to the day.\nWhat\u0026rsquo;s next # The projection engine is ready: what is left is its page, a fan of possible outcomes (median, 25–75 % and 10–90 % bands), the chance of reaching an amount by a date, the median by category and two scenarios to compare, with its limits spelled out. The same engine will then give each goal its chance of being reached on time, then a retirement projection.\nCash-flow forecasting, for its part, waits for the spending to build up enough history again since v1.0. It will be treated as a real time-series problem: a protocol written in advance, a backtest with no leakage from the future, and a naive model to beat before anything else. This page will be updated with each step shipped.\nTry it # git clone https://github.com/gwils28/kwak-finance cd kwak-finance make setup # dependencies and Git hooks make check # everything CI runs make up # the full stack on https://localhost:8443 The steps to create the owner account and the encryption key are in the README, and the app itself offers, at the bottom of every page, a guide to start, stop and update it.\n","date":"8 October 2026","externalUrl":null,"permalink":"/en/projects/kwak-finance/","section":"Projects","summary":"Self-hosted web app for household budgeting and net worth. Keyboard entry, fixed expenses, budget targets, reports, a monthly net-worth matrix, financial goals in a pyramid with their analysis; a Monte Carlo net-worth projection is on its way.","title":"Kwak Finance: a household's budget and net worth, self-hosted","type":"projects"},{"content":"","date":"8 October 2026","externalUrl":null,"permalink":"/en/tags/mint/","section":"Tags","summary":"","title":"MinT","type":"tags"},{"content":"","date":"8 October 2026","externalUrl":null,"permalink":"/en/tags/postgresql/","section":"Tags","summary":"","title":"Postgresql","type":"tags"},{"content":"","date":"8 October 2026","externalUrl":null,"permalink":"/tags/r%C3%A9conciliation-hi%C3%A9rarchique/","section":"Tags","summary":"","title":"Réconciliation Hiérarchique","type":"tags"},{"content":" The national forecast doesn\u0026rsquo;t match the sum of the regional forecasts. Every month, someone has to decide which number wins. Hierarchical reconciliation promises to remove that arbitration. It keeps that promise… and only that one. Reading time: about 22 minutes. Every formula is followed by its intuition, so you can skim the maths without losing the thread. The figures come from the original notebooks and are labelled in French.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nThis article distils my active research for week 37 (September 2026). I didn\u0026rsquo;t just read: I checked the sources, rewrote MinT from scratch, ran controlled experiments, reconciled the electricity demand forecasts of France\u0026rsquo;s 12 mainland regions, and tested a Bayesian reconciliation package. Everything is reproducible, with pinned versions, in the W37_hierarchical_reconciliation repository.\nIn short. Reconciliation guarantees coherence, not accuracy, and not calibration. I found four pieces of evidence:\nIn theory: a bias in a single forecast contaminates all the others. Probabilistically: an interval advertised as 90 % at the total level can cover only 76 %. On real data: over 119 days of 2024, the simplest method (bottom-up) beats MinT. On retail sales: 98 % of the intervals reconciled by MinT start below zero. 1. The problem: forecasts that don\u0026rsquo;t add up # Take a cooperative that sells in two French regions, each through two channels: supermarkets (GMS) and food service (RHD). That\u0026rsquo;s 7 series to forecast: the total, 2 regions and 4 region × channel pairs. Each series has its own model, tuned as well as possible. But nobody asked the models to agree with each other.\nThe regions forecast 1,030 tonnes, the total forecasts 1,000. Which one do you trust? The structure is described by a summing matrix \\(S\\). The bottom series, the leaves \\(b_t\\), determine everything else:\n$$ y_t = S\\,b_t, \\qquad S = \\begin{pmatrix} 1 \u0026 1 \u0026 1 \u0026 1\\\\ 1 \u0026 1 \u0026 0 \u0026 0\\\\ 0 \u0026 0 \u0026 1 \u0026 1\\\\ 1 \u0026 0 \u0026 0 \u0026 0\\\\ 0 \u0026 1 \u0026 0 \u0026 0\\\\ 0 \u0026 0 \u0026 1 \u0026 0\\\\ 0 \u0026 0 \u0026 0 \u0026 1 \\end{pmatrix} \\begin{matrix}\\leftarrow \\text{total}\\\\ \\leftarrow \\text{Bretagne}\\\\ \\leftarrow \\text{Pays de la Loire}\\\\ \\leftarrow \\text{Bretagne / GMS}\\\\ \\leftarrow \\text{Bretagne / RHD}\\\\ \\leftarrow \\text{PdL / GMS}\\\\ \\leftarrow \\text{PdL / RHD}\\end{matrix} $$A forecast vector is coherent if it can be written \\(S\\,b\\) for some \\(b\\): every sum adds up. Base forecasts \\(\\hat y\\), produced series by series, almost never are.\nThe two classic answers throw information away:\nbottom-up sums the leaves and ignores the forecasts of the aggregates, which are often more stable; top-down splits the total using historical proportions and ignores what the leaf models know. If it computes those proportions on a history that includes the test period, it also leaks information from the future, and nothing in the metrics will flag it. Optimal reconciliation sets out to use all the forecasts at once.\n2. MinT in five minutes: reconciling is projecting # The framework # We look for a matrix \\(G\\) that maps the \\(m\\) base forecasts to \\(n_b\\) leaves, which are then summed back up:\n$$ \\tilde y = S\\,G\\,\\hat y . $$Two conditions constrain the choice of \\(G\\):\nCondition Formula What it means unbiasedness preservation \\(SGS = S\\) a vector that is already coherent is left unchanged reconciled error variance \\(\\operatorname{Var}(y - \\tilde y) = S G W G^\\top S^\\top\\) with \\(W = \\operatorname{Var}(y - \\hat y)\\), the covariance of the base errors MinT (Minimum Trace, Wickramasuriya, Athanasopoulos and Hyndman, 2019) minimises the trace of that variance subject to \\(SGS = S\\). The solution has a closed form:\n$$ G = \\left(S^\\top W^{-1} S\\right)^{-1} S^\\top W^{-1}, \\qquad P = S\\,G . $$ The geometric intuition # \\(P\\) is a projection (\\(P^2 = P\\)). Coherent vectors form a subspace of dimension \\(n_b\\) in \\(\\mathbb{R}^m\\), and reconciling means bringing \\(\\hat y\\) back onto that subspace. The whole question is: in which direction?\nOLS takes the shortest path. MinT looks for the most plausible coherent point given the errors. OLS (\\(W = I\\)) projects orthogonally: every series is treated as equally reliable. MinT projects according to \\(W^{-1}\\). It picks the coherent point that is most plausible given the shape of the errors: it moves badly forecast series a lot and the others very little. Bottom-up is a projection too, with \\(G = [\\,0 \\mid I\\,]\\): it only moves the aggregates. Method Matrix \\(W\\) used What it assumes bottom-up — (\\(G = [0 \\mid I]\\)) the aggregates add nothing OLS \\(I\\) all errors are alike WLS struct \\(\\operatorname{diag}(S\\mathbf{1})\\) variance proportional to the number of aggregated leaves WLS var \\(\\operatorname{diag}(\\hat W)\\) individual variances, no correlations MinT shrink \\(\\lambda D + (1 - \\lambda)\\hat W\\) variances and correlations, regularised The detail that will matter later # \\(W\\) is unknown. In practice it is estimated on the in-sample residuals of the base models, then regularised with Schäfer–Strimmer shrinkage: correlations are pulled towards zero with an intensity \\(\\lambda \\in [0, 1]\\) estimated from the data. Keep this in mind: it\u0026rsquo;s what makes MinT lose on real data (section 6).\n3. What the 2026 literature says # I read six papers, chosen to cover MinT\u0026rsquo;s blind spots. For each one, I checked what was claimed about it against its official abstract. The reading plan sometimes oversold them.\nEach recent paper tackles one of MinT\u0026rsquo;s blind spots. Paper What it brings What the check corrected Li, Chen, Taylor, Mao — A Forecast Combination Framework for Hierarchical and Grouped Time Series Reconciliation (2608.13886) The optimal combination weights exactly recover MinT. Forty years of forecast-combination literature (shrinkage, penalties, positivity) become reusable. — Rønlev-Knudsen, Madsen, Møller — Online forecast reconciliation using linear models (2606.23326) Reconciliation as a regularised regression, updated recursively: no need to re-estimate \\(W\\) at every run. The case study is a temporal hierarchy, not a geographical one. Validate before transposing. Nugteren, Abolghasemi, Mengersen, Drovandi — Hierarchical Bayes meets hierarchical forecasting (2606.23009) A Bayesian model that targets the level where the decision is made. Coherence is soft (penalised), not exact. Wang, Johnson, Klee, Malloy — Billions-Scale Forecast Reconciliation (2602.05030) Reconciliation over more than four billion values; least squares and share-based allocation coincide under conditions. — Panagiotelis, Gamakumara, Athanasopoulos, Hyndman — Probabilistic forecast reconciliation (EJOR, 2023) The proper definition of reconciling a distribution, evaluated with the energy score and the variogram score. — Biswas, Zambon, Nespoli, Corani — Nonlinear Probabilistic Forecast Reconciliation (2604.26668) What happens when \\(y = Sb\\) no longer holds: logs, ratios, average prices. — The check was limited to abstracts and official pages, not the full PDFs.\nThree trends stand out:\nReconciliation is becoming an estimator, not a post-processing step. Seen as forecast combination or as regularised regression, it can be regularised, tested and defended in an architecture review. The scale objections are gone. Four billion values on one side, recursive estimation on the other. The open front is probabilistic and nonlinear reconciliation. That is exactly where point-forecast MinT has nothing left to say. Section 5 shows it. 4. Rewriting MinT yourself # Calling MinTrace(method=\u0026quot;mint_shrink\u0026quot;) takes one line. But until you have written the formula yourself, you can\u0026rsquo;t explain a strange result. So I reimplemented MinT shrink in NumPy, and I didn\u0026rsquo;t allow myself to read the HierarchicalForecast code until mine was frozen. Here is the core, condensed:\nimport numpy as np def mint_shrink(E, S): \u0026#34;\u0026#34;\u0026#34;E: in-sample residuals (T × m), S: summing matrix (m × nb). Returns P = SG.\u0026#34;\u0026#34;\u0026#34; T, m = E.shape W1 = E.T @ E / T # sample covariance Z = E / np.sqrt(np.diag(W1)) # standardised residuals R = Z.T @ Z / T # correlations w = Z[:, :, None] * Z[:, None, :] # T × m × m: naive, see below var_r = T / (T - 1) ** 3 * ((w - R) ** 2).sum(axis=0) off = ~np.eye(m, dtype=bool) lam = np.clip(var_r[off].sum() / (R[off] ** 2).sum(), 0, 1) # Schäfer–Strimmer λ W = lam * np.diag(np.diag(W1)) + (1 - lam) * W1 WiS = np.linalg.solve(W, S) # W⁻¹S, without inverting W G = np.linalg.solve(S.T @ WiS, WiS.T) # (S\u0026#39;W⁻¹S)⁻¹ S\u0026#39;W⁻¹ return S @ G On a synthetic hierarchy (2 regions × 2 channels, 80 quarters, AutoETS models), the checks pass:\nCheck Threshold Measured \\(P^2 = P\\) and \\(SGS = S\\) \u0026lt; 1e-12 2.2e-16 incoherence of the reconciled forecasts machine precision 5.7e-14 relative gap to HierarchicalForecast 1.5.1 \u0026lt; 1e-3 1.07e-5 non-zero gap \u0026gt; 0 yes The last check is deliberate: an exactly zero gap would have meant I had copied their code. That small residual gap taught me three things.\nLesson 1: one method name covers several estimators # The research plan put the gap down to a difference in how \\(\\lambda\\) is computed. That\u0026rsquo;s wrong: the two \\(\\lambda\\) values agree to within 2e-5. About 99.8 % of the gap comes from the base covariance. The library centres the residuals and divides by \\(T - 1\\) (np.cov), whereas my version uses \\(E^\\top E / T\\). With the same convention, the gap drops to 3e-6.\nThe practical consequence: mint_shrink covers at least three implementation choices (centring, ddof, the \\(\\lambda\\) variant) that change the published number. If two teams compare their runs, they need to pin the version and document the estimator.\nLesson 2: shrinkage is a condition for existence # Even with 72 observations for 6 series, the sample covariance is ill-conditioned (condition number ≈ 3·10⁴). An aggregate\u0026rsquo;s residual is almost the sum of its leaves\u0026rsquo; residuals, so the columns of \\(E\\) are nearly collinear. Shrinkage brings the condition number down to ≈ 23.\nWith more series than observations (\\(m \u003e T\\)), it gets worse: \\(\\hat W\\) has rank \\(T\\), so it is singular. With \\(T = 4\\) and \\(m = 6\\), its condition number reaches 1.7·10¹⁷. Shrinkage then picks \\(\\lambda = 0.907\\) on its own and brings the condition number down to 10. Shrinkage is not a convenience: without it, MinT doesn\u0026rsquo;t exist on a real catalogue. 40,000 products over three years of weekly history means \\(m/T \\approx 250\\).\nLesson 3: the wall is also a memory wall # The line marked \u0026ldquo;naive\u0026rdquo; builds a \\(T \\times m \\times m\\) tensor: 31 GB for 5,000 series. My first version got the Jupyter kernel killed by the operating system. Yet \\(\\lambda\\) only depends on off-diagonal sums, which can be computed from the \\(T \\times T\\) Gram matrix. Memory goes from \\(O(T m^2)\\) to \\(O(T^2 + Tm)\\), i.e. 0.6 GB, for a result identical to within 1e-10. Beyond that, it\u0026rsquo;s \\(W\\) itself that must no longer be formed. A diagonal plus a rank-\\(T\\) matrix can be inverted with the Woodbury identity, without ever building an \\(m \\times m\\) matrix.\n5. Three places where MinT breaks # MinT is optimal under assumptions: unbiased base forecasts, known \\(W\\), linear constraints, continuous quantities. I set up a controlled experiment for each of these assumptions. Each time, I looked for the point where it gives way.\n5.1 MinT spreads bias # The condition \\(SGS = S\\) preserves unbiasedness; it doesn\u0026rsquo;t create it. The counter-example fits in four series: one total and three leaves. The forecasts are exact everywhere except for a +15 bias on the total. \\(W\\) trusts the total 9 times more than each leaf.\nSeries Truth Base Bottom-up MinT MinT error Total 60 75 60 74.46 14.46 Leaf 1 10 10 10 14.82 4.82 Leaf 2 20 20 20 24.82 4.82 Leaf 3 30 30 30 34.82 4.82 Bottom-up ignores the total and sees nothing. MinT, which trusts it, injects the bias into every leaf. Because reconciliation is linear, the reconciled bias equals \\(P \\times\\) the base bias: it spreads along a column of \\(P\\).\nA bias on the total alone ends up in every leaf. It isn\u0026rsquo;t all or nothing: there is a threshold. In simulation (20,000 draws), MinT stays ahead of bottom-up as long as the bias on the total is below ≈ 4.5. Above that, it becomes worse at the total; above ≈ 5.25, it is also worse at the leaves, i.e. at every level.\nMean error as a function of the bias on the total: MinT loses past a threshold, first at the total, then at the leaves. The reflex. Test the mean of the residuals at each level, then correct the base forecasts before reconciling. Never adjust a reconciled forecast afterwards: changing a single series breaks coherence. 5.2 MinT reconciles means, not quantiles # This is the heart of the week. An inventory manager doesn\u0026rsquo;t decide on a mean. They decide on a quantile: the stock that covers demand 90 % of the time. And the quantile of a sum is not the sum of the quantiles.\nAdding up the regions\u0026rsquo; 90 % quantiles gives a total that covers 93.1 % of the time, not 90 %. Consequence: no vector of quantiles is coherent. Coherence is a property of the joint distribution, not of the marginals.\nThe experiment is fully simulated, so the truth is known: one total and three Gaussian regions, correlated at \\(\\rho = 0.6\\) (the weather moves them together). Each base model is perfect on its own series. The means are already coherent, so everything we observe is a pure \u0026ldquo;quantile\u0026rdquo; effect. We compare three ways of getting reconciled 90 % quantiles:\nA. Apply \\(P\\) directly to the quantiles. This is the most common mistake. B. Draw samples from each series independently, project each sample, then read off the quantiles. C. The same, but with joint samples that respect the dependence between series. Series Coverage A Coverage B Coverage C Total 0.922 0.764 0.900 Region 1 0.887 0.878 0.900 Region 2 0.890 0.888 0.899 Region 3 0.893 0.894 0.901 Target: 0.900. 400,000 draws, fixed seed.\nOnly method C, which keeps the dependence, reaches the target coverage at every level. How to read this table:\nA is wrong in both directions at once. It over-covers at the total and under-covers in the regions. No global factor can fix it. The resulting vector adds up perfectly, but it isn\u0026rsquo;t the quantile of anything. B, which looks rigorous, is the worst: 76 % coverage at the total for 90 % advertised. Projecting samples is the right mechanism. But independent draws erase the dependence, and the variance of the sum is massively underestimated. C is calibrated everywhere. Its only difference from B is the dependence structure. Sweeping the correlation makes the result hard to dispute:\nAs correlation rises, the naive error (A) becomes less and less visible and the sophisticated error (B) becomes more and more costly. \\(\\rho\\) Gap \u0026ldquo;sum of q90s\u0026rdquo; / \u0026ldquo;q90 of the sum\u0026rdquo; Coverage B at the total 0.0 +6.6 % 0.816 0.3 +4.2 % 0.787 0.6 +2.2 % 0.764 0.9 +0.5 % 0.746 The counter-intuitive fact. The two errors move in opposite directions. The more correlated the series, the less visible the naive error, and the more costly the sophisticated one. That is exactly the regime of energy (the weather) and retail (promotions). In practice, the true covariance is unknown. The solution is a joint bootstrap of the in-sample residuals: we resample time points, and all series are drawn at the same time point. This gets back to 0.902 at the total. A series-by-series bootstrap reproduces B\u0026rsquo;s error (0.760). In HierarchicalForecast, these are the Bootstrap and PERMBU methods.\nResampling time points, not series, is enough to restore calibration. A simple rule follows: same time point, all series. A bootstrap that draws residuals at different dates for different series (histories of unequal length, gaps) is method B in disguise.\n5.3 MinT doesn\u0026rsquo;t know about zero # The third implicit assumption: quantities are continuous and symmetric. On slow-moving product sales (often 0, sometimes a spike), a Gaussian with the right variance spills well below zero.\nA Gaussian fitted to counts puts part of its mass on negative sales. I measured it on one store from the M5 dataset (Walmart, store CA_1: 3,049 products and 11 aggregates, one-day-ahead forecast), comparing Gaussian MinT with BayesReconPy. This package doesn\u0026rsquo;t project: it conditions. It keeps the joint distribution of the products on the non-negative integers and reweights it by what the aggregate forecasts say (Bayes\u0026rsquo; rule).\nGaussian MinT bottom-up MixCond TD-cond products with a negative mean 11 0 0 0 products whose 90 % lower bound is \u0026lt; 0 2,992 (98 %) 0 0 0 product RPS (lower = better) 0.819 0.676 0.674 0.669 aggregate MIS (lower = better) 1,825 2,330 2,312 444 The number that matters depends on the decision. For a decision based on the mean, 0.36 % of the forecasts are negative. For a service level, it\u0026rsquo;s 98 %. The mechanism is easy to read: with a diagonal \\(W\\), each product\u0026rsquo;s adjustment is proportional to its variance. The \u0026ldquo;almost always 0, sometimes a big spike\u0026rdquo; products therefore absorb most of the correction and go below zero.\nThe products that go below zero have a small mean and a huge dispersion. Two more observations:\nBottom-up and MixCond make the aggregate intervals five times worse. Assuming 3,049 independent products gives a total that is far too narrow: it\u0026rsquo;s method B from the previous section, at scale. TD-cond, which starts from the aggregates, avoids that trap and stays the most accurate at both levels. My numbers reproduce the official R vignette of the bayesRecon package on the same store, with a median gap of 0.15 skill-score points. Check a package before recommending it. At the time of testing, BayesReconPy 0.5.0 no longer installed as is: the release of PuLP 4 breaks an import (you need pulp\u0026lt;4). Its licence is contradictory: MIT in the metadata, LGPL-3.0 in the repository. Finally, reconc_td_cond modifies its inputs in place. It\u0026rsquo;s an excellent specialised tool, but you need to know its conditions of use. 6. On real data: twelve regions, one France # The previous experiments are controlled. The next step was to confront MinT with real data. I used the electricity demand of France\u0026rsquo;s 12 mainland regions (éCO2mix, RTE, via ODRÉ, Licence Ouverte v2.0). The business goal: deliver hourly day-ahead forecasts where the sum of the regions exactly matches the national total.\nCheck the data before forecasting # Three findings came before the first forecast:\nNouvelle-Aquitaine stops at the end of 2024, while the other regions continue. Since 2025, the sum of the regions is no longer the national total. The protocol is therefore limited to 2013–2024. The source has a daylight-saving defect, every year and in every region: two duplicates at the end of March, one missing hour at the end of October. A missing row distorts a total without raising any error, which is more dangerous than a NaN. The official total isn\u0026rsquo;t always the sum of the regions. The gap exceeds 200 MW in October 2016. A year-by-year test shows it stays below 6 MW (rounding) from 2014 to 2024, except 2016: over the period actually used, reconciling towards that total makes sense. Every year, two duplicates in March and one missing hour in October. The protocol, fixed before seeing the results # Hourly step: mean of the half-hours, in UTC. MW stay MW, and no day has 23 or 25 hours. Base models: SeasonalNaive(168) as the floor, MSTL with double seasonality (24 h and 168 h) as the reconciled model. Window: rolling 52 weeks, strictly prior to each origin. Methods compared: bottom-up, OLS, WLS struct, WLS var, MinT shrink. Metric: MASE per level (France, then the average over regions), never a global average. The original plan called for 4 origins: four Tuesdays in November 2024. I added a robustness study over 119 origins, one every three days across 2024.\nThe ranking flips # Coherence is guaranteed: the maximum incoherence is exactly 0 MW after reconciliation, versus 672 MW for the base forecasts. But accuracy tells a different story depending on the number of origins:\nMean MASE, 119 origins France Regions SeasonalNaive (floor) 0.947 0.939 bottom-up 0.352 0.427 WLS var 0.356 0.430 WLS struct 0.358 0.432 OLS 0.366 0.439 MinT shrink 0.370 0.439 Over 4 origins, MinT seemed to win at the regional level (−2.2 %), without significance. Over 119 origins, bottom-up is the best method at both levels, and MinT shrink does worse than it on all 13 series, without exception. The four Tuesdays had landed on the favourable side of a very wide distribution. After Holm correction, the evidence is moderate (p ≈ 0.05), but the sign is consistent. Four origins are not enough to rank methods.\nThe cause: a hidden assumption in mint_shrink # MinT assumes that the estimated covariance \\(W\\) resembles that of the true forecast errors. I tested that assumption directly. MSTL\u0026rsquo;s in-sample residuals are smoothing leftovers, correlated at 0.11 on average between regions. The true out-of-sample errors are correlated at 0.56 (bootstrap interval [0.45; 0.65]). MinT is therefore reconciling with the wrong map of the errors.\nLeft: what W sees. Right: what the real errors look like. To isolate the cause, I ran a controlled experiment. Everything stays the same except \\(W\\), which I estimate on the out-of-sample errors of past origins, so still without leakage.\n99 origins Regional MASE vs bottom-up p (Wilcoxon) MinT shrink, in-sample \\(W\\) +2.6 % 0.017 MinT, \\(W\\) estimated on past errors −0.1 % 0.83 Fixing W removes MinT\u0026rsquo;s handicap, without letting it beat bottom-up. Fixing \\(W\\) removes the handicap, but doesn\u0026rsquo;t make MinT win. That\u0026rsquo;s plausible: regional errors are strongly correlated and share the same sign (a cold snap hits everyone), so there is no cross-sectional information to exploit.\nMy recommendation for this case:\nShip bottom-up. It meets the goal (exact coherence), it\u0026rsquo;s the most accurate here, and it can be explained in one sentence: \u0026ldquo;the national figure is the sum of the regions\u0026rdquo;. Don\u0026rsquo;t use mint_shrink with the in-sample residuals of a smoothing model. If MinT is needed, estimate \\(W\\) on rolling backtest errors. Invest in the base models (temperature, public holidays). The largest errors are shared by all regions, and no reconciliation can recover them. 7. In production: where does reconciliation live? # One last question, about architecture: should reconciliation be a batch job, a step inside the API, an SQL view or an estimator updated online? I chose a materialised batch job with a blocking quality gate. The reasons fit in one diagram:\nflowchart TD H[History and covariates] --\u003e B[\"1. Base forecastsone model per series → incoherent ŷ\"] B --\u003e|ŷ + training residuals| R[\"2. ReconciliationP = S(S'W⁻¹S)⁻¹S'W⁻¹\"] S[\"Versioned S + fingerprint\"] --\u003e R R --\u003e Q([\"3. Quality gatecoherence · negatives · MASE per level\"]) Q --\u003e|fail| X[Publication blocked + alert] Q --\u003e|pass| T[(\"forecast_reconciled tableimmutable artefact: run_id, S fingerprint, W version\")] T --\u003e A[\"API /forecast?level=read-only\"] The non-negotiable point: the API never reconciles on the fly. Otherwise two simultaneous calls on two levels can return numbers that don\u0026rsquo;t add up, which is exactly the problem we set out to remove. Coherence is a property of the artefact, not of the request.\nOption For Against Materialised batch coherent by construction, auditable refresh latency ✅ chosen On the fly in the API always up to date coherence not guaranteed across calls ❌ SQL view zero infrastructure MinT in SQL is unmanageable, you end up with bottom-up in disguise ❌ Recursive online estimation \\(W\\) tracks regime changes mutable state to version and replay ⏳ later Writing this contract as executable code, with tests where each rule fails, corrected four things in the paper version:\nFallback to a simpler method must not trigger when \\(m \u003e T\\). That is precisely the case shrinkage exists for (lesson 2). It triggers on the measured condition number of \\(W\\). An absolute coherence tolerance doesn\u0026rsquo;t mean the same thing at 10 MW and at 50 GW. I added a relative tolerance. The \u0026ldquo;no MASE degradation\u0026rdquo; rule applies level by level, never on average. Otherwise a gain at the national level hides a loss at the decision level. The fingerprint of \\(S\\) also covers the labels. A renamed leaf changes the hierarchy without changing the matrix. reconciler: method: mint_shrink fallback: wls_struct # triggered by measured cond(W), not by m \u0026gt; T covariance_window: 104 # weeks, strictly prior to the origin quality_gate: coherence_tol_abs: 1e-6 coherence_tol_rel: 1e-9 # floating-point noise grows with the values max_negative_share: 0.001 # Gaussian MinT on M5: 0.0036 → blocked compare_to_baseline: bottom_up # fail if MASE degrades by more than 2 % at ANY level fail_action: block_publish 8. What I take away # Reconciliation sells coherence, not accuracy. Accuracy gains are frequent, never guaranteed, and they depend on assumptions you can test. Always compare with bottom-up, level by level. Beating incoherent forecasts proves nothing, and a global average hides the level where the decision is made. \\(W\\) is the weak link. Estimated on in-sample residuals, it can describe errors that don\u0026rsquo;t exist. Shrinkage is essential, but it doesn\u0026rsquo;t fix a bad source. Never reconcile quantiles. Reconciling a distribution means projecting joint samples. Independent draws give a 90 % interval that covers 76 %. Counts need different tools. A Gaussian has no business near zero. Conditioning (TD-cond) is coherent, non-negative and more accurate, all at once. Check before you believe: the data (daylight saving, a missing region), the sources (what the abstract actually says), the packages (installation, licence) and the rankings (4 origins versus 119). The comprehension test. \u0026ldquo;You reconcile with MinT. MASE improves at the national level but degrades at the product level. What\u0026rsquo;s going on?\u0026rdquo; Three hypotheses, in order: a biased base forecast (5.1), a poorly estimated \\(W\\) (section 6), a leak or an estimation window that overlaps a regime change. And the real question is a business one: which level carries the decision? If replenishment is decided per product, a national gain paid for at the product level is a regression. What\u0026rsquo;s next. Everything above collapses if \\(W\\) is estimated on contaminated residuals, if the hierarchy changes without notice, or if a leaf has a gap. So week 38\u0026rsquo;s research is about building leak-free temporal features: correct rolling windows, point-in-time data, reference tables that change over time. The common thread: it\u0026rsquo;s not the model that leaks, it\u0026rsquo;s the join.\nReproduce and go further # The code, the 14 notebooks and the 88 tests are in the gwils28/W37_hierarchical_reconciliation repository (Python 3.12, HierarchicalForecast 1.5.1, statsforecast 2.1.1, BayesReconPy 0.5.0, versions locked with uv). Every number in this article comes from a uv run w37 … command. The repository is in French.\nSources\nWickramasuriya, Athanasopoulos, Hyndman, Optimal forecast reconciliation for hierarchical and grouped time series through trace minimization, JASA, 2019. Li, Chen, Taylor, Mao, A Forecast Combination Framework for Hierarchical and Grouped Time Series Reconciliation, arXiv, 2026. Rønlev-Knudsen, Madsen, Møller, Online forecast reconciliation using linear models, arXiv, 2026. Nugteren, Abolghasemi, Mengersen, Drovandi, Hierarchical Bayes meets hierarchical forecasting, arXiv, 2026. Wang, Johnson, Klee, Malloy, Billions-Scale Forecast Reconciliation, arXiv, 2026. Panagiotelis, Gamakumara, Athanasopoulos, Hyndman, Probabilistic forecast reconciliation: properties, evaluation and score optimisation, EJOR 306(2), 2023. Biswas, Zambon, Nespoli, Corani, Nonlinear Probabilistic Forecast Reconciliation, arXiv, 2026. Schäfer, Strimmer, A shrinkage approach to large-scale covariance matrix estimation, SAGMB, 2005. Biswas et al., BayesReconPy, JOSS 10(111), 2025. Living bibliography: awesome-forecast-reconciliation. Data: regional éCO2mix, Open Data Réseaux Énergies (ODRÉ), RTE, Licence Ouverte v2.0; M5 Forecasting. ","date":"6 October 2026","externalUrl":null,"permalink":"/en/posts/reconciliation-hierarchique/","section":"Posts \u0026 notes","summary":"MinT guarantees coherence, not accuracy, and not calibration. Shown through theory, simulation, French grid data and count sales.","title":"Hierarchical forecast reconciliation: promises and limits","type":"posts"},{"content":"","date":"6 October 2026","externalUrl":null,"permalink":"/tags/probabiliste/","section":"Tags","summary":"","title":"Probabiliste","type":"tags"},{"content":"","date":"6 October 2026","externalUrl":null,"permalink":"/en/tags/probabilistic/","section":"Tags","summary":"","title":"Probabilistic","type":"tags"},{"content":" An agent that writes code faster than I do didn\u0026rsquo;t make me a better data scientist. What made the difference is the method I built around it: knowing what to give it, what to ask of it, and above all what to check. Reading time: about 27 minutes. No prior knowledge of Claude Code needed: every feature is explained the first time it appears.\nTransparency note. This work was carried out on a personal basis, for learning purposes, with the help of an AI:\nfor the code and the technical and methodological review; to understand the tools and the mathematical methods; for help with writing, translating and correcting the text; to generate the cover image of the post. The choice of topic, the protocol, the checking of the results and the conclusions are mine, and so are any mistakes. How I use AI\nUpdate, 29 October 2026. A revised version of this method exists: Claude Code for data science: what changes in my working method. This article stays as published; its principles still hold. Claude Code is a coding agent that runs in the terminal. Unlike a chatbot, it doesn\u0026rsquo;t just answer: it reads the project\u0026rsquo;s files, runs commands, edits code, runs the tests and iterates until the work seems done to it. That \u0026ldquo;seems done\u0026rdquo; is exactly what calls for a method.\nI use it every day: for my data science and ML-AI engineering projects, for my research notes, and to build this blog. This post describes how I work with it. It is not an exhaustive feature guide: the official documentation does that better, and it changes every week. It is the method I settled on, with its reasons.\nIn short. My method comes down to five rules:\nContext is the scarce resource: one session per task, and I start over rather than correct three times. CLAUDE.md stays short: it only holds what the agent can\u0026rsquo;t guess. I explore and plan before coding, as soon as a task touches several files. Nothing is done without proof: a test, a build, a screenshot, never an \u0026ldquo;it\u0026rsquo;s done\u0026rdquo;. I stay responsible for every line, and I decide every commit. 1. Why I needed a method # Vibe coding is a way of programming where you describe what you want, accept the code without reading it, feed the errors back to the AI and repeat until it works. For a throwaway prototype, it\u0026rsquo;s very effective. For a forecasting model that will drive real decisions, it\u0026rsquo;s a disaster waiting to happen.\nTwo practitioners drew the distinction I use as a compass. Simon Willison first, in a March 2025 post: if an LLM wrote the code, but you reviewed it, tested it thoroughly and can explain to someone else how it works, that\u0026rsquo;s not vibe coding, it\u0026rsquo;s software development. Then Kent Beck, who talks about augmented coding: in vibe coding you only care about the behaviour of the system; in augmented coding you also care about the code, its complexity, the tests and their coverage.\nMy rule follows from that: I am responsible for the code produced, however it was written. If I can\u0026rsquo;t explain a line, it has no business in a public repository. The rest of the method exists to keep that rule without losing what the agent brings.\nAnd what it brings is real: in a week, a well-framed agent makes it possible to write an implementation from scratch, a full backtest, dozens of tests and teaching notebooks. But part of that work consists in contradicting what was announced, by the agent and by the sources alike, and that is what gives it its value. More on that below.\n2. Context is the scarce resource # This is the principle everything else follows from, and the official documentation says so from the outset: the context window fills up fast, and performance degrades as it fills.\nWhat context is # The context window is everything the model \u0026ldquo;sees\u0026rdquo; at a given moment: my messages, its replies, every file it has read, every command output. A debugging session or a codebase exploration can use tens of thousands of tokens. When the window is full, the agent forgets earlier instructions and makes more mistakes. Before I\u0026rsquo;ve even asked anything, part of it is already taken by the system instructions, the tool definitions and the project\u0026rsquo;s CLAUDE.md.\nThe analogy that works for me, as a data scientist: context is an attention budget. Every file read \u0026ldquo;just to see\u0026rdquo; is an expense. A 2,000-line log pasted in full dilutes the three lines that matter.\nMy reflexes # 1 One task, one session I start every independent task with /clear, which resets the context. Mixing \u0026ldquo;fix this bug\u0026rdquo;, then \u0026ldquo;by the way, what do you think of this chart?\u0026rdquo;, then going back to the bug is what the documentation calls the kitchen sink session: a context full of noise. 2 Two corrections, not three If the agent gets the same thing wrong twice, I don\u0026rsquo;t correct it a third time. The context is already polluted by failed approaches. I start a clean session, with a better first message that includes what I learned. It\u0026rsquo;s counter-intuitive, and almost always faster. 3 Rewind rather than pile up Pressing Esc twice, or /rewind, takes the conversation and the code back to an earlier point. A failed attempt is erased instead of staying in the context as a counter-example. 4 Compact, choosing what stays In a long, useful session, /compact summarises the history to free up space. I tell it what to keep: /compact keep the list of modified files and the test commands. An automatic summary loses details, and not always the ones I would have chosen. Write the state to disk # The corollary is never to let the context be the only place a decision lives. Whatever must survive the session goes into a file: the plan in a PLAN.md, the protocol in a versioned YAML, the conclusions in the README. I commit often. A session then becomes disposable: if it goes off the rails, I close it, and nothing important is lost.\nFor a modelling project, this matters most for the protocol: I write it in a versioned YAML before seeing a single result. Nobody, neither I nor the agent, can tweak it afterwards to flatter a number.\n3. CLAUDE.md: what the agent can\u0026rsquo;t guess # CLAUDE.md is a Markdown file at the root of the project that Claude Code reads at the start of every session. It\u0026rsquo;s where conventions, commands and gotchas go. It is the most powerful lever, and also the easiest to waste.\nShort, or ignored # The trap is to put everything in it. A CLAUDE.md that is too long is partly ignored: the important rules drown in the noise. HumanLayer gives a useful order of magnitude: frontier models follow around 150 to 200 instructions with reasonable consistency, and Claude Code\u0026rsquo;s system prompt already contains about fifty. Their own root file is under sixty lines.\nThe official documentation suggests a test I apply line by line: if I remove this line, will the agent make a mistake? If not, I delete it.\n✅ I put in ❌ I leave out Commands you can\u0026rsquo;t guess Anything readable from the code Architecture choices specific to the project Standard language conventions Gotchas and surprising behaviours Detailed API documentation (a link is enough) How to verify (build, tests) Tutorials and long explanations The why behind important rules The obvious (\u0026ldquo;write clean code\u0026rdquo;) A real example: this blog # This blog has its own CLAUDE.md. It doesn\u0026rsquo;t describe Hugo, which the model already knows. It describes what a developer, human or not, would otherwise learn the hard way:\nBlowfish requires the `config/_default/` layout. There is **no root `hugo.toml`** — adding one would shadow this directory. `themes/blowfish/` is a git submodule — never edit inside it; changes are lost on update. **`public/` is gitignored** and must stay that way — it\u0026#39;s a build artifact produced by CI, not source. Every rule carries its reason. This is not a detail: an explained rule generalises. \u0026ldquo;Never edit the theme\u0026rdquo; says nothing about the case where a template needs overriding; \u0026ldquo;the theme is a submodule, changes are lost on update\u0026rdquo; lets the agent work out that it should copy the template into layouts/.\nTreat it like code # I have the first version generated with /init, which analyses the repository, then I prune it. A generated version shipped without review always contains file descriptions the agent would have found on its own. After that, the file lives with the project: when the agent keeps making a mistake despite a rule, it\u0026rsquo;s often because the file is too long, not because the rule needs shouting. And when a feature changes, I update CLAUDE.md in the same commit.\nMemory: what I only had to say once # Claude Code also keeps a memory across sessions: short notes it writes itself when I give it a lasting instruction, or when it discovers a gotcha. Mine are modest and concrete:\nalways answer me in French, including progress updates; the Hugo installed here is a snap package: its /tmp is private, so a check build written to /tmp \u0026ldquo;succeeds\u0026rdquo; without producing anything visible. You have to build elsewhere before inspecting the HTML; the development server doesn\u0026rsquo;t always re-render a changed page: a real build is the reference, not the preview. The last two came from real mistakes. Each cost me time once; memory means it doesn\u0026rsquo;t cost it twice. I reread these notes now and then, and delete the ones that are no longer true: a wrong memory is worse than none.\n4. The loop: explore, plan, code, verify, commit # The documentation recommends four phases. I add a fifth, explicit one, because it is the one that makes the difference.\n1 Explore In plan mode (Shift+Tab), the agent reads and answers, but changes nothing. I ask it to understand before acting: \u0026ldquo;read src/backtest/ and explain how the test windows are split\u0026rdquo;. I often discover at this stage that my request rested on a wrong assumption. 2 Plan I have it write a detailed plan: files touched, order of steps, how to verify. I annotate it directly (Ctrl+G opens it in the editor) before any code is written. Fixing a plan costs a sentence; fixing an implementation that went the wrong way costs a session. 3 Code The agent implements, following the plan. I iterate on the diff, not on the description: \u0026ldquo;line 42, why a mean and not a sum?\u0026rdquo; works better than \u0026ldquo;the result looks odd\u0026rdquo;. 4 Verify Tests, build, screenshot. That\u0026rsquo;s the next section, and the most important one. 5 Commit The agent proposes; I decide. I read the full diff before every commit, and a commit never mixes two topics. On this blog, when a CSS change touches both the menu and the Projects page, that\u0026rsquo;s two commits. When I skip the plan # Plan mode has a cost. The documentation gives a simple criterion I adopted: if I can describe the diff in one sentence, I skip the plan. Renaming a variable, fixing a typo, adding a log line: I just ask. As soon as the task touches several files, I don\u0026rsquo;t know the code, or I\u0026rsquo;m not yet sure how I would go about it, I plan.\n5. Verify: the one non-negotiable step # The agent stops when the work looks done. Without a way to check, \u0026ldquo;looking done\u0026rdquo; is the only signal available, and I become the verification loop. The documentation puts it bluntly: if you can\u0026rsquo;t verify it, don\u0026rsquo;t ship it.\nGive the agent a check it can run # My request always includes the means to prove it has been met: a test, a build, a script that compares an output with a reference. The agent does the work, runs the check, reads the result and iterates until it passes. And I ask it to show the evidence rather than claim success: the test output, the command it ran and what it returned. Reviewing evidence is faster than redoing the check.\nTest first, explicitly enforced # By default, the agent writes the implementation, then tests that confirm it. Tests written after the code tend to test what the code does, not what it should do. For a function that matters, I enforce the order:\n\u0026ldquo;write a test that fails, implement nothing\u0026rdquo;; I check that it fails for the right reason; \u0026ldquo;implement the minimum to make it pass\u0026rdquo;; I review, then we refactor while keeping the test green. For a metric, that means tests on the awkward cases: a series that is constant over the training period (zero denominator), a perfect forecast, missing values. For a transformation, a property to preserve: forecasts that must add up, an output that must stay positive.\nFresh eyes for review # An agent that has just written code is badly placed to criticise it: its context is full of the reasoning that produced it. For a serious review, I use a fresh context that only sees the diff and the criteria, and I ask it to flag only what affects correctness. Otherwise, a reviewer asked to find flaws always finds some. Section 8 shows how to organise this with a dedicated subagent.\nNever trust a single signal # The most telling example is recent, and it comes from this blog. When I added a callout to a post, the development server\u0026rsquo;s preview didn\u0026rsquo;t show it. The log said \u0026ldquo;source changed\u0026rdquo;, but the page served was the old one. A production build, on the other hand, did contain the callout. The first reflex would have been to \u0026ldquo;fix\u0026rdquo; a problem that didn\u0026rsquo;t exist.\nSince then, the rule is in memory: the build is the reference, the preview is for looking. And for the interface, I ask the agent to check for itself in a real browser, using Claude Code\u0026rsquo;s Chrome extension: it opens the page, clicks the filter, reads the DOM state and takes a screenshot. That\u0026rsquo;s how it spotted near-white bold text on a yellow background, unreadable in dark mode, which neither the build nor the tests would have flagged.\nAlso check what I\u0026rsquo;m told # Verification isn\u0026rsquo;t only about code. A plan, an answer from the agent or a paper contains precise claims: where a gap comes from, how far a dataset goes, how many test windows are enough to rank two models. Each is a hypothesis, not a fact. I check them one by one, and a false claim is often the most interesting finding of the project.\nThat\u0026rsquo;s why my project READMEs have a section \u0026ldquo;what I correct relative to the plan\u0026rdquo;. It keeps track of what was announced, what was measured, and the gap between the two.\n6. What changes in data science and ML engineering # Most advice about Claude Code comes from software development. There, code that passes its tests does roughly what you expect. In data science, that isn\u0026rsquo;t true: code can be right and the result wrong. The pipeline runs, the tests pass, the metric is excellent… because information from the future slipped into training.\nSo I verify at three levels, and the agent needs to know all three:\nLevel The question How I check it The code Does it do what it claims? Unit tests, edge cases, comparison with a reference The data Is it what I think it is? Quality checks run on every load Statistical validity Does the result mean what I conclude? Protocol fixed in advance, baselines, variance across windows, the statistician\u0026rsquo;s review An agent is naturally good at the first level, decent at the second if asked, and weak at the third. That\u0026rsquo;s where my attention goes.\nData before models # An agent handed a dataset wants to model. I first ask it to check: gaps, duplicates, units, time bounds, consistency of aggregates. Hourly data, for instance, often bears the mark of daylight saving time: duplicates in spring, a missing hour in autumn. A missing row raises no error; it silently skews a total. That\u0026rsquo;s the kind of defect a model swallows without complaint.\nMy typical request doesn\u0026rsquo;t say \u0026ldquo;do an EDA\u0026rdquo;. It says: \u0026ldquo;before any modelling, list what could make this data misleading, and write an executable check for each point.\u0026rdquo; The checks become tests, run on every load, not a notebook cell you forget to rerun:\ndef test_no_gap_in_hourly_grid(load): expected = pd.date_range(load.ts.min(), load.ts.max(), freq=\u0026#34;h\u0026#34;, tz=\u0026#34;UTC\u0026#34;) missing = expected.difference(load.ts) assert missing.empty, f\u0026#34;{len(missing)} missing hours, e.g. {missing[:3].tolist()}\u0026#34; def test_total_equals_sum_of_parts(load, total): gap = (load.groupby(\u0026#34;ts\u0026#34;).value.sum() - total.set_index(\u0026#34;ts\u0026#34;).value).abs() assert gap.max() \u0026lt; 1.0, f\u0026#34;max gap {gap.max():.1f} at {gap.idxmax()}\u0026#34; The error message matters as much as the assertion: when the test fails, the agent must be able to read where and how much, not just that.\nWhat I know about the business goes into a skill, a data dictionary: what each column means, its unit, its known gotchas (\u0026ldquo;zero sales during a stock-out are not zero demand\u0026rdquo;). The agent loads it when it touches the data, and I no longer have to repeat it.\nLeakage, the number one trap # Leakage is information available at training time that won\u0026rsquo;t be available when forecasting. It makes the model excellent in validation and mediocre in production. An agent falls into it as easily as a rushed human, and for the same reason: the shortest code is often the code that leaks.\nLeak The leaking code What I ask for Temporal train_test_split(X, y, shuffle=True) on a time series A rolling-origin split Preprocessing StandardScaler().fit(X) before splitting Preprocessing inside a Pipeline, fitted on the training set only Imputation df.fillna(df.mean()) over the whole history A mean computed on the past of each origin Target A variable computed after the event (end-of-week stock) The availability date of every feature, documented Group The same customer in train and test A group-based split The rule is written in the protocol and in CLAUDE.md, with its reason: everything that is estimated must be estimated strictly before each origin. And it has its test, which I have written first:\ndef test_no_feature_reads_the_future(build_features, history): origin = pd.Timestamp(\u0026#34;2025-06-01\u0026#34;) before = build_features(history, origin) # Tamper with everything after the origin: the features must not move. tampered = history.assign( value=history.value.where(history.ts \u0026lt; origin, -999.0)) pd.testing.assert_frame_equal(before, build_features(tampered, origin)) This test doesn\u0026rsquo;t check a particular implementation; it checks a property. It stays valid when the agent rewrites the features, which is exactly what you need to keep a fast-writing collaborator in check.\nA baseline, always, and first # An agent readily suggests the most sophisticated model it knows. I impose the opposite order: first a naive baseline, then the model. For a weekly series, that\u0026rsquo;s \u0026ldquo;the same week last year\u0026rdquo;; for a classification, the majority class and a logistic regression. If the model doesn\u0026rsquo;t clearly beat the baseline, nothing else matters, and I\u0026rsquo;ve learned that before spending a day on hyperparameters.\nThe agent\u0026rsquo;s suggestions for those hyperparameters, incidentally, are search ranges, not recommendations. The final choice is justified by validation, not by how confident the answer sounds.\nEvaluate the way you\u0026rsquo;ll decide # The agent spontaneously reports one number: \u0026ldquo;MAPE of 8.3% on the test set\u0026rdquo;. That\u0026rsquo;s almost always not enough. I ask for three things.\nThe right metric, at the right level. The one that matches the decision: an error per product if supply is decided per product, a quantile loss if the decision is about a stock level. Never a global average that mixes everything. The spread, not just the mean. A metric over several test windows, with its standard deviation. A model that is better on average but only on two windows out of five is not better. A paired test when I compare two models on the same windows, with a correction if I make many comparisons. Without it, I\u0026rsquo;m very likely to \u0026ldquo;discover\u0026rdquo; a gain that is only noise. This is typically when I bring in the statistician subagent: before looking at the results, to fix the protocol; afterwards, to tell me what I\u0026rsquo;m not allowed to conclude.\nPredicting is not explaining # A subtler trap awaits reports. An agent asked to interpret a model readily writes \u0026ldquo;temperature increases demand by 12%\u0026rdquo; from a feature importance or SHAP values. These describe the model, not the world. They say what the model relies on to forecast, not what would happen if you intervened.\nMy instruction is explicit: no causal vocabulary in a report without an identification strategy. Measuring the effect of a promotion requires a comparison group, an experiment or a quasi-experimental method, and that\u0026rsquo;s a different question from forecasting. An agent knows this perfectly well when asked; it forgets when it isn\u0026rsquo;t.\nNumbers come from code, never from the conversation # It\u0026rsquo;s one of my publishing rules: a published number is the output of code that was run and versioned. Never a value a language model \u0026ldquo;computed\u0026rdquo; in its head in a reply, nor a percentage copied from a summary. In practice, every results table is produced by a command (make report) that anyone can rerun, with:\nlibrary versions pinned in uv.lock; random seeds fixed and recorded; the fingerprint (sha256) of the raw data in a manifest, to know what the number was computed on. Code in a package, notebooks to explain # An agent readily produces two-hundred-cell notebooks with duplicated logic. I separate them: the code lives in a single, tested Python package; the notebooks call it and exist to explain. An estimation function is written once, and every notebook that needs it imports it. Notebooks are run and versioned with their outputs, which lets the agent read them, figures included.\nFrom maths to code, then to the reference # Translating an equation from a paper into code is one of the tasks where the agent saves the most time. It\u0026rsquo;s also one of the most dangerous: a forgotten transpose gives a plausible, wrong result. My rule: every \u0026ldquo;from scratch\u0026rdquo; implementation is compared with a reference implementation, with a gap that is measured and explained. If the gap is exactly zero, I suspect a copy; if it\u0026rsquo;s large, I look for the bug; if it\u0026rsquo;s small, I must be able to say where it comes from. When there\u0026rsquo;s no reference, I test the mathematical properties: a projection applied twice changes nothing, a covariance matrix is symmetric and positive.\nFrom model to system # On the ML engineering side, the difficulty shifts: the model works, and it has to keep working. Three points come up in every project.\nThe same code in training and in production. A feature computed in pandas in the notebook and rewritten in SQL for production always ends up diverging. I ask the agent for a single, tested implementation, called from both sides. Versions of everything. Data, code, model and parameters are versioned together, and every run is tracked in MLflow. A number in production must be replayable. Monitoring. Input drift, error drift, share of outlier forecasts. The agent writes the checks; I set the thresholds that trigger an alert. For LLM-based applications, the logic is the same, with one more difficulty: the output is not deterministic. So I build a versioned evaluation set, with the hard cases, before touching the prompt; I version prompts like code; and when an LLM acts as a judge, I first check on a hand-labelled sample that it judges the way I do.\n7. Extending the tool without weighing it down # Claude Code can be extended in several ways. The risk is stacking extensions that each consume context permanently. My rule: an extension must answer a recurring need. If I do something more than once a day, it deserves automation; otherwise, a sentence in the request is enough.\nExtension What it is When I use it CLAUDE.md Rules read at every session Permanent project conventions Skill Procedure loaded on demand (SKILL.md) Occasional know-how: a backtest protocol, a data dictionary Hook Script run automatically at a specific point What must happen every time, without exception Subagent Isolated instance, with its own context Research or review, to keep the noise out of my session MCP Connector to an external tool Structured access to a database or a model registry Hooks, because CLAUDE.md is advisory # The difference is fundamental: a CLAUDE.md instruction is advice, which the agent follows… most of the time. A hook is deterministic: a script Claude Code runs itself at a precise point, for example after every file edit, or before handing back control. Forbidding writes to data/raw/, running the formatter after every edit, refusing to finish while tests fail: these are rules that allow no exception, so they are hooks, not sentences.\nCLIs before connectors # To talk to external services, I favour the command-line tools the agent already knows how to use: gh for GitHub, gcloud for GCP. It\u0026rsquo;s the most context-efficient way. An MCP connector is worth it when structured access really adds something, for instance querying an MLflow registry. For a database, access is read-only: every query is a real job, potentially billed, launched by an agent that doesn\u0026rsquo;t have an analyst\u0026rsquo;s caution.\nUnattended: non-interactive mode # With claude -p \u0026quot;…\u0026quot;, Claude Code runs without an interface, for example in CI to review a pull request. The precautions then scale up: allowed tools reduced to the strict minimum (--allowedTools), a dedicated key with a spending cap, and never broad permissions outside an isolated container.\n8. A complete example: forecasting demand # To make all this concrete, here is what a complete setup looks like on a typical case: forecasting weekly demand for 2,000 products, with a pipeline deployed on GCP. It\u0026rsquo;s an example built for this post. It brings together the pieces described above and shows how they fit.\nThe main session receives the rules, the memory and my decisions; it loads skills, triggers hooks, delegates to five subagents and calls external tools. The result goes through verification, review, my commit, then CI. Click to enlarge. What surrounds the session # Coming in, three sources: my decisions (the question, the protocol), the project\u0026rsquo;s CLAUDE.md (conventions, commands, the no-temporal-leakage rule) and the memory (gotchas already met). On the left, skills, loaded only when the task calls for them: the team\u0026rsquo;s backtest protocol, the data dictionary (what each column means, its units, its gotchas), and the model report template. On the right, hooks, which run without exception: no writes to data/raw/, formatting and targeted tests after every edit, no handing back control while a test fails. Below, the tools: BigQuery read-only, MLflow for runs and the model registry, the gh, gcloud and terraform CLIs, and Chrome to look at a dashboard. Five subagents, five roles # A subagent is a Markdown file in .claude/agents/: a name, a description saying when to call it, the list of allowed tools, the model to use, and instructions. Each works in its own context and only sends a summary back to the main session.\nSubagent Its role Its tools data-explorer Explore and check the data: gaps, duplicates, units, series breaks Read-only, BigQuery; a fast model is enough statistician Discuss methods, biases, causality and the protocol, and object Read-only; it doesn\u0026rsquo;t code ml-engineer Build features and models, writing the failing test first Edit, pytest, MLflow mlops-expert Design the pipeline, deployment, monitoring and infrastructure gcloud, terraform plan; never apply reviewer Review the diff in a fresh context and flag what affects correctness Read-only, the diff and the criteria Tools are the real safeguard. The statistician isn\u0026rsquo;t allowed to edit: its job is to argue, not to implement. The MLOps expert can plan an infrastructure change, never apply it: terraform apply is for me.\nThe statistician, in detail # This is the subagent that adds the most, because it plays a role a coding agent doesn\u0026rsquo;t play spontaneously: the colleague who asks \u0026ldquo;are you sure?\u0026rdquo;. Its definition fits in a few lines:\n--- name: statistician description: Senior statistician. Consult before fixing a protocol, choosing a metric or interpreting a result. Discusses methods, biases and causality; never modifies a file. tools: Read, Grep, Glob model: opus --- You are a senior statistician, specialised in time series and causal inference. Your role is to object, not to approve. For each proposal: 1. Name the implicit assumptions (stationarity, independence, no temporal leakage, data missing at random…). 2. Look for biases: selection, survivorship, censoring, leakage, multiple comparisons. 3. Separate what is predictive from what is causal. A correlation does not justify an intervention decision without an identification strategy. 4. Propose the test or experiment that would settle the question. Reply with a list of objections ranked by severity, then the questions to ask the business. If the proposal is sound, say so in one sentence. In our case, here is the kind of objection it raises in the first round. Observed sales are not demand: when a product is out of stock, sales are censored, and a model trained on them learns to under-forecast precisely the products that run out most. It also asks whether the effect of promotions should be forecast or measured: the first is a forecasting problem; the second is a causal question, which calls for a comparison group, not a better model.\nOne turn of the loop # 1 Explore The data explorer queries BigQuery read-only and reports: 4% of weeks with zero sales, some of which coincide with stock-outs; product codes changing partway through the history. 2 Challenge The statistician reads this report and the plan. It raises demand censoring, possible leakage through a stock variable computed after the fact, and proposes rolling-window validation instead of a random split. 3 Decide Me. I fix the protocol in a versioned YAML: metric, windows, baselines, handling of stock-outs. The backtest-protocol skill provides the structure; the decisions are mine. 4 Build The ML engineer first writes the failing test (\u0026ldquo;no feature reads a date after the origin\u0026rdquo;), then the implementation. After every edit, the hook runs the formatter and the relevant tests; runs go to MLflow. 5 Industrialise The MLOps expert designs the scheduled training pipeline, drift monitoring and the Terraform module. It produces a terraform plan that I read before applying it myself. 6 Review and ship The reviewer receives the diff and the criteria, without the reasoning that produced it. The Stop hook refuses to hand back control while a test fails. I read the diff, I commit, and CI runs an automatic review of the pull request with claude -p. Why split it this way # Three reasons. Context, first: data exploration and review read a lot of files; in subagents, that noise doesn\u0026rsquo;t fill the main session. Disagreement, next: a statistician who objects and an engineer who implements, in two separate contexts, recreate the useful tension of a real team. A single agent doing both tends to validate its own choices. Least privilege, finally: each role only has the tools it needs.\nThis split has a cost: each subagent consumes its own tokens, and coordinating five roles takes more attention than a single session. I wouldn\u0026rsquo;t recommend it from day one. Start with one session and a CLAUDE.md; add a subagent when a role keeps coming back, usually starting with the reviewer or the statistician.\n9. What I don\u0026rsquo;t delegate # The agent does a lot. Some things stay mine, on principle.\nChoosing the topic and the question. Knowing which question is worth asking is the job. The protocol. Metrics, splits, baselines: fixed before the results, in writing. Interpretation. A metric that improves by 2% over four test windows is not a result; it\u0026rsquo;s a hypothesis to test. The agent produces the numbers; their meaning is my responsibility. Irreversible operations. Deletions, history rewrites, publishing: the agent prepares them, I trigger them. Irreplaceable files are committed or backed up before it gets access. Publishing. Nothing goes out on this blog unless I\u0026rsquo;ve read it and can explain it. These rules are public, on the How I use AI page. Writing them down makes me keep them.\n10. The signals that make me change course # Over time, I\u0026rsquo;ve learned to recognise a few signals. Each calls for a specific move.\nSignal What I do The agent ignores a CLAUDE.md rule I prune the file, instead of adding capital letters Two failed corrections on the same point /clear, and a better first message The agent reads dozens of files \u0026ldquo;to understand\u0026rdquo; I narrow the question, or delegate to a subagent The result is \u0026ldquo;done\u0026rdquo; without evidence I ask for the test output or the screenshot A gap of exactly zero with a reference I look for the copy A surprising number, in the good direction I look for the leak before celebrating I make the same request for the third time It becomes a rule, a skill or a hook Conclusion # Claude Code didn\u0026rsquo;t replace the way I work: it made it more demanding. When writing code costs almost nothing, value shifts to what still costs: asking the right question, fixing the protocol before looking, and proving the result is right.\nIf I could keep only one rule, it would be this one: nothing is done without proof. The rest, managing context, a short CLAUDE.md, planning before coding, exists to produce proof faster.\nTools change every week; I evolve this method with them. What doesn\u0026rsquo;t change is that, in the end, the code bears my name.\nFurther reading\nBest practices for Claude Code, the official documentation, continuously updated. Writing a good CLAUDE.md, HumanLayer: the instruction budget and the what / why / how structure. Not all AI-assisted programming is vibe coding, Simon Willison, March 2025. Augmented Coding: Beyond the Vibes, Kent Beck, June 2025. ","date":"29 September 2026","externalUrl":null,"permalink":"/en/posts/claude-code-data-science/","section":"Posts \u0026 notes","summary":"A fast-coding agent doesn’t replace method. How I work with Claude Code: context as the scarce resource, verification as the one non-negotiable step.","title":"Claude Code for data science and ML-AI Engineering: my working method","type":"posts"},{"content":"A medley of lush tropical green and the volcanic black of the sand. Fourteen drumbeats, one for every year away. A return to the land of my birth, only to leave again armed with its blessing.\n","date":"20 January 2026","externalUrl":null,"permalink":"/en/migration/martinique-anse-ceron-2026/","section":"Migration","summary":"","title":"Habitation \u0026 Anse Céron Beach","type":"migration"},{"content":"I never really liked Guinness. And that is one of the notable mistakes of my life. My moral debt for this wrong was paid at the source.\n","date":"22 May 2024","externalUrl":null,"permalink":"/en/migration/guinness-2024/","section":"Migration","summary":"","title":"Guinness city","type":"migration"},{"content":"Along the sacred trails of Mount Kurama: a timeless climb, guided by the wisdom of thousand-year-old cedars, the endurance of a friendship and the invisible imprint of reiki.\n","date":"12 April 2019","externalUrl":null,"permalink":"/en/migration/japon-kuramadera-2019/","section":"Migration","summary":"","title":"Kurama dera","type":"migration"},{"content":"I\u0026rsquo;m a data scientist with a passion for jazz and a strong taste for statistics. I specialise in time series and causal inference, with solid experience in LLMs and AI agent orchestration. I\u0026rsquo;m curious about every field where my craft can be applied.\nWhat drives me is building models and systems that help us better understand how things work, and that make better decisions easier to reach. Always up for a good statistical puzzle and attentive to detail, since 2014 I have been developing autonomous systems that predict and simulate complex phenomena with as little bias as possible.\nIf I were… an animal I\u0026#39;d be A duck🦆 why? because On land, on water or in the air, it is at ease everywhere without depending on any one terrain. It inspires me with its adaptability and great resilience, despite its sometimes nonchalant air.🦆 a tune I\u0026#39;d be So WhatMiles Davis🎺 why? because Paul Chambers’ bass asks a question, the horns answer “so what”. All of Kind of Blue lies in that exchange: space, few notes and a deep freedom.🎺 a drink I\u0026#39;d be Ti\u0026#39;punch why? because In my tradition, everyone measures out their own death, adjusting the 3 ingredients as they see fit. That is the best symbol of tolerance and openness I know. an element I\u0026#39;d be Air🌬️ why? because Without it, we would suffocate within moments; yet we more readily think of water as essential. It is the quiet, unassuming strength.🌬️ a dish I\u0026#39;d be A pizza🍕 why? because There are as many possible recipes as imagination allows, and everyone has a clear idea of what makes a good pizza. I see myself in that diversity.🍕 a book I\u0026#39;d be EssaysMichel de Montaigne📖 why? because Doubting oneself (“what do I know?”), distrusting one’s certainties (“everyone calls barbarity what is not his own custom”) and recognising oneself in everyone (“each man bears the entire form of the human condition”).📖 a tree I\u0026#39;d be A coconut palm🌴 why? because Slender and perched up high, the coconut palm overlooks the landscape without ever casting a stifling shade.🌴 a chess piece I\u0026#39;d be A bishop why? because It never moves in a straight line and has no brute force; it bides its time when the game is closed, but it is formidable in open positions. a time of day I\u0026#39;d be Dawn🌅 why? because It is a moment of absolute serenity, yet one that carries all the energy of the day ahead.🌅 ","externalUrl":null,"permalink":"/en/about/","section":"Home","summary":"","title":"About","type":"page"},{"content":"I write down what I learn here: research notes, paper readings, and the projects I build to understand things by making them.\n","externalUrl":null,"permalink":"/en/","section":"Home","summary":"","title":"Home","type":"page"},{"content":"I note down here what I listen to, what I learn, and what I explore: discovered albums and musicians, progress (and struggles) with the instrument, and detours through history and theory.\n","externalUrl":null,"permalink":"/en/jazz/","section":"Jazz","summary":"","title":"Jazz","type":"jazz"},{"content":"And, just like ducks… I sometimes migrate around the globe.\n","externalUrl":null,"permalink":"/en/migration/","section":"Migration","summary":"","title":"Migration","type":"migration"},{"content":"","externalUrl":null,"permalink":"/en/jazz/pantheon/","section":"Jazz","summary":"","title":"Pantheon","type":"jazz"},{"content":"My technical notes: what I read, what I try, what works and what doesn\u0026rsquo;t.\n","externalUrl":null,"permalink":"/en/posts/","section":"Posts \u0026 notes","summary":"","title":"Posts \u0026 notes","type":"posts"},{"content":"Projects I build outside of work, to learn something or to answer a question that was nagging me. Source code available for each.\n","externalUrl":null,"permalink":"/en/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"Experienced Data Scientist and ML Engineer with a strong background in statistics. This page traces my path: employment, freelance and volunteer work, education and certifications.\nExperience # Employed5 Freelance1 Volunteer1 Employed Dec 2023 – present3 yrs Data Scientist \u0026amp; ML Engineer Orange · Rennes, FR Current I design and productionise the forecasting models that help Orange anticipate its technicians\u0026#39; workload across the whole French telecom network, and I explore AI agents to automate back-office work.\nNationwide forecasting: time-series models that anticipate technician activity across every network domain, to size teams ahead of demand. Generative AI (R\u0026amp;D): feasibility study of AI-agent and LLM architectures to automate business processes and speed up decision-making. MLOps on GCP: automated ingestion, preprocessing and training pipelines, with versioning, performance monitoring and continuous deployment. Data Science lead: technical reference for the team, mentoring interns and apprentices. PythonTime seriesLLM \u0026amp; agentsGCPAirflowCI/CDGitLabDockerLinux Jan 2021 – Nov 20232 yrs 11 mo Data Scientist \u0026amp; MLOps BPCE-SI · Aix-en-Provence, FR At the heart of BPCE group\u0026#39;s IT, I took marketing use cases from prototype to production, and helped build the tooling that standardises that path for the whole group.\nCustomer insight: advanced segmentation, churn prediction, sentiment analysis of customer feedback and a recommendation engine. Life events: models predicting key moments of the customer journey (retirement, first job, moving house), deployed to production to offer the right advice at the right time. Generative AI: implemented evaluation methods for a RAG-based conversational agent, to measure how relevant and reliable its answers are. In-house MLOps library: contributed to the Python tool that automates and standardises how the group packages and deploys its machine-learning models. Industrialisation and cloud: CI/CD, production monitoring and migration of use cases to GCP (BigQuery, Vertex AI), hand in hand with Data Engineers and DataOps. Agile delivery: Scrum projects with regular demos to stakeholders. PythonPySparkTime seriesLLMBigQueryVertex AIAirflowMLOpsCI/CDNLPScrumLinux Jul 2019 – Dec 20201 yr 6 mo Data Scientist \u0026amp; IoT Analytics Traxens · Marseille, FR At this smart-container specialist, I turned IoT sensor data into decision tools for shipping logistics: maintenance, cold chain and security.\nPredictive maintenance: estimated the remaining battery life of tracking devices, to plan interventions and manage the fleet's life cycle. Cold chain: temperature forecasting for refrigerated containers, at the core of the alerting systems. Security: door-opening detection algorithms built on sensor signals. Computer vision: processed and classified satellite imagery to annotate areas relevant to maritime transport. Data quality: governance, monitoring and hardening of the analytics and modelling pipelines. Research collaboration: worked with a research lab on automated cleaning of sensor data using reinforcement learning. Data Science and statistics lead: methodological reference for the team on modelling and data analysis. PythonIoTTime seriesComputer visionPredictive maintenanceReinforcement learningLinux Oct 2017 – Jun 20191 yr 9 mo Data Scientist IT-CE · Marseille, FR For the Caisse d\u0026#39;Épargne banks\u0026#39; IT, I put machine learning to work for lending, against fraud, for customer retention and for automated document processing.\nFraud prevention: unsupervised anomaly-detection model to flag fraudulent cheques and better protect customers. Credit scoring: built a scoring model to set the interest rate of the loans offered to customers. Life-insurance beneficiaries: automatically identified beneficiaries from heterogeneous documents (PDFs, scanned handwritten letters, free-text notes), combining named-entity recognition (NER) and computer vision. Retention: churn models for young customers, professionals and businesses, the basis of targeted loyalty strategies. Data lake: ingested and structured data with PySpark to centralise analysis and model deployment. Decision support: interactive dashboards turning model outputs into action plans for business teams. PythonPySparkSQLNERComputer visionCredit scoringAnomaly detectionChurnData LakeLinux Sep 2014 – Sep 20173 yrs 1 mo Biostatistician INSERM · Paris, FR In medical research, I applied statistics to genomics and epidemiology: that is where my methodological rigour was forged.\nGenomics: inferential analyses and clustering of genomic profiles to reveal biologically meaningful patient subgroups. High-dimensional data: selected candidate genes among thousands of variables with penalised regression (lasso) and false discovery rate (FDR) control. Oncology: modelled the clinical impact of biomarkers on leukaemia progression with multivariate methods. Survival analysis: Cox models and Kaplan-Meier curves to assess the prognostic value of these biomarkers on progression-free survival. Epidemiology: dengue spread model combining biological and environmental factors to simulate outbreak dynamics. Causal inference: estimated the effect of vector-control actions on dengue incidence, making confounders explicit with causal graphs (DAGs) and adjusting for them with propensity-score weighting. RMultivariate statisticsCausal inferenceSurvival analysisGenomicsEpidemiologyModellingLinux Freelance Aug 2021 – present5 yrs 4 mo Mentor \u0026amp; Instructor OpenClassrooms · Remote Ongoing I mentor and train learners on the Data Scientist Machine Learning, Data Engineer and AI Engineer paths, project by project, from scoping to deployment.\n18 learners mentored and trained throughout their path, one to one. Machine learning and MLOps: text and image classification, serving models behind an API, experiment tracking and CI/CD pipelines. Generative AI: RAG systems, LLM evaluation, AI agents and fine-tuning. Data engineering: SQL and NoSQL databases, ingestion and orchestration pipelines, cloud infrastructure. PythonSQLscikit-learnMLflowFastAPIStreamlitLangChainPySparkKafkaKestraCI/CDTeaching Volunteer Feb 2019 – Jun 20195 mo Data Scientist Data For Good With this volunteer collective, I worked on the Open Food Facts project, the open database of food products: statistical analysis, Nutri-Score prediction and data quality.\nStatistical analysis and report on the data of the Open Food Facts project. Nutri-Score prediction: a model that estimates it for products that don't have one yet. Data quality: an audit of the database, then improvements to its quality. PythonpandasNumPyscikit-learnstatsmodelsFastAPIData quality Education # Organization Degree Aix-Marseille Université Marseille, FR Master\u0026#39;s in Biostatistics Université des Antilles Martinique, FR Bachelor\u0026#39;s in Mathematics, Computer Science and Biology Certifications # A certification doesn\u0026rsquo;t replace production experience, but it strengthens its foundations. It\u0026rsquo;s also my way of keeping on learning, and of checking that what I know holds up under a demanding exam.\nProfessional Machine Learning Engineer Google Cloud2025 – 2027 2025 – 2027 Skills # Category Skills / technologies Languages Python, R, SQL, Bash, Rust, TypeScript Time series forecasting, pricing, pattern mining, anomaly detection Statistics causal inference, A/B testing, Bayesian inference, survival analysis, multivariate statistics, penalised regression Machine learning gradient boosting, deep learning, clustering, recommender systems, computer vision, reinforcement learning Generative AI LLM, AI agents, RAG Frameworks \u0026amp; tools scikit-learn, PyTorch, statsmodels, LangGraph, MLflow, dbt, Airflow, PySpark, BigQuery Cloud \u0026amp; MLOps GCP, Vertex AI, Docker, Terraform, Git, CI/CD, Linux ","externalUrl":null,"permalink":"/en/cv/","section":"Home","summary":"","title":"Resume","type":"page"},{"content":"","externalUrl":null,"permalink":"/en/series/","section":"Series","summary":"","title":"Series","type":"series"}]