VIGIE-01

Summary

Cut the daily synthesis time without giving up traceability

• The observation. The hard part is not collecting — the tools exist — but classifying a heterogeneous stream consistently, cross-checking it, and being able to justify after the fact why a piece of information was kept.
• The answer. A five-node pipeline where every statement is backed by a quote verified verbatim, and where bounded agentic escalation is triggered by a measured condition — the existence of an antecedent to cross-check against — and not by the article's category.
• The state. V1 working end to end, the first slice of the V2 verifier and the first stage of V3 longitudinal reasoning (event threads) shipped, guardrails coded and tested both ways — not merely declared. In production since 2026-09-05: the digest is served publicly, production persistence has run, and the budget reservation in a transaction — the guardrail that could only be verified under real concurrent conditions — holds under 30 simultaneous reservations for 4 slots. The daily scheduler was armed on 2026-09-06; what remains is observing one unattended cycle.
• The recommendation. Make the pipeline observable before making it autonomous. The deployment diagnosis did not stumble on the infrastructure but on the fact that the pipeline logged nothing: under a scheduler, all the cost instrumentation built over the previous days would have vanished at the exact moment it becomes the only window on the system.

What follows details the scoping, the architecture and the evidence — in that order.

The problem

Manual export-risk monitoring cannot be justified after the fact

• A scattered daily press review, unstructured email alerts
• No trace of why a piece of information was kept or discarded
• Audience: export compliance, competitive intelligence, public affairs functions

An explicit non-goal: the system does not decide. It speeds up and structures the human watch — the decision stays human.

Scoping

Three critical issues, not one

1. Consistent classification of a heterogeneous stream of sources
2. Multi-source cross-checking — telling a confirmed signal from an isolated rumour
3. Systematic traceability — every statement backed by an identified source

1 and 3 covered in V1 · 2 delivered as the first V2 slice — perimeter extended to the five categories on 2026-08-20, escalation being conditioned on a measured candidate antecedent rather than on the category (see roadmap)

MECE perimeter

Scoped as MECE, then corrected on contact with real data

An accepted revision: strict geographic filtering, tested on real data, wrongly rejected nearly all of 2 sources out of 5 → removed, replaced by a location extraction kept as non-filtering metadata. Documented as a revision, not hidden.

Architecture

A traceable agentic pipeline, not a script

Shared state (VigieState) between the nodes, every node traced natively — with no manual instrumentation. The first three nodes are a deterministic workflow; the last two let the model decide on its own tool calls, each under a coded double cap — one more agentic node is not one more cap to forget.

What survives runs — LLM budget, seen links, analysed history — goes through a single persistence layer, local files in development and Firestore in production. The digest served is a sliding window over that history, not a snapshot of the last run: since deduplication empties the later collections, serving the raw run would erase the display on every pass.

Display

The digest makes visible what commits confidence

Capture of 2026-09-06, during the transition to English: the escalation gates found nothing that week, so every card shows the unscored state rather than a score — which is what the design is for. The confidence score sits on a sequential ramp and not in red/green — a score read as a verdict would invalidate the non-goal. An item outside the verifier's perimeter comes out with no score rather than with a zero. The "state media" provenance is displayed, not buried.

Coverage

The map also displays what it cannot place

Built on the verified location of the event, not on the country of the source. Items with no extractable place and places not attachable to a country (maritime spaces, commands) are counted explicitly: a map that showed only its successes would overstate the real coverage.

Synthesis

The thread makes readable what happens between the articles

A thread links the articles of one story — same parties, same operation — and not of one theme. The timeline is at the real scale of time: the gap between publications is the signal, and equidistant markers would make three dispatches in twenty minutes and a story spread over three weeks look identical. No aggregate score is produced at thread level — averaging partly absent scores would fill by computation a gap left deliberately.

Production

A Job for the run, a service for the digest

• The daily run is a Cloud Run Job, not an HTTP call. It takes ~620 s and that duration rises as the verifier covers more, where the scheduler caps at 30 minutes of waiting — triggering it through an endpoint would tie the run to a request timeout.
• The service carries only the digest already produced. One image serves both, with a different command; the front is hosted separately again.
• The Job exits 0 on a truncated run. Cloud Run retries a failed task, and retrying a run that hit the daily cap would produce nothing while burying the work already paid for.

The endpoint that spends is closed behind a token and answers 503 when none is configured, rather than staying open — it triggers a full run, so it burns the day's budget on its own. Reading the digest stays public: that is what a browser does, and it costs nothing.

A non-negotiable guardrail

No summary without evidence

Every summary generated by the LLM must come with a quote verified verbatim against the source text. With no evidence, the item is rejected automatically — not merely flagged.

The same principle applies to the extracted place: empty rather than unverifiable.

Guardrails

Bounded in code, and verified in both directions

• A guardrail declared is not a guardrail. The daily call cap existed in the configuration without being checked anywhere in the code. Found by self-audit, wired, then tested in both directions rather than only in the passing one.
• A cap truncates a run; it never destroys it. When the budget runs out mid-batch the items already produced are kept, the state carries the truncation up to the API, and the item the refusal fell on is not marked as seen — it cost nothing, so it must stay collectable.
• Proven under real concurrency, which only production could give. The reservation is a database transaction: 30 simultaneous reservations across 3 containers for 4 remaining slots, exactly 4 accepted and 26 refused.

The method counted as much as the result. The runbook first prescribed two full runs launched together — which produced a single reservation and did not even overlap, deduplication having starved the second. A counter left under its cap would then have passed for proof.

Estimated value

An illustrative order of magnitude

A favourable ratio even in a degraded scenario (the gain halved). The qualitative benefit — standardisation, traceability — remains decisive beyond time alone.

Risks

Handled by priority, not by exhaustiveness

Evidence · the precision KPI

Four measurements, each stricter than the last

• 27/30 — 90%, above target. Kept only after the annotation itself was questioned.
• 73/88 — 83%, below target. Sources reworked; a bias found in the annotation protocol.
• 51/68 — 75%. Sample rebuilt, and the overall figure decomposed — it was the least informative part of it.
• 37/48 — 77%, CI [63%; 87%]. Annotation now blind. The target sits inside the interval.

These four figures are not a trend, and presenting them as one would be the easiest mistake to make here. Each was taken on a different sample, under a different protocol, against a prompt that had changed in between. What moves from one to the next is the severity of the method, not the quality of the classifier. The last was revised downwards — from 79% — on 2026-09-06, when an annotation inconsistency was settled against the flattering reading.

Evidence · the precision KPI

The first figure was above target. That is why it was questioned.

• Above target, and suspicious for it. 27/30 agreement with the manual annotation. Kept only after questioning the annotation itself — which revealed ambiguous category definitions rather than a defect of the classifier.
• A second, harsher measurement. After the sources were reworked: 73/88 (83%), below target. The annotation protocol carried a bias of its own, documented in the scoping document rather than the figure being presented as settled.

A precision measurement tests the clarity of the definitions as much as the quality of the model. The two are audited together — and the audit corrected itself here: the scoping document stated nothing about the boundary in question, but the prompt had carried an undocumented rule for five days that the model was faithfully applying. A disagreement about the specification, not a gap in it: the rule was changed, not added, and carried back into the scoping document.

Evidence · the precision KPI

One figure was hiding two problems

• 51/68 — 75% on a rebuilt sample. The overall figure was the least informative thing the measurement produced.
• Sorting noise meets the target: 85%. The perimeter decision — does this item belong in the digest at all — holds.
• Qualifying what gets in fails one time in three: 64%. A single figure had been conflating two problems that are not corrected in the same place.

Hence a full confusion matrix rather than a count of disagreements: the direction of an error points at the fix. Confusing two in-scope categories and letting out-of-scope material through are not the same defect, and averaging them hides both.

Evidence · the precision KPI

Blind annotation, and a figure that concludes neither way

• An anchoring bias, removed. The annotation script had until then displayed the system's verdict before asking for the human judgement. Blind by default since.
• 37/48 — 77%, 95% CI [63%; 87%]. At n=48 the 85% target falls inside the interval: the measurement establishes neither that the product meets it nor that it falls short.
• Saying so beats quoting 77% as a verdict. An earlier measurement, whose interval excluded the target, did establish a gap — that one does not, and the difference of status is the result.

Revised downwards on 2026-09-06, from 38/48. Two articles describing the same event — a first example handed to a customer navy — carried opposite labels in the same blind session; settling that inconsistency turned an agreement into a disagreement. The consequence was unfavourable and is published as such. The figure has also aged: the classification prompt has changed three times since, so it no longer describes the current system.

Evidence · the precision KPI

What a probe at n=2 could not see

• The reference cases said the fix worked. A two-arm retest over 17 items: the reworded rule wins both cases it was written for, with no regression introduced.
• The controls said something else. Run this time, where the earlier probe had stopped at two cases, they surfaced two earlier regressions — and a third arm attributed them to the previous day's change rather than to the rewording.
• And the gain the probe had claimed does not reproduce. Replayed four times against the earlier prompt, the case already came out correctly. The probe had been comparing one observation to one observation.

Two rules taken from this, applied to every prompt change since. Check the controls and not only the reference case. And a rule's position counts as much as its content: a first formulation stating exactly the right distinction moved 0 items out of 25 — only rewriting it to lead with the branch that had to win produced the effect.

Evidence · defects found by testing

Symptom at display time, cause upstream

• Two countries on the whole map. The obvious fix was a catch-up rule in the display. Measure first instead: 10 of the 11 missing places had their country named in the title.
• The check was erasing correct extractions. The verbatim comparison only covered the body of the article, so a place named in the headline could never be verified — and was discarded.
• Fixed without weakening the guardrail. A title is text published by the source, so the excerpt must still match it word for word. Coverage went from 6 to 13 places over 16 items.

Fixing it where the symptom appeared would have produced a map that looked right and no longer meant anything — the guardrail is that a place is displayed only if it was verified verbatim.

Evidence · defects found by testing

A crash revealed the loss it was hiding

• One malformed model response interrupted a run. That was the visible failure, and it was the small one.
• On relaunch: zero new items. Deduplication marked links as seen before the analysis — so the items of the lost run were deemed processed while existing nowhere, and were excluded from every later collection.
• Marking now belongs to the node that paid for the call. An interrupted run can no longer leave items presumed seen without ever having been analysed.

None of these defects was visible in a code review. The crash was recoverable in a minute; the silent loss it uncovered would have kept removing items from every subsequent run.

Evidence · defects found by testing

The estimate was contradicted by the real volume

• Feeds exposing months of archive. Reworking the sources surfaced feeds whose RSS carries far more than the recent window the sizing had assumed.
• With no freshness window, the first run spends the day on the backlog. A whole day's LLM budget would have gone into history before the pipeline reached anything current.
• A collection window, and a per-source cap. Both bound the batch before the first call is paid for — a guardrail placed upstream of the spending, not downstream of it.

The same reasoning governs deduplication's position in the graph: it runs before the analysis node, so an item already seen never costs a call to be filtered out.

Roadmap

V1 solid, the next milestone precisely identified

One arbitration stays open on purpose, and three things are not done. Both on the next two slides — stated as what would settle them, not as intentions.

Open arbitration

One budget, three nodes, and no way to serve all three

• A single global counter of 200 calls a day. Extending the verifier to the whole perimeter ran for real on 2026-08-21 — 15 escalations over 36 articles, 5 with an antecedent — and saturated the cap. The shortfall fell on thread grouping, last in the chain.
• First reading of the split, 2026-08-22. Analysis 72 calls, verification 50, grouping 73: 195 of the 200. An escalation costs ~3.7 calls, not one — so the two escalation caps together authorise 140, leaving 60 for an analysis node that pays one call per submitted article and consumed 72. The caps are over-subscribed by construction.
• And the loss is upstream of the arbitration. Broken down over a full batch on 2026-08-30: 99 of 144 analysis calls — 69% — went on articles that were then discarded. Out of scope 72, unverifiable quote 26.

Which is why the arbitration stays open rather than being settled by a round number. The loss is diffuse — the largest single source accounts for 10 calls out of 99, so removing a feed corrects nothing — and the two reasons point at different fixes: composition of the sources on one side, extraction on the other. No redistribution between nodes recovers either. One option is already ruled out: raising the grouping threshold for a budget reason would silently expire a threshold calibrated on a precision measurement.

What is not done

Three open items, and what would close each

• No unattended cycle observed. The scheduler was armed on 2026-09-06 and its trigger validated by a forced run — but an armed trigger is not an observed cycle, and the V1 acceptance criterion asks for the cycle.
• The seven-day purge has never run. It becomes observable on the eighth day of history, not before. Until then the retention guardrail is written and untested against real data.
• Two thresholds are no longer calibrated. The move to English changed the token distribution the escalation score is computed on, and the sample both thresholds were set on is in French. Measured, not assumed — and closing it needs a fresh annotated sample, not a new value.

Observability came before autonomy, deliberately. The deployment diagnosis had not stumbled on the absence of infrastructure but on the absence of a log: the pipeline emitted nothing, so every measurement on the previous slides would have become invisible the moment a scheduler took over. The structured log shipped first, and the container was exercised end to end against real feeds with the call cap forced to zero — which tests truncation without spending anything.