From 58b64c6d5f3218d28869a26bbda9ac4b042c03c4593def4bc9d0f9c4953a96fd Mon Sep 17 00:00:00 2001 From: tlg Date: Tue, 14 Jul 2026 11:54:06 +0200 Subject: [PATCH] docs(projekt-matching): document poison-mail incident and fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design doc: new §4.6 post-mortem (quote char -> guided-decoding whitespace loop -> dead IMAP conn -> non-terminal mail), updated module inventory, vLLM/e-mail experience sections, test counts (51 -> 63) and the latency limitation (now bounded by max_tokens). User manual: version note + troubleshooting row for repeating alerts. Co-Authored-By: Claude Fable 5 --- docs/projekt-matching-design.md | 87 +++++++++++++++++++++++----- docs/projekt-matching-user-manual.md | 5 +- 2 files changed, 75 insertions(+), 17 deletions(-) diff --git a/docs/projekt-matching-design.md b/docs/projekt-matching-design.md index b1999f6..f9750de 100644 --- a/docs/projekt-matching-design.md +++ b/docs/projekt-matching-design.md @@ -5,7 +5,8 @@ as `72a3b17`. Approved spec: `docs/superpowers/specs/2026-07-09-langflow-projekt-matching-design.md`; implementation plan (with deviations preamble): -`docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md`.* +`docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md`. +Updated 2026-07-14 after the poison-mail incident (§4.6, commit `36eac66`).* --- @@ -39,7 +40,7 @@ constraints: ### 2.1 The central decision: logic in a package, Langflow as the runtime All behavior lives in a plain-Python package **`projektmatch`** that is -unit-tested on the host (51 pytest tests) and copied unchanged into the +unit-tested on the host (63 pytest tests) and copied unchanged into the Langflow container. The seven Langflow custom components are ~30-line wrappers that read configuration from Langflow global variables and call one package function each. This kept the visual pipeline Langflow promises while @@ -85,13 +86,13 @@ six visible stages. | `rules.py` | Misc rules, half-up match calc, exact description markdown | | `mailparse.py` | own-mail detection, project splitting (longest-anchor-title dedup by URL path), canonical URL | | `espocrm.py` | CRM client: dedup searches, team lookup (never guessed), ensure-account/contact, unique naming, retries, `X-Status-Reason` surfacing | -| `llm.py` | vLLM calls (`response_format json_schema`, thinking disabled, no `max_tokens`), German prompts + JSON schemas, retry-on-empty, match completeness retry | -| `mailer.py` | IMAP (custom keyword, Trash detection, MOVE+fallback), SMTP (STARTTLS, Date/Message-ID) | -| `stages.py` | Flow-2 stage functions + ctx state machine + CRM verify read-back | +| `llm.py` | vLLM calls (`response_format json_schema`, thinking disabled, `max_tokens` 8192), German prompts + JSON schemas (requirements capped at 40), retry-on-empty, match completeness retry | +| `mailer.py` | IMAP (custom keyword, Trash detection, MOVE+fallback, `ensure_alive()` reconnect), SMTP (STARTTLS, Date/Message-ID, Sent copy) | +| `stages.py` | Flow-2 stage functions + ctx state machine + CRM verify read-back; `page_text()` (project card minus tag cloud, quote sanitization) | | `pm_trace.py` | one consolidated Langfuse trace per project via the ingestion API (never raises) | -| `ingest.py` | Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash | +| `ingest.py` | Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash with flag-checked fallback | | `deploy/` | `deploy_files.sh` (rsync + safe restart), `build_flows.py` (variables, API key, programmatic flow construction), `setup_langfuse.py`, systemd units, webhook trigger | -| `tests/` | 51 unit tests + 5 e2e helper scripts | +| `tests/` | 63 unit tests + 5 e2e helper scripts | ### 2.4 Trust boundaries and secrets @@ -146,9 +147,20 @@ the code and worth knowing before touching the system. - **Empty completions under load** still occur occasionally (shared GPU, concurrent runs): `chat_json` retries empty/unparseable content up to 3× with backoff; non-200 fails fast. -- The old belief "set max_tokens ≥ 1024 for reasoning models" evolved into - **"set no max_tokens at all** and disable thinking where you enforce - schemas". +- **A straight `"` in the INPUT text breaks schema-guided decoding** (found + 2026-07-14, §4.6): the model copies the quote verbatim into a JSON string + value, the grammar reads it as end-of-string, the model cannot continue + its sentence — and escapes into an **infinite whitespace loop** (whitespace + is always grammar-legal) until the token limit, `finish_reason: length`. + vLLM's request-level `structured_outputs.disable_any_whitespace` does NOT + help: the loop simply moves inside a string literal, where spaces are + legal. The robust fix is input-side: `stages.page_text()` maps all quote + characters (`„ “ ” " « » ‹ ›`) to `'` before prompting. +- The old belief "set max_tokens ≥ 1024 for reasoning models" first evolved + into "set no max_tokens at all and disable thinking where you enforce + schemas" — and then, after the whitespace-loop incident, into + **"cap max_tokens at 8192"**: every valid answer fits comfortably, and a + degenerate run now burns ~100 s instead of ~10 min before the retry. ### 4.2 Langflow 1.10 internals @@ -205,6 +217,12 @@ the code and worth knowing before touching the system. mail is flagged and skipped (`summary.unreadable`), the connection is closed in `finally` — otherwise one poison mail would silently stall the pipeline every 5 minutes with no alert. +- **The IMAP server drops idle connections** during long Flow-2 dispatches + (30+ min in the degenerate case, §4.6). `run_ingest` therefore calls + `MailBox.ensure_alive()` (NOOP probe + transparent reconnect) before the + per-mail finalization, and a failing `move_to_trash` falls back to + `flag_checked` (`summary.trashFailed`) so the run continues — the + terminal-state rule survives a dead connection. ### 4.5 Prompt engineering for a 9B model @@ -224,7 +242,41 @@ project (skill result: 86 % Must): deterministic layer plus the >85 gate make this tolerable; the annotation queue exists to watch it. -### 4.6 Process experiences +### 4.6 Post-launch incident: the poison-mail loop (2026-07-13/14) + +Four days after go-live the pipeline stalled for ~26 hours on one mail, +sending an identical `[Projekt-Match-Fehler]` alert every ~35 minutes. The +failure chain — each link necessary, none sufficient alone — is instructive: + +1. **Trigger:** one posting (`nproj/3022414`) contained + `„ISTQB Certified Tester …"` — a typographic opening quote closed by a + straight ASCII `"`. Copied verbatim into a JSON string by the model, the + `"` ended the string for the guided-decoding grammar mid-sentence; the + model escaped into the whitespace loop (§4.1) and burned the full context + on every attempt (~10 min × 3 retries ≈ 35 min per cycle). +2. **Timeout mismatch:** the ingest dispatch timeout (1800 s) fired before + Flow 2 finished, producing a premature `Read timed out` failure. +3. **Trap:** during the long wait the IMAP connection idled out server-side. + The alert still went out (fresh SMTP connection), but `move_to_trash` on + the dead connection raised and aborted `run_ingest` **before** the mail + reached its terminal state — so the next tick reprocessed the same mail, + forever. 66 mails queued up behind it. + +Fix (commit `36eac66`, TDD, live-verified): quote sanitization + feeding only +the project card minus the skill-tag cloud to the LLM (`page_text()`), +`maxItems: 40` on the requirements schema, `max_tokens: 8192`, and the +finalization hardening from §4.4. The poison extraction went from 25+ +consecutive failures to a clean 19-requirement result in 21 s; the backlog +(67 mails) drained in 36 minutes with zero failure alerts and 9 new +opportunities. + +Lessons: **terminal-state guarantees must hold on every path, including +"connection died mid-run"** — the guarantee is only as strong as its weakest +finalization step; and **schema-guided decoding turns bad input characters +into decoding pathologies**, so sanitize model input against the output +grammar, not just for size. + +### 4.7 Process experiences - **Subagent-driven development with per-task adversarial review** caught real defects at every altitude: committed build artifacts, undeclared @@ -245,7 +297,8 @@ project (skill result: 86 % Must): ## 5. Verification summary -- **51 unit tests** (pure package, no network); TDD throughout. +- **63 unit tests** (pure package, no network); TDD throughout (51 at + go-live, +1 Sent-copy fix, +11 poison-mail fix). - **Component gates:** Flow 2 driven directly via API — consider project stable at 86 % across repeated runs with full CRM field verification (team, agency linking, contact, description byte-format); reject project @@ -270,10 +323,12 @@ project (skill result: 86 % Must): - Login-walled postings fail with an alert by design (no snippet fallback). - Created-but-notification-failed edge case is recorded in the trace, not re-sent (spec-accepted). -- Worst-case in-run LLM latency (nested retries) can exceed the 1800 s - dispatch timeout — the ingest side then alerts while Flow 2 may still - finish server-side; dedup prevents damage. Cap the budgets if it ever - bites. +- Worst-case in-run LLM latency is bounded since `36eac66` + (`max_tokens: 8192` → a fully degenerate extraction costs ~100 s × 3 + retries ≈ 6 min, comfortably below the 1800 s dispatch timeout). Should + the timeout ever fire anyway, the ingest side alerts while Flow 2 may + still finish server-side; dedup prevents damage, and the finalization + fallback (§4.4) keeps the trigger mail terminal. - `parse_run_result` anchors on "any embedded JSON with a `status` key" — robust today, but a Langflow upgrade that serializes intermediate ctx objects into the run response would need a stricter anchor. diff --git a/docs/projekt-matching-user-manual.md b/docs/projekt-matching-user-manual.md index cbadc05..a928d13 100644 --- a/docs/projekt-matching-user-manual.md +++ b/docs/projekt-matching-user-manual.md @@ -1,7 +1,9 @@ # Projekt-Matching — User Manual *The automated pipeline that turns freelancermap project e-mails into evaluated -EspoCRM opportunities. Version of 2026-07-09 (all acceptance gates passed).* +EspoCRM opportunities. Version of 2026-07-09 (all acceptance gates passed); +updated 2026-07-14 (robustness fixes: alert-loop bug resolved, outgoing mails +now also appear in the Sent folder).* --- @@ -178,6 +180,7 @@ for the services, `projekt-matching.timer` for the polling). Nothing to do. | No mails processed, INBOX grows | `systemctl --user list-timers` — timer active? Langflow healthy (`curl …/health`)? Restart per section 5. Mails are never lost: the next successful tick processes the backlog. | | Alert mail: `Abruf fehlgeschlagen` / `Seitentext zu kurz` | freelancermap page unreachable or behind a login wall. Evaluate that project manually via the URL in the alert. | | Alert mail: `vLLM HTTP …` / `LLM lieferte kein JSON` / `Read timed out` | the local AI was down or overloaded. Occasional single failures are absorbed by built-in retries; if alerts pile up, check the vLLM service on port 8081. | +| The **same** alert arrives again and again for the same project | this was a bug (fixed 2026-07-14): one poisonous posting could block the pipeline and repeat its alert every ~35 min. A failed project is now reliably final — if you ever see identical repeating alerts again, restart Langflow (section 5) and report it. | | Alert mail: `Team … nicht gefunden` | the CRM user `cowork-api` lost read access to Teams, or a team was renamed. Fix in EspoCRM admin. | | A project you expected is missing | check Langfuse: if its trace says `rejected`, the Must-match was ≤ 85 % — the table shows why. If there is no trace at all, check the alert mails. | | Duplicate opportunity suspected | duplicates are keyed on the canonical URL (`cProjektlink`). The same project posted by two different agencies has two different URLs and is intentionally kept as two opportunities. |