bin/docs/projekt-matching-design.md

# Projekt-Matching — Design Description & Experiences

*Companion to the [user manual](projekt-matching-user-manual.md). Built
2026-07-09 via spec-driven, subagent-executed development; merged to `main`
as `72a3b17`. Approved spec:
`docs/superpowers/specs/2026-07-09-langflow-projekt-matching-design.md`;
implementation plan (with deviations preamble):
`docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md`.
Updated 2026-07-14 after the poison-mail incident (§4.6, commit `36eac66`).*

---

## 1. Goal and constraints

Convert the interactive Claude skill `projekt-anlegen` (evaluate freelance
project postings against Dr.-Ing. Thomas Langer's CV, write results into
EspoCRM) into a fully automated Langflow workflow with these hard
constraints:

- **Local AI only** — vLLM endpoint on this host, model
  `AxionML/Qwen3.5-9B-NVFP4`; no cloud LLM.
- **No human review pause** — the skill's mandatory chat review was replaced
  by defaults (ambiguous offer type → `Projekt`, ambiguous buyer → agency)
  and by after-the-fact quality monitoring in Langfuse.
- **Numbers are never LLM output** — match calculation, Misc rules
  (start-date window, distance via Nominatim/Haversine, security-clearance
  K.-o.) and the exact CRM description markdown are deterministic Python.
- **Gate:** Must-have-Match **> 85 %** (strictly greater) → CRM + mail;
  otherwise reject silently (trace only).
- **Terminal-state rule:** every project ends in exactly one of
  `created / rejected / duplicate / failed` on the **first** attempt; the
  trigger mail then always moves to Trash; failures produce one aggregated
  alert per trigger mail. No cross-cycle retries (only bounded in-run HTTP
  retries) — deliberate protection against retry loops and inbox congestion.
- **Duplicate key:** canonical project URL (redirects followed, query
  stripped) stored in and matched against `cProjektlink`.

## 2. Architecture

### 2.1 The central decision: logic in a package, Langflow as the runtime

All behavior lives in a plain-Python package **`projektmatch`** that is
unit-tested on the host (63 pytest tests) and copied unchanged into the
Langflow container. The seven Langflow custom components are ~30-line
wrappers that read configuration from Langflow global variables and call one
package function each. This kept the visual pipeline Langflow promises while
making every business rule testable without a running container — by far the
highest-leverage choice in the project: 8 of 16 plan tasks were pure TDD on
the package, and almost every live-integration bug was fixable by editing
tested package code and re-running one deploy script.

### 2.2 Runtime topology

```
systemd user timer (5 min) ── curl POST ──▶ /api/v1/webhook/pm-ingest
                                                  │
   ┌── Flow 1 "PM Ingest" ────────────────────────▼────────┐
   │ Webhook → PMIngest → TextOutput                       │
   │   PMIngest = ingest.run_ingest():                     │
   │   flock lock → IMAP poll (UNKEYWORD $ProjektChecked)  │
   │   → own-mail/non-project filter → split projects      │
   │   → canonical URL → CRM dedup → per NEW project:      │
   │       POST /api/v1/run/<flow2> (x-api-key) ───────────┼──┐
   │   → aggregate → ONE alert mail on failures            │  │
   │   → move trigger mail to Trash → summary              │  │
   └───────────────────────────────────────────────────────┘  │
   ┌── Flow 2 "PM Projekt bewerten" ──────────────────────────▼┐
   │ TextInput → PMFetch → PMExtract → PMMatch → PMRules       │
   │           → PMCrm → PMNotify → TextOutput                 │
   │ ctx dict flows through; status ∈ ok/failed/rejected/      │
   │ created; run_stage() catches exceptions into failed and   │
   │ short-circuits later stages; PMNotify runs UNCONDITIONALLY│
   │ (sends mail only on created, posts Langfuse trace always) │
   └───────────────────────────────────────────────────────────┘
```

Flow 1 is a single custom component rather than the spec sketch's three,
because IMAP connections and per-mail state cannot cross Langflow component
boundaries; `run_ingest()` is fully unit-tested instead. Flow 2 keeps the
six visible stages.

### 2.3 Module inventory

| Module | Responsibility |
|---|---|
| `rules.py` | Misc rules, half-up match calc, exact description markdown |
| `mailparse.py` | own-mail detection, project splitting (longest-anchor-title dedup by URL path), canonical URL |
| `espocrm.py` | CRM client: dedup searches, team lookup (never guessed), ensure-account/contact, unique naming, retries, `X-Status-Reason` surfacing |
| `llm.py` | vLLM calls (`response_format json_schema`, thinking disabled, `max_tokens` 8192), German prompts + JSON schemas (requirements capped at 40), retry-on-empty, match completeness retry |
| `mailer.py` | IMAP (custom keyword, Trash detection, MOVE+fallback, `ensure_alive()` reconnect), SMTP (STARTTLS, Date/Message-ID, Sent copy) |
| `stages.py` | Flow-2 stage functions + ctx state machine + CRM verify read-back; `page_text()` (project card minus tag cloud, quote sanitization) |
| `pm_trace.py` | one consolidated Langfuse trace per project via the ingestion API (never raises) |
| `ingest.py` | Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash with flag-checked fallback |
| `deploy/` | `deploy_files.sh` (rsync + safe restart), `build_flows.py` (variables, API key, programmatic flow construction), `setup_langfuse.py`, systemd units, webhook trigger |
| `tests/` | 63 unit tests + 5 e2e helper scripts |

### 2.4 Trust boundaries and secrets

IMAP/SMTP password and EspoCRM API key live in Langflow global variables
(encrypted in Postgres) and chmod-600 files under `~/.config/projekt-matching/`;
they never entered git (verified across all 31 commits in the final review).
Langfuse keys follow the pod script's pre-existing convention (tracked, local
loopback services only — an accepted, user-confirmed trade-off). The CV and
the CRM secrets file are owned by `tlg` and readable by `lwc` via
`setfacl -m u:lwc:r`.

## 3. Key design decisions and their rationale

1. **Canonical-URL dedup** (not name-based): deterministic, immune to title
   edits; the same job at two agencies intentionally stays two opportunities.
2. **Delete-mail-always + alert** (revised mid-design from
   "retry until success"): deterministic failures would otherwise loop every
   5 minutes forever. The alert transfers responsibility to the human;
   in-run HTTP retries (2–3 attempts) absorb transient blips.
3. **Deterministic scoring layer**: the LLM only classifies and rates;
   Python computes. This survived the model's quality variance — when
   ratings flickered, the numbers stayed honest and auditable.
4. **One custom Langfuse trace per project** (via the ingestion REST API, not
   the SDK): stable contract, independent of Langflow's own auto-tracing,
   filterable by outcome tags, and the object your annotation queue feeds on.
   Posted *last* in `stage_notify`, unconditionally — even for failures.
5. **Programmatic flow construction** (`build_flows.py` assembles node/edge
   JSON via `POST /api/v1/custom_component` templates): flows are
   reproducible from git; the Langflow UI is never the source of truth.
6. **Everything idempotent**: variable upserts, flow delete-and-recreate,
   API-key delete-and-recreate, file rsync — any deploy step can be re-run.

## 4. Experiences — what the plan got wrong and what we learned

The plan deliberately marked three "known unknowns" with verify-and-adapt
steps; live integration surfaced several more. Each item below is baked into
the code and worth knowing before touching the system.

### 4.1 vLLM / Qwen3.5 structured output

- **`guided_json` is silently ignored** by vLLM 0.22.1 — HTTP 200,
  unconstrained prose output. The plan's guided_json-primary +
  400-triggered-fallback design could therefore never fall back. Fixed by
  using **`response_format {"type": "json_schema"}`** as the sole mechanism
  (verified enforced: exact schema keys returned).
- **Reasoning + schema enforcement degenerate**: with the qwen3 reasoning
  parser active, `json_schema` decoding sent the model into thinking loops
  that consumed the full 63k context (`finish_reason: length`, empty
  content). Root-cause fix: `chat_template_kwargs {"enable_thinking": false}`
  for structured calls — extraction dropped from minutes/never to ~35 s.
  A test locks this parameter in place.
- **Empty completions under load** still occur occasionally (shared GPU,
  concurrent runs): `chat_json` retries empty/unparseable content up to 3×
  with backoff; non-200 fails fast.
- **A straight `"` in the INPUT text breaks schema-guided decoding** (found
  2026-07-14, §4.6): the model copies the quote verbatim into a JSON string
  value, the grammar reads it as end-of-string, the model cannot continue
  its sentence — and escapes into an **infinite whitespace loop** (whitespace
  is always grammar-legal) until the token limit, `finish_reason: length`.
  vLLM's request-level `structured_outputs.disable_any_whitespace` does NOT
  help: the loop simply moves inside a string literal, where spaces are
  legal. The robust fix is input-side: `stages.page_text()` maps all quote
  characters (`„ “ ” " « » ‹ ›`) to `'` before prompting.
- The old belief "set max_tokens ≥ 1024 for reasoning models" first evolved
  into "set no max_tokens at all and disable thinking where you enforce
  schemas" — and then, after the whitespace-loop incident, into
  **"cap max_tokens at 8192"**: every valid answer fits comfortably, and a
  degenerate run now burns ~100 s instead of ~10 min before the retry.

### 4.2 Langflow 1.10 internals

- **`Component.ctx` is a reserved property** — a `DataInput(name="ctx")`
  silently shadows the wired edge payload. All data inputs are named
  `ctx_in`.
- **The custom-component loader resolves imports statically** — in-file
  `sys.path.insert()` hacks don't work. First fix (a `.pth` file in the
  container venv) turned out non-durable because the systemd `--new` unit
  **recreates the container on every restart**, wiping ephemeral writes. The
  durable fix bakes `PYTHONPATH=/app/langflow` into the container definition
  in `create_pod_langflow.sh`.
- **Never `podman restart` a systemd-managed `--rm/--new` container** — it
  can end up removed with the unit inactive. Always
  `systemctl --user restart container-langflow_ctr.service`
  (`deploy_files.sh` does this and waits for health).
- Webhook lives in the `input_output` category of `GET /api/v1/all` (not
  `data`); `Data` outputs wire as type `JSON`; edge handles use the
  `œ`-escaped JSON encoding. `build_flows.py` derives types from live
  templates instead of hardcoding — the plan's explicit "verify against a
  dumped existing flow first" step paid off.
- `POST /api/v1/api_key` accumulates same-named keys; the builder now
  deletes existing `projekt-matching` keys before minting (list shape:
  `{"api_keys": [...]}`).
- **`podman logs langflow_ctr` is useless** (a structlog recursion loop
  floods it); the monitor/builds API is the working observability path.
- Auth split (from earlier experiments, confirmed): flow CRUD/variables need
  the Bearer JWT from `/api/v1/auto_login`; `run` and webhook need
  `x-api-key`; responses are gzip (`curl --compressed`).
- Global variables with `load_from_db=True` resolve **at run time** — the
  Gate-3 recipient flip needed no restart.

### 4.3 Langfuse 3.195 (OSS)

- The **Organization Management API is EE-gated** (routes exist, return 403).
  `setup_langfuse.py` falls back to a NextAuth session + tRPC calls to create
  org/project/keys — fragile by nature, acceptable for one-shot setup, kept
  behind an admin-API-first attempt.
- Score configs can be created via the public API with project keys;
  **annotation queues are UI-only** (manual step, done).
- Score-config creation is not idempotent (archive-only API) — documented in
  the script.

### 4.4 E-mail

- Outgoing `EmailMessage`s **need explicit `Date` and `Message-ID` headers**;
  without them clients display 01/01/1970 (user-reported, fixed, and the fix
  had to wait for a container restart — a long-lived process keeps the old
  module in memory).
- The mail server (Dovecot) supports custom IMAP keywords (`PERMANENTFLAGS \*`)
  and UID MOVE; the Trash folder is literally `Trash`. The `$ProjektChecked`
  keyword implements "inspect non-project mail exactly once, touch nothing".
- A malformed FETCH must not kill the poll loop: hardened so an unreadable
  mail is flagged and skipped (`summary.unreadable`), the connection is
  closed in `finally` — otherwise one poison mail would silently stall the
  pipeline every 5 minutes with no alert.
- **The IMAP server drops idle connections** during long Flow-2 dispatches
  (30+ min in the degenerate case, §4.6). `run_ingest` therefore calls
  `MailBox.ensure_alive()` (NOOP probe + transparent reconnect) before the
  per-mail finalization, and a failing `move_to_trash` falls back to
  `flag_checked` (`summary.trashFailed`) so the run continues — the
  terminal-state rule survives a dead connection.

### 4.5 Prompt engineering for a 9B model

The extraction/matching prompts embed the skill's German calibration rules
verbatim (strictness: PoC ≠ production, "mehrjährig" literal, "wie z. B."
alternatives count). Two tuning iterations were needed against the reference
project (skill result: 86 % Must):

- **Signal words must bind to their own line only** — trailing unmarked lines
  after a "von Vorteil" line were bleeding into Nice; as Must the reference
  computes 18✅+3❌ of 21 = 85.71 → 86, exactly reproducing the skill.
- …with an **explicit exception for a literal "Nice-to-haves" heading**
  (heading precedence stays above line-level signals) — added after review
  caught the over-generalization.
- Residual variance: Must-match stable (86 in all post-fix runs); Nice-match
  flips a few points run-to-run; Misc rows occasionally missing. The
  deterministic layer plus the >85 gate make this tolerable; the annotation
  queue exists to watch it.

### 4.6 Post-launch incident: the poison-mail loop (2026-07-13/14)

Four days after go-live the pipeline stalled for ~26 hours on one mail,
sending an identical `[Projekt-Match-Fehler]` alert every ~35 minutes. The
failure chain — each link necessary, none sufficient alone — is instructive:

1. **Trigger:** one posting (`nproj/3022414`) contained
   `„ISTQB Certified Tester …"` — a typographic opening quote closed by a
   straight ASCII `"`. Copied verbatim into a JSON string by the model, the
   `"` ended the string for the guided-decoding grammar mid-sentence; the
   model escaped into the whitespace loop (§4.1) and burned the full context
   on every attempt (~10 min × 3 retries ≈ 35 min per cycle).
2. **Timeout mismatch:** the ingest dispatch timeout (1800 s) fired before
   Flow 2 finished, producing a premature `Read timed out` failure.
3. **Trap:** during the long wait the IMAP connection idled out server-side.
   The alert still went out (fresh SMTP connection), but `move_to_trash` on
   the dead connection raised and aborted `run_ingest` **before** the mail
   reached its terminal state — so the next tick reprocessed the same mail,
   forever. 66 mails queued up behind it.

Fix (commit `36eac66`, TDD, live-verified): quote sanitization + feeding only
the project card minus the skill-tag cloud to the LLM (`page_text()`),
`maxItems: 40` on the requirements schema, `max_tokens: 8192`, and the
finalization hardening from §4.4. The poison extraction went from 25+
consecutive failures to a clean 19-requirement result in 21 s; the backlog
(67 mails) drained in 36 minutes with zero failure alerts and 9 new
opportunities.

Lessons: **terminal-state guarantees must hold on every path, including
"connection died mid-run"** — the guarantee is only as strong as its weakest
finalization step; and **schema-guided decoding turns bad input characters
into decoding pathologies**, so sanitize model input against the output
grammar, not just for size.

### 4.7 Process experiences

- **Subagent-driven development with per-task adversarial review** caught
  real defects at every altitude: committed build artifacts, undeclared
  dependencies, a read-back that verified presence instead of equality,
  alert-send blocking the trash guarantee, the prompt over-generalization.
  Two verification-only gates were audited as *evidence* (timestamps, IDs,
  counts, alternative explanations) rather than trusted — one audit forced a
  trace-level proof that an unexpected notification during Gate 2 was
  genuine production traffic (it was: a real project, Must 100 %, evaluated
  and filed while we were testing).
- **Live systems interleave with tests.** Real project mails arrived and
  were processed during every e2e window. Distinguishing test artifacts from
  production artifacts by canonical-URL tokens — and writing that discipline
  into the helper scripts — was essential.
- The **progress ledger** (`.superpowers/sdd/progress.md`, git-ignored)
  carried environment facts between tasks; most of its content graduated
  into this document and the memory notes.

## 5. Verification summary

- **63 unit tests** (pure package, no network); TDD throughout (51 at
  go-live, +1 Sent-copy fix, +11 poison-mail fix).
- **Component gates:** Flow 2 driven directly via API — consider project
  stable at 86 % across repeated runs with full CRM field verification
  (team, agency linking, contact, description byte-format); reject project
  at 0 %, no writes.
- **Gate 1 (e2e consider):** re-sent original example-1 mail (5 projects) →
  timer → project 4 created (86 %), notification with valid Date, trigger
  mail in Trash, 5 traces (1 created / 4 rejected), zero unwanted entries.
- **Gate 2 (e2e reject):** example-2 mail → full table in trace, Must 0 %,
  no CRM entry, no mail, mail in Trash.
- **Dedup:** re-sent Gate-1 mail → ingest summary
  `{projects: 5, created: 0, rejected: 4, duplicate: 1}`, notification count
  unchanged, no ` (2)` entry.
- **Gate 3 (production recipient):** variables flipped live (no restart),
  opportunity re-created at 86 %, no mail in the design-phase inbox, SMTP
  acceptance proven via trace error-absence, **receipt at
  Thomas.Langer@destengs.com confirmed by Thomas**.

## 6. Known limitations & future work

- freelancermap-only detection (extension point: `mailparse.PROJECT_URL_RE`
  + splitting logic).
- Login-walled postings fail with an alert by design (no snippet fallback).
- Created-but-notification-failed edge case is recorded in the trace, not
  re-sent (spec-accepted).
- Worst-case in-run LLM latency is bounded since `36eac66`
  (`max_tokens: 8192` → a fully degenerate extraction costs ~100 s × 3
  retries ≈ 6 min, comfortably below the 1800 s dispatch timeout). Should
  the timeout ever fire anyway, the ingest side alerts while Flow 2 may
  still finish server-side; dedup prevents damage, and the finalization
  fallback (§4.4) keeps the trigger mail terminal.
- `parse_run_result` anchors on "any embedded JSON with a `status` key" —
  robust today, but a Langflow upgrade that serializes intermediate ctx
  objects into the run response would need a stricter anchor.
- niceMatch variance / Misc extraction variance: monitor via the
  `projekt-matching-review` annotation queue; if quality drifts, tuning
  lives in `llm.py`'s prompts and is regression-tested by
  `tests/e2e/run_flow2.py` against the two reference projects.

## 7. Restore from a fresh clone (runbook)

1. `bash create_pod_langflow.sh` — pod, DBs, Langflow, Langfuse (bootstrap
   admin + tracing keys baked in).
2. Ensure ACLs: `setfacl -m u:lwc:r` on the CV and
   `.secrets/espocrm-api.md` (as `tlg`); create
   `~/.config/projekt-matching/secrets.env` (0600) with `PM_IMAP_PASSWORD`.
3. `projekt-matching/deploy/deploy_files.sh` — package + vorgaben into the
   data dir.
4. `set -a; source projekt-matching/deploy/secrets.local.env; set +a` then
   `.venv/bin/python projekt-matching/deploy/build_flows.py` — variables,
   API key, both flows, `~/.config/projekt-matching/env`.
5. `.venv/bin/python projekt-matching/deploy/setup_langfuse.py` — org,
   project, keys, score configs (annotation queue: manual UI step); wire the
   printed keys into the pod script and rerun it.
6. `cp projekt-matching/deploy/projekt-matching.{service,timer} ~/.config/systemd/user/`
   then `systemctl --user daemon-reload && systemctl --user enable --now projekt-matching.timer`.
7. Optional clean start: `.venv/bin/python projekt-matching/tests/e2e/flag_inbox.py`
   to baseline the current INBOX before the first tick.