bin/docs/projekt-matching-design.md

Projekt-Matching — Design Description & Experiences

Companion to the user manual. Built 2026-07-09 via spec-driven, subagent-executed development; merged to main as 72a3b17. Approved spec: docs/superpowers/specs/2026-07-09-langflow-projekt-matching-design.md; implementation plan (with deviations preamble): docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md. Updated 2026-07-14 after the poison-mail incident (§4.6, commit 36eac66).

---

1. Goal and constraints

Convert the interactive Claude skill projekt-anlegen (evaluate freelance project postings against Dr.-Ing. Thomas Langer's CV, write results into EspoCRM) into a fully automated Langflow workflow with these hard constraints:

• Local AI only — vLLM endpoint on this host, model AxionML/Qwen3.5-9B-NVFP4; no cloud LLM.
• No human review pause — the skill's mandatory chat review was replaced by defaults (ambiguous offer type → Projekt, ambiguous buyer → agency) and by after-the-fact quality monitoring in Langfuse.
• Numbers are never LLM output — match calculation, Misc rules (start-date window, distance via Nominatim/Haversine, security-clearance K.-o.) and the exact CRM description markdown are deterministic Python.
• Gate: Must-have-Match > 85 % (strictly greater) → CRM + mail; otherwise reject silently (trace only).
• Terminal-state rule: every project ends in exactly one of created / rejected / duplicate / failed on the first attempt; the trigger mail then always moves to Trash; failures produce one aggregated alert per trigger mail. No cross-cycle retries (only bounded in-run HTTP retries) — deliberate protection against retry loops and inbox congestion.
• Duplicate key: canonical project URL (redirects followed, query stripped) stored in and matched against cProjektlink.

2. Architecture

2.1 The central decision: logic in a package, Langflow as the runtime

All behavior lives in a plain-Python package projektmatch that is unit-tested on the host (63 pytest tests) and copied unchanged into the Langflow container. The seven Langflow custom components are ~30-line wrappers that read configuration from Langflow global variables and call one package function each. This kept the visual pipeline Langflow promises while making every business rule testable without a running container — by far the highest-leverage choice in the project: 8 of 16 plan tasks were pure TDD on the package, and almost every live-integration bug was fixable by editing tested package code and re-running one deploy script.

2.2 Runtime topology

systemd user timer (5 min) ── curl POST ──▶ /api/v1/webhook/pm-ingest
                                                  │
   ┌── Flow 1 "PM Ingest" ────────────────────────▼────────┐
   │ Webhook → PMIngest → TextOutput                       │
   │   PMIngest = ingest.run_ingest():                     │
   │   flock lock → IMAP poll (UNKEYWORD $ProjektChecked)  │
   │   → own-mail/non-project filter → split projects      │
   │   → canonical URL → CRM dedup → per NEW project:      │
   │       POST /api/v1/run/<flow2> (x-api-key) ───────────┼──┐
   │   → aggregate → ONE alert mail on failures            │  │
   │   → move trigger mail to Trash → summary              │  │
   └───────────────────────────────────────────────────────┘  │
   ┌── Flow 2 "PM Projekt bewerten" ──────────────────────────▼┐
   │ TextInput → PMFetch → PMExtract → PMMatch → PMRules       │
   │           → PMCrm → PMNotify → TextOutput                 │
   │ ctx dict flows through; status ∈ ok/failed/rejected/      │
   │ created; run_stage() catches exceptions into failed and   │
   │ short-circuits later stages; PMNotify runs UNCONDITIONALLY│
   │ (sends mail only on created, posts Langfuse trace always) │
   └───────────────────────────────────────────────────────────┘

Flow 1 is a single custom component rather than the spec sketch's three, because IMAP connections and per-mail state cannot cross Langflow component boundaries; run_ingest() is fully unit-tested instead. Flow 2 keeps the six visible stages.

2.3 Module inventory

Module	Responsibility
rules.py	Misc rules, half-up match calc, exact description markdown
mailparse.py	own-mail detection, project splitting (longest-anchor-title dedup by URL path), canonical URL
espocrm.py	CRM client: dedup searches, team lookup (never guessed), ensure-account/contact, unique naming, retries, X-Status-Reason surfacing
llm.py	vLLM calls (response_format json_schema, thinking disabled, max_tokens 8192), German prompts + JSON schemas (requirements capped at 40), retry-on-empty, match completeness retry
mailer.py	IMAP (custom keyword, Trash detection, MOVE+fallback, ensure_alive() reconnect), SMTP (STARTTLS, Date/Message-ID, Sent copy)
stages.py	Flow-2 stage functions + ctx state machine + CRM verify read-back; page_text() (project card minus tag cloud, quote sanitization)
pm_trace.py	one consolidated Langfuse trace per project via the ingestion API (never raises)
ingest.py	Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash with flag-checked fallback
deploy/	deploy_files.sh (rsync + safe restart), build_flows.py (variables, API key, programmatic flow construction), setup_langfuse.py, systemd units, webhook trigger
tests/	63 unit tests + 5 e2e helper scripts

2.4 Trust boundaries and secrets

IMAP/SMTP password and EspoCRM API key live in Langflow global variables (encrypted in Postgres) and chmod-600 files under ~/.config/projekt-matching/; they never entered git (verified across all 31 commits in the final review). Langfuse keys follow the pod script's pre-existing convention (tracked, local loopback services only — an accepted, user-confirmed trade-off). The CV and the CRM secrets file are owned by tlg and readable by lwc via setfacl -m u:lwc:r.

3. Key design decisions and their rationale

1. Canonical-URL dedup (not name-based): deterministic, immune to title edits; the same job at two agencies intentionally stays two opportunities.
2. Delete-mail-always + alert (revised mid-design from "retry until success"): deterministic failures would otherwise loop every 5 minutes forever. The alert transfers responsibility to the human; in-run HTTP retries (2–3 attempts) absorb transient blips.
3. Deterministic scoring layer: the LLM only classifies and rates; Python computes. This survived the model's quality variance — when ratings flickered, the numbers stayed honest and auditable.
4. One custom Langfuse trace per project (via the ingestion REST API, not the SDK): stable contract, independent of Langflow's own auto-tracing, filterable by outcome tags, and the object your annotation queue feeds on. Posted last in stage_notify, unconditionally — even for failures.
5. Programmatic flow construction (build_flows.py assembles node/edge JSON via POST /api/v1/custom_component templates): flows are reproducible from git; the Langflow UI is never the source of truth.
6. Everything idempotent: variable upserts, flow delete-and-recreate, API-key delete-and-recreate, file rsync — any deploy step can be re-run.

4. Experiences — what the plan got wrong and what we learned

The plan deliberately marked three "known unknowns" with verify-and-adapt steps; live integration surfaced several more. Each item below is baked into the code and worth knowing before touching the system.

4.1 vLLM / Qwen3.5 structured output

• guided_json is silently ignored by vLLM 0.22.1 — HTTP 200, unconstrained prose output. The plan's guided_json-primary + 400-triggered-fallback design could therefore never fall back. Fixed by using response_format {"type": "json_schema"} as the sole mechanism (verified enforced: exact schema keys returned).
• Reasoning + schema enforcement degenerate: with the qwen3 reasoning parser active, json_schema decoding sent the model into thinking loops that consumed the full 63k context (finish_reason: length, empty content). Root-cause fix: chat_template_kwargs {"enable_thinking": false} for structured calls — extraction dropped from minutes/never to ~35 s. A test locks this parameter in place.
• Empty completions under load still occur occasionally (shared GPU, concurrent runs): chat_json retries empty/unparseable content up to 3× with backoff; non-200 fails fast.
• A straight " in the INPUT text breaks schema-guided decoding (found 2026-07-14, §4.6): the model copies the quote verbatim into a JSON string value, the grammar reads it as end-of-string, the model cannot continue its sentence — and escapes into an infinite whitespace loop (whitespace is always grammar-legal) until the token limit, finish_reason: length. vLLM's request-level structured_outputs.disable_any_whitespace does NOT help: the loop simply moves inside a string literal, where spaces are legal. The robust fix is input-side: stages.page_text() maps all quote characters („ “ ” " « » ‹ ›) to ' before prompting.
• The old belief "set max_tokens ≥ 1024 for reasoning models" first evolved into "set no max_tokens at all and disable thinking where you enforce schemas" — and then, after the whitespace-loop incident, into "cap max_tokens at 8192": every valid answer fits comfortably, and a degenerate run now burns ~100 s instead of ~10 min before the retry.

4.2 Langflow 1.10 internals

• Component.ctx is a reserved property — a DataInput(name="ctx") silently shadows the wired edge payload. All data inputs are named ctx_in.
• The custom-component loader resolves imports statically — in-file sys.path.insert() hacks don't work. First fix (a .pth file in the container venv) turned out non-durable because the systemd --new unit recreates the container on every restart, wiping ephemeral writes. The durable fix bakes PYTHONPATH=/app/langflow into the container definition in create_pod_langflow.sh.
• Never podman restart a systemd-managed --rm/--new container — it can end up removed with the unit inactive. Always systemctl --user restart container-langflow_ctr.service (deploy_files.sh does this and waits for health).
• Webhook lives in the input_output category of GET /api/v1/all (not data); Data outputs wire as type JSON; edge handles use the œ-escaped JSON encoding. build_flows.py derives types from live templates instead of hardcoding — the plan's explicit "verify against a dumped existing flow first" step paid off.
• POST /api/v1/api_key accumulates same-named keys; the builder now deletes existing projekt-matching keys before minting (list shape: {"api_keys": [...]}).
• podman logs langflow_ctr is useless (a structlog recursion loop floods it); the monitor/builds API is the working observability path.
• Auth split (from earlier experiments, confirmed): flow CRUD/variables need the Bearer JWT from /api/v1/auto_login; run and webhook need x-api-key; responses are gzip (curl --compressed).
• Global variables with load_from_db=True resolve at run time — the Gate-3 recipient flip needed no restart.

4.3 Langfuse 3.195 (OSS)

• The Organization Management API is EE-gated (routes exist, return 403). setup_langfuse.py falls back to a NextAuth session + tRPC calls to create org/project/keys — fragile by nature, acceptable for one-shot setup, kept behind an admin-API-first attempt.
• Score configs can be created via the public API with project keys; annotation queues are UI-only (manual step, done).
• Score-config creation is not idempotent (archive-only API) — documented in the script.

4.4 E-mail

• Outgoing EmailMessages need explicit Date and Message-ID headers; without them clients display 01/01/1970 (user-reported, fixed, and the fix had to wait for a container restart — a long-lived process keeps the old module in memory).
• The mail server (Dovecot) supports custom IMAP keywords (PERMANENTFLAGS \*) and UID MOVE; the Trash folder is literally Trash. The $ProjektChecked keyword implements "inspect non-project mail exactly once, touch nothing".
• A malformed FETCH must not kill the poll loop: hardened so an unreadable mail is flagged and skipped (summary.unreadable), the connection is closed in finally — otherwise one poison mail would silently stall the pipeline every 5 minutes with no alert.
• The IMAP server drops idle connections during long Flow-2 dispatches (30+ min in the degenerate case, §4.6). run_ingest therefore calls MailBox.ensure_alive() (NOOP probe + transparent reconnect) before the per-mail finalization, and a failing move_to_trash falls back to flag_checked (summary.trashFailed) so the run continues — the terminal-state rule survives a dead connection.

4.5 Prompt engineering for a 9B model

The extraction/matching prompts embed the skill's German calibration rules verbatim (strictness: PoC ≠ production, "mehrjährig" literal, "wie z. B." alternatives count). Two tuning iterations were needed against the reference project (skill result: 86 % Must):

• Signal words must bind to their own line only — trailing unmarked lines after a "von Vorteil" line were bleeding into Nice; as Must the reference computes 18✅+3❌ of 21 = 85.71 → 86, exactly reproducing the skill.
• …with an explicit exception for a literal "Nice-to-haves" heading (heading precedence stays above line-level signals) — added after review caught the over-generalization.
• Residual variance: Must-match stable (86 in all post-fix runs); Nice-match flips a few points run-to-run; Misc rows occasionally missing. The deterministic layer plus the >85 gate make this tolerable; the annotation queue exists to watch it.

4.6 Post-launch incident: the poison-mail loop (2026-07-13/14)

Four days after go-live the pipeline stalled for ~26 hours on one mail, sending an identical [Projekt-Match-Fehler] alert every ~35 minutes. The failure chain — each link necessary, none sufficient alone — is instructive:

1. Trigger: one posting (nproj/3022414) contained „ISTQB Certified Tester …" — a typographic opening quote closed by a straight ASCII ". Copied verbatim into a JSON string by the model, the " ended the string for the guided-decoding grammar mid-sentence; the model escaped into the whitespace loop (§4.1) and burned the full context on every attempt (~10 min × 3 retries ≈ 35 min per cycle).
2. Timeout mismatch: the ingest dispatch timeout (1800 s) fired before Flow 2 finished, producing a premature Read timed out failure.
3. Trap: during the long wait the IMAP connection idled out server-side. The alert still went out (fresh SMTP connection), but move_to_trash on the dead connection raised and aborted run_ingest before the mail reached its terminal state — so the next tick reprocessed the same mail, forever. 66 mails queued up behind it.

Fix (commit 36eac66, TDD, live-verified): quote sanitization + feeding only the project card minus the skill-tag cloud to the LLM (page_text()), maxItems: 40 on the requirements schema, max_tokens: 8192, and the finalization hardening from §4.4. The poison extraction went from 25+ consecutive failures to a clean 19-requirement result in 21 s; the backlog (67 mails) drained in 36 minutes with zero failure alerts and 9 new opportunities.

Lessons: terminal-state guarantees must hold on every path, including "connection died mid-run" — the guarantee is only as strong as its weakest finalization step; and schema-guided decoding turns bad input characters into decoding pathologies, so sanitize model input against the output grammar, not just for size.

4.7 Process experiences

• Subagent-driven development with per-task adversarial review caught real defects at every altitude: committed build artifacts, undeclared dependencies, a read-back that verified presence instead of equality, alert-send blocking the trash guarantee, the prompt over-generalization. Two verification-only gates were audited as evidence (timestamps, IDs, counts, alternative explanations) rather than trusted — one audit forced a trace-level proof that an unexpected notification during Gate 2 was genuine production traffic (it was: a real project, Must 100 %, evaluated and filed while we were testing).
• Live systems interleave with tests. Real project mails arrived and were processed during every e2e window. Distinguishing test artifacts from production artifacts by canonical-URL tokens — and writing that discipline into the helper scripts — was essential.
• The progress ledger (.superpowers/sdd/progress.md, git-ignored) carried environment facts between tasks; most of its content graduated into this document and the memory notes.

5. Verification summary

• 63 unit tests (pure package, no network); TDD throughout (51 at go-live, +1 Sent-copy fix, +11 poison-mail fix).
• Component gates: Flow 2 driven directly via API — consider project stable at 86 % across repeated runs with full CRM field verification (team, agency linking, contact, description byte-format); reject project at 0 %, no writes.
• Gate 1 (e2e consider): re-sent original example-1 mail (5 projects) → timer → project 4 created (86 %), notification with valid Date, trigger mail in Trash, 5 traces (1 created / 4 rejected), zero unwanted entries.
• Gate 2 (e2e reject): example-2 mail → full table in trace, Must 0 %, no CRM entry, no mail, mail in Trash.
• Dedup: re-sent Gate-1 mail → ingest summary {projects: 5, created: 0, rejected: 4, duplicate: 1}, notification count unchanged, no (2) entry.
• Gate 3 (production recipient): variables flipped live (no restart), opportunity re-created at 86 %, no mail in the design-phase inbox, SMTP acceptance proven via trace error-absence, receipt at Thomas.Langer@destengs.com confirmed by Thomas.

6. Known limitations & future work

• freelancermap-only detection (extension point: mailparse.PROJECT_URL_RE
  • splitting logic).
• Login-walled postings fail with an alert by design (no snippet fallback).
• Created-but-notification-failed edge case is recorded in the trace, not re-sent (spec-accepted).
• Worst-case in-run LLM latency is bounded since 36eac66 (max_tokens: 8192 → a fully degenerate extraction costs ~100 s × 3 retries ≈ 6 min, comfortably below the 1800 s dispatch timeout). Should the timeout ever fire anyway, the ingest side alerts while Flow 2 may still finish server-side; dedup prevents damage, and the finalization fallback (§4.4) keeps the trigger mail terminal.
• parse_run_result anchors on "any embedded JSON with a status key" — robust today, but a Langflow upgrade that serializes intermediate ctx objects into the run response would need a stricter anchor.
• niceMatch variance / Misc extraction variance: monitor via the projekt-matching-review annotation queue; if quality drifts, tuning lives in llm.py's prompts and is regression-tested by tests/e2e/run_flow2.py against the two reference projects.

7. Restore from a fresh clone (runbook)

1. bash create_pod_langflow.sh — pod, DBs, Langflow, Langfuse (bootstrap admin + tracing keys baked in).
2. Ensure ACLs: setfacl -m u:lwc:r on the CV and .secrets/espocrm-api.md (as tlg); create ~/.config/projekt-matching/secrets.env (0600) with PM_IMAP_PASSWORD.
3. projekt-matching/deploy/deploy_files.sh — package + vorgaben into the data dir.
4. set -a; source projekt-matching/deploy/secrets.local.env; set +a then .venv/bin/python projekt-matching/deploy/build_flows.py — variables, API key, both flows, ~/.config/projekt-matching/env.
5. .venv/bin/python projekt-matching/deploy/setup_langfuse.py — org, project, keys, score configs (annotation queue: manual UI step); wire the printed keys into the pod script and rerun it.
6. cp projekt-matching/deploy/projekt-matching.{service,timer} ~/.config/systemd/user/ then systemctl --user daemon-reload && systemctl --user enable --now projekt-matching.timer.
7. Optional clean start: .venv/bin/python projekt-matching/tests/e2e/flag_inbox.py to baseline the current INBOX before the first tick.