304 lines
17 KiB
Markdown
304 lines
17 KiB
Markdown
# Projekt-Matching — Design Description & Experiences
|
||
|
||
*Companion to the [user manual](projekt-matching-user-manual.md). Built
|
||
2026-07-09 via spec-driven, subagent-executed development; merged to `main`
|
||
as `72a3b17`. Approved spec:
|
||
`docs/superpowers/specs/2026-07-09-langflow-projekt-matching-design.md`;
|
||
implementation plan (with deviations preamble):
|
||
`docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md`.*
|
||
|
||
---
|
||
|
||
## 1. Goal and constraints
|
||
|
||
Convert the interactive Claude skill `projekt-anlegen` (evaluate freelance
|
||
project postings against Dr.-Ing. Thomas Langer's CV, write results into
|
||
EspoCRM) into a fully automated Langflow workflow with these hard
|
||
constraints:
|
||
|
||
- **Local AI only** — vLLM endpoint on this host, model
|
||
`AxionML/Qwen3.5-9B-NVFP4`; no cloud LLM.
|
||
- **No human review pause** — the skill's mandatory chat review was replaced
|
||
by defaults (ambiguous offer type → `Projekt`, ambiguous buyer → agency)
|
||
and by after-the-fact quality monitoring in Langfuse.
|
||
- **Numbers are never LLM output** — match calculation, Misc rules
|
||
(start-date window, distance via Nominatim/Haversine, security-clearance
|
||
K.-o.) and the exact CRM description markdown are deterministic Python.
|
||
- **Gate:** Must-have-Match **> 85 %** (strictly greater) → CRM + mail;
|
||
otherwise reject silently (trace only).
|
||
- **Terminal-state rule:** every project ends in exactly one of
|
||
`created / rejected / duplicate / failed` on the **first** attempt; the
|
||
trigger mail then always moves to Trash; failures produce one aggregated
|
||
alert per trigger mail. No cross-cycle retries (only bounded in-run HTTP
|
||
retries) — deliberate protection against retry loops and inbox congestion.
|
||
- **Duplicate key:** canonical project URL (redirects followed, query
|
||
stripped) stored in and matched against `cProjektlink`.
|
||
|
||
## 2. Architecture
|
||
|
||
### 2.1 The central decision: logic in a package, Langflow as the runtime
|
||
|
||
All behavior lives in a plain-Python package **`projektmatch`** that is
|
||
unit-tested on the host (51 pytest tests) and copied unchanged into the
|
||
Langflow container. The seven Langflow custom components are ~30-line
|
||
wrappers that read configuration from Langflow global variables and call one
|
||
package function each. This kept the visual pipeline Langflow promises while
|
||
making every business rule testable without a running container — by far the
|
||
highest-leverage choice in the project: 8 of 16 plan tasks were pure TDD on
|
||
the package, and almost every live-integration bug was fixable by editing
|
||
tested package code and re-running one deploy script.
|
||
|
||
### 2.2 Runtime topology
|
||
|
||
```
|
||
systemd user timer (5 min) ── curl POST ──▶ /api/v1/webhook/pm-ingest
|
||
│
|
||
┌── Flow 1 "PM Ingest" ────────────────────────▼────────┐
|
||
│ Webhook → PMIngest → TextOutput │
|
||
│ PMIngest = ingest.run_ingest(): │
|
||
│ flock lock → IMAP poll (UNKEYWORD $ProjektChecked) │
|
||
│ → own-mail/non-project filter → split projects │
|
||
│ → canonical URL → CRM dedup → per NEW project: │
|
||
│ POST /api/v1/run/<flow2> (x-api-key) ───────────┼──┐
|
||
│ → aggregate → ONE alert mail on failures │ │
|
||
│ → move trigger mail to Trash → summary │ │
|
||
└───────────────────────────────────────────────────────┘ │
|
||
┌── Flow 2 "PM Projekt bewerten" ──────────────────────────▼┐
|
||
│ TextInput → PMFetch → PMExtract → PMMatch → PMRules │
|
||
│ → PMCrm → PMNotify → TextOutput │
|
||
│ ctx dict flows through; status ∈ ok/failed/rejected/ │
|
||
│ created; run_stage() catches exceptions into failed and │
|
||
│ short-circuits later stages; PMNotify runs UNCONDITIONALLY│
|
||
│ (sends mail only on created, posts Langfuse trace always) │
|
||
└───────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
Flow 1 is a single custom component rather than the spec sketch's three,
|
||
because IMAP connections and per-mail state cannot cross Langflow component
|
||
boundaries; `run_ingest()` is fully unit-tested instead. Flow 2 keeps the
|
||
six visible stages.
|
||
|
||
### 2.3 Module inventory
|
||
|
||
| Module | Responsibility |
|
||
|---|---|
|
||
| `rules.py` | Misc rules, half-up match calc, exact description markdown |
|
||
| `mailparse.py` | own-mail detection, project splitting (longest-anchor-title dedup by URL path), canonical URL |
|
||
| `espocrm.py` | CRM client: dedup searches, team lookup (never guessed), ensure-account/contact, unique naming, retries, `X-Status-Reason` surfacing |
|
||
| `llm.py` | vLLM calls (`response_format json_schema`, thinking disabled, no `max_tokens`), German prompts + JSON schemas, retry-on-empty, match completeness retry |
|
||
| `mailer.py` | IMAP (custom keyword, Trash detection, MOVE+fallback), SMTP (STARTTLS, Date/Message-ID) |
|
||
| `stages.py` | Flow-2 stage functions + ctx state machine + CRM verify read-back |
|
||
| `pm_trace.py` | one consolidated Langfuse trace per project via the ingestion API (never raises) |
|
||
| `ingest.py` | Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash |
|
||
| `deploy/` | `deploy_files.sh` (rsync + safe restart), `build_flows.py` (variables, API key, programmatic flow construction), `setup_langfuse.py`, systemd units, webhook trigger |
|
||
| `tests/` | 51 unit tests + 5 e2e helper scripts |
|
||
|
||
### 2.4 Trust boundaries and secrets
|
||
|
||
IMAP/SMTP password and EspoCRM API key live in Langflow global variables
|
||
(encrypted in Postgres) and chmod-600 files under `~/.config/projekt-matching/`;
|
||
they never entered git (verified across all 31 commits in the final review).
|
||
Langfuse keys follow the pod script's pre-existing convention (tracked, local
|
||
loopback services only — an accepted, user-confirmed trade-off). The CV and
|
||
the CRM secrets file are owned by `tlg` and readable by `lwc` via
|
||
`setfacl -m u:lwc:r`.
|
||
|
||
## 3. Key design decisions and their rationale
|
||
|
||
1. **Canonical-URL dedup** (not name-based): deterministic, immune to title
|
||
edits; the same job at two agencies intentionally stays two opportunities.
|
||
2. **Delete-mail-always + alert** (revised mid-design from
|
||
"retry until success"): deterministic failures would otherwise loop every
|
||
5 minutes forever. The alert transfers responsibility to the human;
|
||
in-run HTTP retries (2–3 attempts) absorb transient blips.
|
||
3. **Deterministic scoring layer**: the LLM only classifies and rates;
|
||
Python computes. This survived the model's quality variance — when
|
||
ratings flickered, the numbers stayed honest and auditable.
|
||
4. **One custom Langfuse trace per project** (via the ingestion REST API, not
|
||
the SDK): stable contract, independent of Langflow's own auto-tracing,
|
||
filterable by outcome tags, and the object your annotation queue feeds on.
|
||
Posted *last* in `stage_notify`, unconditionally — even for failures.
|
||
5. **Programmatic flow construction** (`build_flows.py` assembles node/edge
|
||
JSON via `POST /api/v1/custom_component` templates): flows are
|
||
reproducible from git; the Langflow UI is never the source of truth.
|
||
6. **Everything idempotent**: variable upserts, flow delete-and-recreate,
|
||
API-key delete-and-recreate, file rsync — any deploy step can be re-run.
|
||
|
||
## 4. Experiences — what the plan got wrong and what we learned
|
||
|
||
The plan deliberately marked three "known unknowns" with verify-and-adapt
|
||
steps; live integration surfaced several more. Each item below is baked into
|
||
the code and worth knowing before touching the system.
|
||
|
||
### 4.1 vLLM / Qwen3.5 structured output
|
||
|
||
- **`guided_json` is silently ignored** by vLLM 0.22.1 — HTTP 200,
|
||
unconstrained prose output. The plan's guided_json-primary +
|
||
400-triggered-fallback design could therefore never fall back. Fixed by
|
||
using **`response_format {"type": "json_schema"}`** as the sole mechanism
|
||
(verified enforced: exact schema keys returned).
|
||
- **Reasoning + schema enforcement degenerate**: with the qwen3 reasoning
|
||
parser active, `json_schema` decoding sent the model into thinking loops
|
||
that consumed the full 63k context (`finish_reason: length`, empty
|
||
content). Root-cause fix: `chat_template_kwargs {"enable_thinking": false}`
|
||
for structured calls — extraction dropped from minutes/never to ~35 s.
|
||
A test locks this parameter in place.
|
||
- **Empty completions under load** still occur occasionally (shared GPU,
|
||
concurrent runs): `chat_json` retries empty/unparseable content up to 3×
|
||
with backoff; non-200 fails fast.
|
||
- The old belief "set max_tokens ≥ 1024 for reasoning models" evolved into
|
||
**"set no max_tokens at all** and disable thinking where you enforce
|
||
schemas".
|
||
|
||
### 4.2 Langflow 1.10 internals
|
||
|
||
- **`Component.ctx` is a reserved property** — a `DataInput(name="ctx")`
|
||
silently shadows the wired edge payload. All data inputs are named
|
||
`ctx_in`.
|
||
- **The custom-component loader resolves imports statically** — in-file
|
||
`sys.path.insert()` hacks don't work. First fix (a `.pth` file in the
|
||
container venv) turned out non-durable because the systemd `--new` unit
|
||
**recreates the container on every restart**, wiping ephemeral writes. The
|
||
durable fix bakes `PYTHONPATH=/app/langflow` into the container definition
|
||
in `create_pod_langflow.sh`.
|
||
- **Never `podman restart` a systemd-managed `--rm/--new` container** — it
|
||
can end up removed with the unit inactive. Always
|
||
`systemctl --user restart container-langflow_ctr.service`
|
||
(`deploy_files.sh` does this and waits for health).
|
||
- Webhook lives in the `input_output` category of `GET /api/v1/all` (not
|
||
`data`); `Data` outputs wire as type `JSON`; edge handles use the
|
||
`œ`-escaped JSON encoding. `build_flows.py` derives types from live
|
||
templates instead of hardcoding — the plan's explicit "verify against a
|
||
dumped existing flow first" step paid off.
|
||
- `POST /api/v1/api_key` accumulates same-named keys; the builder now
|
||
deletes existing `projekt-matching` keys before minting (list shape:
|
||
`{"api_keys": [...]}`).
|
||
- **`podman logs langflow_ctr` is useless** (a structlog recursion loop
|
||
floods it); the monitor/builds API is the working observability path.
|
||
- Auth split (from earlier experiments, confirmed): flow CRUD/variables need
|
||
the Bearer JWT from `/api/v1/auto_login`; `run` and webhook need
|
||
`x-api-key`; responses are gzip (`curl --compressed`).
|
||
- Global variables with `load_from_db=True` resolve **at run time** — the
|
||
Gate-3 recipient flip needed no restart.
|
||
|
||
### 4.3 Langfuse 3.195 (OSS)
|
||
|
||
- The **Organization Management API is EE-gated** (routes exist, return 403).
|
||
`setup_langfuse.py` falls back to a NextAuth session + tRPC calls to create
|
||
org/project/keys — fragile by nature, acceptable for one-shot setup, kept
|
||
behind an admin-API-first attempt.
|
||
- Score configs can be created via the public API with project keys;
|
||
**annotation queues are UI-only** (manual step, done).
|
||
- Score-config creation is not idempotent (archive-only API) — documented in
|
||
the script.
|
||
|
||
### 4.4 E-mail
|
||
|
||
- Outgoing `EmailMessage`s **need explicit `Date` and `Message-ID` headers**;
|
||
without them clients display 01/01/1970 (user-reported, fixed, and the fix
|
||
had to wait for a container restart — a long-lived process keeps the old
|
||
module in memory).
|
||
- The mail server (Dovecot) supports custom IMAP keywords (`PERMANENTFLAGS \*`)
|
||
and UID MOVE; the Trash folder is literally `Trash`. The `$ProjektChecked`
|
||
keyword implements "inspect non-project mail exactly once, touch nothing".
|
||
- A malformed FETCH must not kill the poll loop: hardened so an unreadable
|
||
mail is flagged and skipped (`summary.unreadable`), the connection is
|
||
closed in `finally` — otherwise one poison mail would silently stall the
|
||
pipeline every 5 minutes with no alert.
|
||
|
||
### 4.5 Prompt engineering for a 9B model
|
||
|
||
The extraction/matching prompts embed the skill's German calibration rules
|
||
verbatim (strictness: PoC ≠ production, "mehrjährig" literal, "wie z. B."
|
||
alternatives count). Two tuning iterations were needed against the reference
|
||
project (skill result: 86 % Must):
|
||
|
||
- **Signal words must bind to their own line only** — trailing unmarked lines
|
||
after a "von Vorteil" line were bleeding into Nice; as Must the reference
|
||
computes 18✅+3❌ of 21 = 85.71 → 86, exactly reproducing the skill.
|
||
- …with an **explicit exception for a literal "Nice-to-haves" heading**
|
||
(heading precedence stays above line-level signals) — added after review
|
||
caught the over-generalization.
|
||
- Residual variance: Must-match stable (86 in all post-fix runs); Nice-match
|
||
flips a few points run-to-run; Misc rows occasionally missing. The
|
||
deterministic layer plus the >85 gate make this tolerable; the annotation
|
||
queue exists to watch it.
|
||
|
||
### 4.6 Process experiences
|
||
|
||
- **Subagent-driven development with per-task adversarial review** caught
|
||
real defects at every altitude: committed build artifacts, undeclared
|
||
dependencies, a read-back that verified presence instead of equality,
|
||
alert-send blocking the trash guarantee, the prompt over-generalization.
|
||
Two verification-only gates were audited as *evidence* (timestamps, IDs,
|
||
counts, alternative explanations) rather than trusted — one audit forced a
|
||
trace-level proof that an unexpected notification during Gate 2 was
|
||
genuine production traffic (it was: a real project, Must 100 %, evaluated
|
||
and filed while we were testing).
|
||
- **Live systems interleave with tests.** Real project mails arrived and
|
||
were processed during every e2e window. Distinguishing test artifacts from
|
||
production artifacts by canonical-URL tokens — and writing that discipline
|
||
into the helper scripts — was essential.
|
||
- The **progress ledger** (`.superpowers/sdd/progress.md`, git-ignored)
|
||
carried environment facts between tasks; most of its content graduated
|
||
into this document and the memory notes.
|
||
|
||
## 5. Verification summary
|
||
|
||
- **51 unit tests** (pure package, no network); TDD throughout.
|
||
- **Component gates:** Flow 2 driven directly via API — consider project
|
||
stable at 86 % across repeated runs with full CRM field verification
|
||
(team, agency linking, contact, description byte-format); reject project
|
||
at 0 %, no writes.
|
||
- **Gate 1 (e2e consider):** re-sent original example-1 mail (5 projects) →
|
||
timer → project 4 created (86 %), notification with valid Date, trigger
|
||
mail in Trash, 5 traces (1 created / 4 rejected), zero unwanted entries.
|
||
- **Gate 2 (e2e reject):** example-2 mail → full table in trace, Must 0 %,
|
||
no CRM entry, no mail, mail in Trash.
|
||
- **Dedup:** re-sent Gate-1 mail → ingest summary
|
||
`{projects: 5, created: 0, rejected: 4, duplicate: 1}`, notification count
|
||
unchanged, no ` (2)` entry.
|
||
- **Gate 3 (production recipient):** variables flipped live (no restart),
|
||
opportunity re-created at 86 %, no mail in the design-phase inbox, SMTP
|
||
acceptance proven via trace error-absence, **receipt at
|
||
Thomas.Langer@destengs.com confirmed by Thomas**.
|
||
|
||
## 6. Known limitations & future work
|
||
|
||
- freelancermap-only detection (extension point: `mailparse.PROJECT_URL_RE`
|
||
+ splitting logic).
|
||
- Login-walled postings fail with an alert by design (no snippet fallback).
|
||
- Created-but-notification-failed edge case is recorded in the trace, not
|
||
re-sent (spec-accepted).
|
||
- Worst-case in-run LLM latency (nested retries) can exceed the 1800 s
|
||
dispatch timeout — the ingest side then alerts while Flow 2 may still
|
||
finish server-side; dedup prevents damage. Cap the budgets if it ever
|
||
bites.
|
||
- `parse_run_result` anchors on "any embedded JSON with a `status` key" —
|
||
robust today, but a Langflow upgrade that serializes intermediate ctx
|
||
objects into the run response would need a stricter anchor.
|
||
- niceMatch variance / Misc extraction variance: monitor via the
|
||
`projekt-matching-review` annotation queue; if quality drifts, tuning
|
||
lives in `llm.py`'s prompts and is regression-tested by
|
||
`tests/e2e/run_flow2.py` against the two reference projects.
|
||
|
||
## 7. Restore from a fresh clone (runbook)
|
||
|
||
1. `bash create_pod_langflow.sh` — pod, DBs, Langflow, Langfuse (bootstrap
|
||
admin + tracing keys baked in).
|
||
2. Ensure ACLs: `setfacl -m u:lwc:r` on the CV and
|
||
`.secrets/espocrm-api.md` (as `tlg`); create
|
||
`~/.config/projekt-matching/secrets.env` (0600) with `PM_IMAP_PASSWORD`.
|
||
3. `projekt-matching/deploy/deploy_files.sh` — package + vorgaben into the
|
||
data dir.
|
||
4. `set -a; source projekt-matching/deploy/secrets.local.env; set +a` then
|
||
`.venv/bin/python projekt-matching/deploy/build_flows.py` — variables,
|
||
API key, both flows, `~/.config/projekt-matching/env`.
|
||
5. `.venv/bin/python projekt-matching/deploy/setup_langfuse.py` — org,
|
||
project, keys, score configs (annotation queue: manual UI step); wire the
|
||
printed keys into the pod script and rerun it.
|
||
6. `cp projekt-matching/deploy/projekt-matching.{service,timer} ~/.config/systemd/user/`
|
||
then `systemctl --user daemon-reload && systemctl --user enable --now projekt-matching.timer`.
|
||
7. Optional clean start: `.venv/bin/python projekt-matching/tests/e2e/flag_inbox.py`
|
||
to baseline the current INBOX before the first tick.
|