Compare commits
3 Commits
49aed3b243
...
58b64c6d5f
| Author | SHA256 | Date | |
|---|---|---|---|
| 58b64c6d5f | |||
| 36eac66abc | |||
| f40a5bb0f8 |
@@ -5,7 +5,8 @@
|
||||
as `72a3b17`. Approved spec:
|
||||
`docs/superpowers/specs/2026-07-09-langflow-projekt-matching-design.md`;
|
||||
implementation plan (with deviations preamble):
|
||||
`docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md`.*
|
||||
`docs/superpowers/plans/2026-07-09-langflow-projekt-matching.md`.
|
||||
Updated 2026-07-14 after the poison-mail incident (§4.6, commit `36eac66`).*
|
||||
|
||||
---
|
||||
|
||||
@@ -39,7 +40,7 @@ constraints:
|
||||
### 2.1 The central decision: logic in a package, Langflow as the runtime
|
||||
|
||||
All behavior lives in a plain-Python package **`projektmatch`** that is
|
||||
unit-tested on the host (51 pytest tests) and copied unchanged into the
|
||||
unit-tested on the host (63 pytest tests) and copied unchanged into the
|
||||
Langflow container. The seven Langflow custom components are ~30-line
|
||||
wrappers that read configuration from Langflow global variables and call one
|
||||
package function each. This kept the visual pipeline Langflow promises while
|
||||
@@ -85,13 +86,13 @@ six visible stages.
|
||||
| `rules.py` | Misc rules, half-up match calc, exact description markdown |
|
||||
| `mailparse.py` | own-mail detection, project splitting (longest-anchor-title dedup by URL path), canonical URL |
|
||||
| `espocrm.py` | CRM client: dedup searches, team lookup (never guessed), ensure-account/contact, unique naming, retries, `X-Status-Reason` surfacing |
|
||||
| `llm.py` | vLLM calls (`response_format json_schema`, thinking disabled, no `max_tokens`), German prompts + JSON schemas, retry-on-empty, match completeness retry |
|
||||
| `mailer.py` | IMAP (custom keyword, Trash detection, MOVE+fallback), SMTP (STARTTLS, Date/Message-ID) |
|
||||
| `stages.py` | Flow-2 stage functions + ctx state machine + CRM verify read-back |
|
||||
| `llm.py` | vLLM calls (`response_format json_schema`, thinking disabled, `max_tokens` 8192), German prompts + JSON schemas (requirements capped at 40), retry-on-empty, match completeness retry |
|
||||
| `mailer.py` | IMAP (custom keyword, Trash detection, MOVE+fallback, `ensure_alive()` reconnect), SMTP (STARTTLS, Date/Message-ID, Sent copy) |
|
||||
| `stages.py` | Flow-2 stage functions + ctx state machine + CRM verify read-back; `page_text()` (project card minus tag cloud, quote sanitization) |
|
||||
| `pm_trace.py` | one consolidated Langfuse trace per project via the ingestion API (never raises) |
|
||||
| `ingest.py` | Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash |
|
||||
| `ingest.py` | Flow-1 orchestration: lock, loop, dispatch, alert aggregation, trash with flag-checked fallback |
|
||||
| `deploy/` | `deploy_files.sh` (rsync + safe restart), `build_flows.py` (variables, API key, programmatic flow construction), `setup_langfuse.py`, systemd units, webhook trigger |
|
||||
| `tests/` | 51 unit tests + 5 e2e helper scripts |
|
||||
| `tests/` | 63 unit tests + 5 e2e helper scripts |
|
||||
|
||||
### 2.4 Trust boundaries and secrets
|
||||
|
||||
@@ -146,9 +147,20 @@ the code and worth knowing before touching the system.
|
||||
- **Empty completions under load** still occur occasionally (shared GPU,
|
||||
concurrent runs): `chat_json` retries empty/unparseable content up to 3×
|
||||
with backoff; non-200 fails fast.
|
||||
- The old belief "set max_tokens ≥ 1024 for reasoning models" evolved into
|
||||
**"set no max_tokens at all** and disable thinking where you enforce
|
||||
schemas".
|
||||
- **A straight `"` in the INPUT text breaks schema-guided decoding** (found
|
||||
2026-07-14, §4.6): the model copies the quote verbatim into a JSON string
|
||||
value, the grammar reads it as end-of-string, the model cannot continue
|
||||
its sentence — and escapes into an **infinite whitespace loop** (whitespace
|
||||
is always grammar-legal) until the token limit, `finish_reason: length`.
|
||||
vLLM's request-level `structured_outputs.disable_any_whitespace` does NOT
|
||||
help: the loop simply moves inside a string literal, where spaces are
|
||||
legal. The robust fix is input-side: `stages.page_text()` maps all quote
|
||||
characters (`„ “ ” " « » ‹ ›`) to `'` before prompting.
|
||||
- The old belief "set max_tokens ≥ 1024 for reasoning models" first evolved
|
||||
into "set no max_tokens at all and disable thinking where you enforce
|
||||
schemas" — and then, after the whitespace-loop incident, into
|
||||
**"cap max_tokens at 8192"**: every valid answer fits comfortably, and a
|
||||
degenerate run now burns ~100 s instead of ~10 min before the retry.
|
||||
|
||||
### 4.2 Langflow 1.10 internals
|
||||
|
||||
@@ -205,6 +217,12 @@ the code and worth knowing before touching the system.
|
||||
mail is flagged and skipped (`summary.unreadable`), the connection is
|
||||
closed in `finally` — otherwise one poison mail would silently stall the
|
||||
pipeline every 5 minutes with no alert.
|
||||
- **The IMAP server drops idle connections** during long Flow-2 dispatches
|
||||
(30+ min in the degenerate case, §4.6). `run_ingest` therefore calls
|
||||
`MailBox.ensure_alive()` (NOOP probe + transparent reconnect) before the
|
||||
per-mail finalization, and a failing `move_to_trash` falls back to
|
||||
`flag_checked` (`summary.trashFailed`) so the run continues — the
|
||||
terminal-state rule survives a dead connection.
|
||||
|
||||
### 4.5 Prompt engineering for a 9B model
|
||||
|
||||
@@ -224,7 +242,41 @@ project (skill result: 86 % Must):
|
||||
deterministic layer plus the >85 gate make this tolerable; the annotation
|
||||
queue exists to watch it.
|
||||
|
||||
### 4.6 Process experiences
|
||||
### 4.6 Post-launch incident: the poison-mail loop (2026-07-13/14)
|
||||
|
||||
Four days after go-live the pipeline stalled for ~26 hours on one mail,
|
||||
sending an identical `[Projekt-Match-Fehler]` alert every ~35 minutes. The
|
||||
failure chain — each link necessary, none sufficient alone — is instructive:
|
||||
|
||||
1. **Trigger:** one posting (`nproj/3022414`) contained
|
||||
`„ISTQB Certified Tester …"` — a typographic opening quote closed by a
|
||||
straight ASCII `"`. Copied verbatim into a JSON string by the model, the
|
||||
`"` ended the string for the guided-decoding grammar mid-sentence; the
|
||||
model escaped into the whitespace loop (§4.1) and burned the full context
|
||||
on every attempt (~10 min × 3 retries ≈ 35 min per cycle).
|
||||
2. **Timeout mismatch:** the ingest dispatch timeout (1800 s) fired before
|
||||
Flow 2 finished, producing a premature `Read timed out` failure.
|
||||
3. **Trap:** during the long wait the IMAP connection idled out server-side.
|
||||
The alert still went out (fresh SMTP connection), but `move_to_trash` on
|
||||
the dead connection raised and aborted `run_ingest` **before** the mail
|
||||
reached its terminal state — so the next tick reprocessed the same mail,
|
||||
forever. 66 mails queued up behind it.
|
||||
|
||||
Fix (commit `36eac66`, TDD, live-verified): quote sanitization + feeding only
|
||||
the project card minus the skill-tag cloud to the LLM (`page_text()`),
|
||||
`maxItems: 40` on the requirements schema, `max_tokens: 8192`, and the
|
||||
finalization hardening from §4.4. The poison extraction went from 25+
|
||||
consecutive failures to a clean 19-requirement result in 21 s; the backlog
|
||||
(67 mails) drained in 36 minutes with zero failure alerts and 9 new
|
||||
opportunities.
|
||||
|
||||
Lessons: **terminal-state guarantees must hold on every path, including
|
||||
"connection died mid-run"** — the guarantee is only as strong as its weakest
|
||||
finalization step; and **schema-guided decoding turns bad input characters
|
||||
into decoding pathologies**, so sanitize model input against the output
|
||||
grammar, not just for size.
|
||||
|
||||
### 4.7 Process experiences
|
||||
|
||||
- **Subagent-driven development with per-task adversarial review** caught
|
||||
real defects at every altitude: committed build artifacts, undeclared
|
||||
@@ -245,7 +297,8 @@ project (skill result: 86 % Must):
|
||||
|
||||
## 5. Verification summary
|
||||
|
||||
- **51 unit tests** (pure package, no network); TDD throughout.
|
||||
- **63 unit tests** (pure package, no network); TDD throughout (51 at
|
||||
go-live, +1 Sent-copy fix, +11 poison-mail fix).
|
||||
- **Component gates:** Flow 2 driven directly via API — consider project
|
||||
stable at 86 % across repeated runs with full CRM field verification
|
||||
(team, agency linking, contact, description byte-format); reject project
|
||||
@@ -270,10 +323,12 @@ project (skill result: 86 % Must):
|
||||
- Login-walled postings fail with an alert by design (no snippet fallback).
|
||||
- Created-but-notification-failed edge case is recorded in the trace, not
|
||||
re-sent (spec-accepted).
|
||||
- Worst-case in-run LLM latency (nested retries) can exceed the 1800 s
|
||||
dispatch timeout — the ingest side then alerts while Flow 2 may still
|
||||
finish server-side; dedup prevents damage. Cap the budgets if it ever
|
||||
bites.
|
||||
- Worst-case in-run LLM latency is bounded since `36eac66`
|
||||
(`max_tokens: 8192` → a fully degenerate extraction costs ~100 s × 3
|
||||
retries ≈ 6 min, comfortably below the 1800 s dispatch timeout). Should
|
||||
the timeout ever fire anyway, the ingest side alerts while Flow 2 may
|
||||
still finish server-side; dedup prevents damage, and the finalization
|
||||
fallback (§4.4) keeps the trigger mail terminal.
|
||||
- `parse_run_result` anchors on "any embedded JSON with a `status` key" —
|
||||
robust today, but a Langflow upgrade that serializes intermediate ctx
|
||||
objects into the run response would need a stricter anchor.
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
# Projekt-Matching — User Manual
|
||||
|
||||
*The automated pipeline that turns freelancermap project e-mails into evaluated
|
||||
EspoCRM opportunities. Version of 2026-07-09 (all acceptance gates passed).*
|
||||
EspoCRM opportunities. Version of 2026-07-09 (all acceptance gates passed);
|
||||
updated 2026-07-14 (robustness fixes: alert-loop bug resolved, outgoing mails
|
||||
now also appear in the Sent folder).*
|
||||
|
||||
---
|
||||
|
||||
@@ -178,6 +180,7 @@ for the services, `projekt-matching.timer` for the polling). Nothing to do.
|
||||
| No mails processed, INBOX grows | `systemctl --user list-timers` — timer active? Langflow healthy (`curl …/health`)? Restart per section 5. Mails are never lost: the next successful tick processes the backlog. |
|
||||
| Alert mail: `Abruf fehlgeschlagen` / `Seitentext zu kurz` | freelancermap page unreachable or behind a login wall. Evaluate that project manually via the URL in the alert. |
|
||||
| Alert mail: `vLLM HTTP …` / `LLM lieferte kein JSON` / `Read timed out` | the local AI was down or overloaded. Occasional single failures are absorbed by built-in retries; if alerts pile up, check the vLLM service on port 8081. |
|
||||
| The **same** alert arrives again and again for the same project | this was a bug (fixed 2026-07-14): one poisonous posting could block the pipeline and repeat its alert every ~35 min. A failed project is now reliably final — if you ever see identical repeating alerts again, restart Langflow (section 5) and report it. |
|
||||
| Alert mail: `Team … nicht gefunden` | the CRM user `cowork-api` lost read access to Teams, or a team was renamed. Fix in EspoCRM admin. |
|
||||
| A project you expected is missing | check Langfuse: if its trace says `rejected`, the Must-match was ≤ 85 % — the table shows why. If there is no trace at all, check the alert mails. |
|
||||
| Duplicate opportunity suspected | duplicates are keyed on the canonical URL (`cProjektlink`). The same project posted by two different agencies has two different URLs and is intentionally kept as two opportunities. |
|
||||
|
||||
@@ -119,6 +119,10 @@ def run_ingest(cfg, mailbox=None, dispatch=None, espo=None, resolve=None):
|
||||
or f"unbekannter Status: {result.get('status')!r}")
|
||||
results.append(result)
|
||||
summary[result["status"]] += 1
|
||||
try:
|
||||
box.ensure_alive() # Flow-2-Dispatches können >30 min dauern — Server-Idle-Timeout
|
||||
except Exception: # noqa: BLE001
|
||||
pass # Alert/Trash unten haben eigene Fallbacks
|
||||
failed = [r for r in results if r["status"] == "failed"]
|
||||
if failed:
|
||||
lines = "\n\n".join(
|
||||
@@ -134,7 +138,14 @@ def run_ingest(cfg, mailbox=None, dispatch=None, espo=None, resolve=None):
|
||||
"werden:\n\n" + lines + "\n")
|
||||
except Exception: # noqa: BLE001
|
||||
summary["alertFailed"] = summary.get("alertFailed", 0) + 1
|
||||
box.move_to_trash(uid)
|
||||
try:
|
||||
box.move_to_trash(uid)
|
||||
except Exception: # noqa: BLE001
|
||||
summary["trashFailed"] = summary.get("trashFailed", 0) + 1
|
||||
try:
|
||||
box.flag_checked(uid) # nie erneut verarbeiten — Terminal-Zustand
|
||||
except Exception: # noqa: BLE001
|
||||
pass # nächster Lauf räumt auf
|
||||
return summary
|
||||
finally:
|
||||
box.close()
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
"""vLLM (OpenAI-compatible) calls with response_format json_schema + German prompts.
|
||||
|
||||
max_tokens stays UNSET so nothing can truncate the final answer (65k
|
||||
context window). Thinking is DISABLED via chat_template_kwargs: with the
|
||||
max_tokens is capped at 8192: every valid answer fits comfortably, and an
|
||||
uncapped run lets a degenerate generation grind for ~10 min before failing
|
||||
(observed live: 100+-requirement JSON, broken at ~char 1725, 3 retries =
|
||||
~35 min per cycle). Thinking is DISABLED via chat_template_kwargs: with the
|
||||
vLLM qwen3 reasoning parser active, json_schema guided decoding plus
|
||||
thinking degenerates — the run burns the entire context window
|
||||
(finish_reason "length", ~63k completion tokens) and returns empty or
|
||||
@@ -34,6 +36,7 @@ EXTRACT_SCHEMA = {
|
||||
"contactPerson": {"type": ["string", "null"]},
|
||||
"requirements": {
|
||||
"type": "array",
|
||||
"maxItems": 40,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
@@ -139,6 +142,7 @@ z. B. vLLM, TGI, Triton" ab).
|
||||
|
||||
def _post(base, model, messages, schema):
|
||||
body = {"model": model, "messages": messages, "temperature": 0.1,
|
||||
"max_tokens": 8192,
|
||||
"chat_template_kwargs": {"enable_thinking": False},
|
||||
"response_format": {"type": "json_schema", "json_schema": {
|
||||
"name": "out", "schema": schema}}}
|
||||
|
||||
@@ -16,12 +16,31 @@ KEYWORD = "$ProjektChecked"
|
||||
|
||||
class MailBox:
|
||||
def __init__(self, user, password, conn=None):
|
||||
self.conn = conn or imaplib.IMAP4_SSL(IMAP_HOST, IMAP_PORT)
|
||||
if conn is None:
|
||||
self.conn.login(user, password)
|
||||
self.user, self.password = user, password
|
||||
self.conn = conn or self._connect()
|
||||
self.conn.select("INBOX")
|
||||
self._trash = None
|
||||
|
||||
def _connect(self):
|
||||
conn = imaplib.IMAP4_SSL(IMAP_HOST, IMAP_PORT)
|
||||
conn.login(self.user, self.password)
|
||||
return conn
|
||||
|
||||
def ensure_alive(self):
|
||||
"""Reconnect if the server dropped the connection (idle timeout)."""
|
||||
try:
|
||||
typ, _ = self.conn.noop()
|
||||
if typ == "OK":
|
||||
return
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
try:
|
||||
self.conn.logout()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
self.conn = self._connect()
|
||||
self.conn.select("INBOX")
|
||||
|
||||
def unchecked_uids(self):
|
||||
typ, data = self.conn.uid("SEARCH", None, f"UNKEYWORD {KEYWORD}")
|
||||
if typ != "OK" or not data or not data[0]:
|
||||
@@ -71,6 +90,39 @@ class MailBox:
|
||||
pass
|
||||
|
||||
|
||||
def _sent_folder(conn):
|
||||
"""Find the server's Sent folder via special-use flag, fallback names."""
|
||||
typ, data = conn.list()
|
||||
candidates = []
|
||||
for raw in data or []:
|
||||
line = raw.decode() if isinstance(raw, bytes) else str(raw)
|
||||
name = line.rsplit(" ", 1)[-1].strip('"')
|
||||
if "\\Sent" in line.split(")")[0]:
|
||||
return name
|
||||
candidates.append(name)
|
||||
for cand in ("Sent", "INBOX.Sent"):
|
||||
if cand in candidates:
|
||||
return cand
|
||||
return "Sent"
|
||||
|
||||
|
||||
def append_sent(user, password, msg, conn=None):
|
||||
"""Store a copy of an outgoing message in the Sent folder (best effort)."""
|
||||
own = conn is None
|
||||
conn = conn or imaplib.IMAP4_SSL(IMAP_HOST, IMAP_PORT)
|
||||
try:
|
||||
if own:
|
||||
conn.login(user, password)
|
||||
folder = _sent_folder(conn)
|
||||
conn.append(folder, "(\\Seen)", None, msg.as_bytes())
|
||||
finally:
|
||||
if own:
|
||||
try:
|
||||
conn.logout()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def send_mail(user, password, to, subject, body):
|
||||
msg = EmailMessage()
|
||||
msg["From"], msg["To"], msg["Subject"] = user, to, subject
|
||||
@@ -81,3 +133,7 @@ def send_mail(user, password, to, subject, body):
|
||||
server.starttls(context=ssl.create_default_context())
|
||||
server.login(user, password)
|
||||
server.send_message(msg)
|
||||
try:
|
||||
append_sent(user, password, msg)
|
||||
except Exception: # noqa: BLE001 — copy is best effort; the mail went out
|
||||
pass
|
||||
|
||||
@@ -20,6 +20,7 @@ TEAM_BY_OFFER = {"Projekt": "DesTEngS",
|
||||
"Arbeitnehmer-Angebot": "Arbeitnehmer",
|
||||
"ANÜ": "ANÜ"}
|
||||
MIN_PAGE_CHARS = 200
|
||||
QUOTES = str.maketrans({c: "'" for c in "„“”\"«»‹›"})
|
||||
|
||||
|
||||
def run_stage(name, fn, ctx, cfg, **kw):
|
||||
@@ -32,6 +33,25 @@ def run_stage(name, fn, ctx, cfg, **kw):
|
||||
return ctx
|
||||
|
||||
|
||||
def page_text(html):
|
||||
"""Project text for the LLM: header + description, WITHOUT the skill-tag
|
||||
cloud (div.project-body-badges, ~100 entries on some pages — blows the
|
||||
requirements list up until the model emits broken JSON) and without
|
||||
site chrome. Falls back to whole-page text if div.content is missing."""
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
for tag in soup(["script", "style", "noscript"]):
|
||||
tag.decompose()
|
||||
root = soup.find("div", class_="content") or soup
|
||||
for tag in root.find_all(class_="project-body-badges"):
|
||||
tag.decompose()
|
||||
# Anführungszeichen -> Apostroph: ein wörtlich übernommenes " beendet den
|
||||
# JSON-String vorzeitig und treibt guided decoding in eine Whitespace-
|
||||
# Endlosschleife (finish_reason "length", live verifiziert an nproj/3022414)
|
||||
return "\n".join(line for line in
|
||||
root.get_text("\n", strip=True).splitlines()
|
||||
if line.strip()).translate(QUOTES)
|
||||
|
||||
|
||||
def stage_fetch(ctx, cfg):
|
||||
last = None
|
||||
for attempt in range(3):
|
||||
@@ -39,12 +59,7 @@ def stage_fetch(ctx, cfg):
|
||||
resp = requests.get(ctx["canonical"], timeout=30,
|
||||
headers={"User-Agent": BROWSER_UA})
|
||||
resp.raise_for_status()
|
||||
soup = BeautifulSoup(resp.text, "html.parser")
|
||||
for tag in soup(["script", "style", "noscript"]):
|
||||
tag.decompose()
|
||||
text = "\n".join(line for line in
|
||||
soup.get_text("\n", strip=True).splitlines()
|
||||
if line.strip())
|
||||
text = page_text(resp.text)
|
||||
if len(text) < MIN_PAGE_CHARS:
|
||||
raise ValueError(
|
||||
f"Seitentext zu kurz ({len(text)} Zeichen) — "
|
||||
|
||||
@@ -156,6 +156,63 @@ def test_close_called_even_if_unchecked_uids_raises(tmp_path):
|
||||
box.close.assert_called_once()
|
||||
|
||||
|
||||
def test_ensure_alive_called_before_finalize(tmp_path):
|
||||
mails = {"1": mail("Mail", PROJECT_HTML)}
|
||||
summary, box, sent = run(tmp_path, mails,
|
||||
dispatch_result={"status": "failed",
|
||||
"error": "Read timed out"})
|
||||
names = [c[0] for c in box.method_calls]
|
||||
assert "ensure_alive" in names
|
||||
assert names.index("ensure_alive") < names.index("move_to_trash")
|
||||
box.move_to_trash.assert_called_once_with("1")
|
||||
assert len(sent) == 1
|
||||
|
||||
|
||||
def test_trash_failure_falls_back_to_flag_and_continues(tmp_path):
|
||||
mails = {"1": mail("Mail A", PROJECT_HTML),
|
||||
"2": mail("Mail B", PROJECT_HTML)}
|
||||
box = mock.Mock()
|
||||
box.unchecked_uids.return_value = ["1", "2"]
|
||||
box.fetch.side_effect = lambda uid: mails[uid]
|
||||
box.move_to_trash.side_effect = [OSError("socket error: EOF"), None]
|
||||
espo = mock.Mock()
|
||||
espo.find_opportunity_by_link.return_value = None
|
||||
with mock.patch("projektmatch.ingest.mailparse.bodies",
|
||||
side_effect=lambda m: (m["html"], "")), \
|
||||
mock.patch("projektmatch.ingest.mailer.send_mail", lambda *a: None):
|
||||
summary = ingest.run_ingest(
|
||||
cfg(tmp_path), mailbox=box, espo=espo,
|
||||
resolve=lambda url: "https://www.freelancermap.de/projekt/eins",
|
||||
dispatch=lambda item: {"status": "created"})
|
||||
assert summary["trashFailed"] == 1
|
||||
assert summary["created"] == 2
|
||||
box.flag_checked.assert_called_once_with("1")
|
||||
assert box.move_to_trash.call_count == 2
|
||||
|
||||
|
||||
def test_dead_connection_everywhere_still_finishes_run(tmp_path):
|
||||
mails = {"1": mail("Mail A", PROJECT_HTML),
|
||||
"2": mail("Mail B", PROJECT_HTML)}
|
||||
box = mock.Mock()
|
||||
box.unchecked_uids.return_value = ["1", "2"]
|
||||
box.fetch.side_effect = lambda uid: mails[uid]
|
||||
box.ensure_alive.side_effect = OSError("server unreachable")
|
||||
box.move_to_trash.side_effect = OSError("socket error: EOF")
|
||||
box.flag_checked.side_effect = OSError("socket error: EOF")
|
||||
espo = mock.Mock()
|
||||
espo.find_opportunity_by_link.return_value = None
|
||||
with mock.patch("projektmatch.ingest.mailparse.bodies",
|
||||
side_effect=lambda m: (m["html"], "")), \
|
||||
mock.patch("projektmatch.ingest.mailer.send_mail", lambda *a: None):
|
||||
summary = ingest.run_ingest(
|
||||
cfg(tmp_path), mailbox=box, espo=espo,
|
||||
resolve=lambda url: "https://www.freelancermap.de/projekt/eins",
|
||||
dispatch=lambda item: {"status": "created"})
|
||||
assert summary["trashFailed"] == 2
|
||||
assert summary["created"] == 2
|
||||
box.close.assert_called_once()
|
||||
|
||||
|
||||
def test_parse_run_result_finds_status_json():
|
||||
inner = json.dumps({"status": "created", "mustMatch": 90})
|
||||
payload = {"outputs": [{"outputs": [{"results": {"text": {
|
||||
|
||||
@@ -24,11 +24,16 @@ def test_chat_json_strips_think_and_parses():
|
||||
{"type": "object"})
|
||||
assert out == {"a": 1}
|
||||
body = p.call_args.kwargs["json"]
|
||||
assert "max_tokens" not in body
|
||||
assert body["max_tokens"] == 8192
|
||||
assert body["chat_template_kwargs"] == {"enable_thinking": False}
|
||||
assert body["response_format"]["json_schema"]["schema"] == {"type": "object"}
|
||||
|
||||
|
||||
def test_extract_schema_caps_requirements_list():
|
||||
reqs = llm.EXTRACT_SCHEMA["properties"]["requirements"]
|
||||
assert reqs["maxItems"] == 40
|
||||
|
||||
|
||||
def test_chat_json_strips_markdown_fences():
|
||||
content = "```json\n{\"a\": 2}\n```"
|
||||
with mock.patch("projektmatch.llm.requests.post", fake_post([content])):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import email
|
||||
from email.message import EmailMessage
|
||||
|
||||
import pytest
|
||||
from unittest import mock
|
||||
@@ -62,7 +63,8 @@ def test_fetch_raises_on_malformed_ok_response():
|
||||
|
||||
|
||||
def test_send_mail_starttls():
|
||||
with mock.patch("projektmatch.mailer.smtplib.SMTP") as smtp:
|
||||
with mock.patch("projektmatch.mailer.smtplib.SMTP") as smtp, \
|
||||
mock.patch("projektmatch.mailer.append_sent") as append_sent:
|
||||
server = smtp.return_value.__enter__.return_value
|
||||
mailer.send_mail("u@x", "pw", "to@x", "Subj", "Body")
|
||||
server.starttls.assert_called_once()
|
||||
@@ -71,3 +73,52 @@ def test_send_mail_starttls():
|
||||
assert msg["Subject"] == "Subj" and msg["To"] == "to@x"
|
||||
assert msg["Date"]
|
||||
assert msg["Message-ID"].endswith("@destengs.com>")
|
||||
append_sent.assert_called_once_with("u@x", "pw", msg)
|
||||
|
||||
|
||||
def test_send_mail_survives_append_failure():
|
||||
with mock.patch("projektmatch.mailer.smtplib.SMTP") as smtp, \
|
||||
mock.patch("projektmatch.mailer.append_sent",
|
||||
side_effect=OSError("imap down")):
|
||||
mailer.send_mail("u@x", "pw", "to@x", "Subj", "Body")
|
||||
|
||||
|
||||
def test_ensure_alive_keeps_healthy_connection():
|
||||
conn = fake_conn()
|
||||
conn.noop.return_value = ("OK", [b""])
|
||||
box = mailer.MailBox("u", "p", conn=conn)
|
||||
box.ensure_alive()
|
||||
assert box.conn is conn
|
||||
conn.noop.assert_called_once()
|
||||
|
||||
|
||||
def test_ensure_alive_reconnects_on_dead_connection():
|
||||
conn = fake_conn()
|
||||
conn.noop.side_effect = OSError("connection closed")
|
||||
box = mailer.MailBox("u", "p", conn=conn)
|
||||
fresh = fake_conn()
|
||||
with mock.patch("projektmatch.mailer.imaplib.IMAP4_SSL",
|
||||
return_value=fresh):
|
||||
box.ensure_alive()
|
||||
assert box.conn is fresh
|
||||
fresh.login.assert_called_once_with("u", "p")
|
||||
fresh.select.assert_called_once_with("INBOX")
|
||||
|
||||
|
||||
def test_append_sent_uses_special_use_folder():
|
||||
conn = mock.Mock()
|
||||
conn.list.return_value = ("OK", [
|
||||
b'(\\HasNoChildren) "." "INBOX"',
|
||||
b'(\\HasNoChildren \\Sent) "." "Sent"'])
|
||||
msg = EmailMessage()
|
||||
msg["Subject"] = "Subj"
|
||||
msg.set_content("Body")
|
||||
|
||||
mailer.append_sent("u@x", "pw", msg, conn=conn)
|
||||
|
||||
args = conn.append.call_args.args
|
||||
assert args[0] == "Sent"
|
||||
assert "\\Seen" in args[1]
|
||||
assert args[3] == msg.as_bytes()
|
||||
conn.login.assert_not_called()
|
||||
conn.logout.assert_not_called()
|
||||
|
||||
@@ -33,6 +33,68 @@ def ctx_after_match(ratings):
|
||||
return ctx
|
||||
|
||||
|
||||
PROJECT_PAGE = """
|
||||
<html><body>
|
||||
<nav>Login Registrieren</nav>
|
||||
<div class="content">
|
||||
<div class="project-header">
|
||||
<h1>ISTQB Testautomatisierung (m/w/d)</h1>
|
||||
<span>iSAX Consulting GmbH</span>
|
||||
<span>Start 8/2026</span>
|
||||
</div>
|
||||
<div class="project-body">
|
||||
<div class="project-body-badges badges-container">
|
||||
<a href="#">ISTQB</a><a href="#">Testsuite</a><a href="#">IBM Aix</a>
|
||||
</div>
|
||||
<div class="project-body-description">
|
||||
<p>Beschreibung: Muss-Anforderungen: 3 Jahre Testautomatisierung.
|
||||
Erstellung, Beratung und methodische Unterstuetzung der Projektteams
|
||||
bei Testkonzept und Teststrategie. Organisation der Testkoordination
|
||||
fuer einen uebergreifenden Test. Durchfuehrung von manuellen und
|
||||
automatisierten Tests sowie Last- und Performancetests.</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<footer>Impressum Datenschutz</footer>
|
||||
</body></html>
|
||||
"""
|
||||
|
||||
|
||||
def test_page_text_keeps_header_and_description_drops_tag_cloud():
|
||||
text = stages.page_text(PROJECT_PAGE)
|
||||
assert "ISTQB Testautomatisierung (m/w/d)" in text
|
||||
assert "Start 8/2026" in text
|
||||
assert "Muss-Anforderungen: 3 Jahre Testautomatisierung" in text
|
||||
assert "IBM Aix" not in text
|
||||
assert "Testsuite" not in text
|
||||
assert "Impressum" not in text
|
||||
|
||||
|
||||
def test_page_text_replaces_quotes_that_break_json_strings():
|
||||
html = ('<div class="content">Zertifizierung „ISTQB Certified Tester" '
|
||||
'sowie "Scrum" und “Kanban”.</div>')
|
||||
text = stages.page_text(html)
|
||||
assert '"' not in text
|
||||
assert "„" not in text and "“" not in text and "”" not in text
|
||||
assert "'ISTQB Certified Tester'" in text
|
||||
|
||||
|
||||
def test_page_text_falls_back_to_whole_page():
|
||||
html = "<html><body><p>Nur Text ohne content-Div.</p></body></html>"
|
||||
assert "Nur Text ohne content-Div." in stages.page_text(html)
|
||||
|
||||
|
||||
def test_stage_fetch_uses_project_section():
|
||||
resp = mock.Mock(status_code=200, text=PROJECT_PAGE)
|
||||
resp.raise_for_status = lambda: None
|
||||
ctx = {"canonical": "https://www.freelancermap.de/projekt/x", "title": "t"}
|
||||
with mock.patch("projektmatch.stages.requests.get", return_value=resp):
|
||||
out = stages.stage_fetch(dict(ctx), CFG)
|
||||
assert out["status"] == "ok"
|
||||
assert "IBM Aix" not in out["page_text"]
|
||||
assert "Muss-Anforderungen" in out["page_text"]
|
||||
|
||||
|
||||
def test_run_stage_catches_and_skips():
|
||||
def boom(ctx, cfg):
|
||||
raise ValueError("kaputt")
|
||||
|
||||
Reference in New Issue
Block a user