Add lead-engine source code to main repo

2026-01-31 17:25:01 +00:00
parent 8347d5c7ae
commit 8cbac74b2f
8 changed files with 736 additions and 0 deletions
--- a/lead-engine/AUTOMATION_DESIGN.md
+++ b/lead-engine/AUTOMATION_DESIGN.md
@@ -0,0 +1,80 @@
+# 🤖 Automation Workflow Design (n8n + Python)
+
+This document outlines the architecture for the **TradingTwins Lead Engine** automation, combining **n8n** (Trigger/Flow Control) and **Python** (Business Logic/Sync).
+
+## 1. High-Level Architecture
+
+We use a **Hybrid Approach**:
+*   **n8n:** Handles connectivity (IMAP, Slack, CRM) and triggers.
+*   **Python Lead Engine:** Handles data processing, parsing, and syncing with Company Explorer.
+
+```mermaid
+sequenceDiagram
+    participant Mail as 📧 Email Server (IMAP)
+    participant n8n as ⚡ n8n Workflow
+    participant API as 🐍 Lead Engine API
+    participant DB as 🗄️ SQLite (leads.db)
+    participant CE as 🏢 Company Explorer
+
+    Mail->>n8n: New Email (Subject: "Lead...")
+    n8n->>n8n: Extract Body & Subject
+    n8n->>API: POST /api/ingest { "body": "...", "subject": "..." }
+    
+    rect rgb(240, 248, 255)
+        Note right of API: Python Async Process
+        API->>DB: Save Lead (Status: 'new')
+        API-->>n8n: 200 OK (Lead Queued)
+        
+        API->>CE: Check Existence / Create
+        CE-->>API: Company ID
+        API->>CE: Trigger Discovery
+        API->>DB: Update Status: 'synced'
+    end
+    
+    API->>n8n: (Optional) Webhook "Sync Complete"
+    n8n->>Slack: Notify Sales "New Lead Synced!"
+```
+
+## 2. n8n Workflow Specification
+
+**Trigger:** `IMAP Read` Node
+*   **Filter:** Subject contains "TradingTwins" OR Sender is "notification@tradingtwins.com"
+*   **Interval:** Every 5 minutes (or 1 min).
+
+**Action:** `HTTP Request` Node
+*   **Method:** POST
+*   **URL:** `http://lead-engine:8000/api/ingest` (Internal Docker Network)
+*   **Body:** JSON
+    ```json
+    {
+      "source": "tradingtwins",
+      "subject": "New Lead: Musterfirma GmbH",
+      "raw_body": "..."
+    }
+    ```
+
+## 3. Python Lead Engine Changes
+
+To support this, the current Streamlit-only application needs a **FastAPI backend** (or similar) running alongside or instead of the Streamlit loop for API access.
+
+### Requirements:
+1.  **Split `app.py`:** Separate UI (Streamlit) from Logic/API.
+2.  **New `api.py`:** FastAPI instance exposing `POST /api/ingest`.
+3.  **Refactor `enrich.py`:** Ensure `run_sync` can be called programmatically for a specific lead ID.
+
+### Proposed File Structure:
+```
+lead-engine/
+├── api.py          # FastAPI (New)
+├── ui.py           # Streamlit (Renamed from app.py)
+├── core/
+│   ├── db.py       # Shared DB access
+│   ├── enrich.py   # Sync Logic
+│   └── parser.py   # Parsing Logic
+├── Dockerfile      # Needs to run both (e.g. via supervisord or entrypoint script)
+```
+
+## 4. Why this approach?
+*   **Reliability:** n8n is better at keeping IMAP connections alive than a custom Python loop.
+*   **Scalability:** We can easily add other lead sources (e.g., Typeform, LinkedIn) just by adding another n8n trigger pointing to the same Python API.
+*   **Focus:** Python code focuses purely on "What to do with the data", not "How to get the data".