🤖 Automation Workflow Design (n8n + Python)

This document outlines the architecture for the TradingTwins Lead Engine automation, combining n8n (Trigger/Flow Control) and Python (Business Logic/Sync).

1. High-Level Architecture

We use a Hybrid Approach:

n8n: Handles connectivity (IMAP, Slack, CRM) and triggers.
Python Lead Engine: Handles data processing, parsing, and syncing with Company Explorer.

sequenceDiagram
    participant Mail as 📧 Email Server (IMAP)
    participant n8n as ⚡ n8n Workflow
    participant API as 🐍 Lead Engine API
    participant DB as 🗄️ SQLite (leads.db)
    participant CE as 🏢 Company Explorer

    Mail->>n8n: New Email (Subject: "Lead...")
    n8n->>n8n: Extract Body & Subject
    n8n->>API: POST /api/ingest { "body": "...", "subject": "..." }
    
    rect rgb(240, 248, 255)
        Note right of API: Python Async Process
        API->>DB: Save Lead (Status: 'new')
        API-->>n8n: 200 OK (Lead Queued)
        
        API->>CE: Check Existence / Create
        CE-->>API: Company ID
        API->>CE: Trigger Discovery
        API->>DB: Update Status: 'synced'
    end
    
    API->>n8n: (Optional) Webhook "Sync Complete"
    n8n->>Slack: Notify Sales "New Lead Synced!"

2. n8n Workflow Specification

Trigger: IMAP Read Node

Filter: Subject contains "TradingTwins" OR Sender is "notification@tradingtwins.com"
Interval: Every 5 minutes (or 1 min).

Action: HTTP Request Node

Method: POST
URL: http://lead-engine:8000/api/ingest (Internal Docker Network)

Body: JSON

{
  "source": "tradingtwins",
  "subject": "New Lead: Musterfirma GmbH",
  "raw_body": "..."
}

3. Python Lead Engine Changes

To support this, the current Streamlit-only application needs a FastAPI backend (or similar) running alongside or instead of the Streamlit loop for API access.

Requirements:

Split app.py: Separate UI (Streamlit) from Logic/API.
New api.py: FastAPI instance exposing POST /api/ingest.
Refactor enrich.py: Ensure run_sync can be called programmatically for a specific lead ID.

Proposed File Structure:

lead-engine/
├── api.py          # FastAPI (New)
├── ui.py           # Streamlit (Renamed from app.py)
├── core/
│   ├── db.py       # Shared DB access
│   ├── enrich.py   # Sync Logic
│   └── parser.py   # Parsing Logic
├── Dockerfile      # Needs to run both (e.g. via supervisord or entrypoint script)

4. Why this approach?

Reliability: n8n is better at keeping IMAP connections alive than a custom Python loop.
Scalability: We can easily add other lead sources (e.g., Typeform, LinkedIn) just by adding another n8n trigger pointing to the same Python API.
Focus: Python code focuses purely on "What to do with the data", not "How to get the data".

2.9 KiB Raw Blame History