79 lines
6.0 KiB
Markdown
79 lines
6.0 KiB
Markdown
# SuperOffice Connector README
|
|
|
|
## Overview
|
|
This directory contains Python scripts designed to integrate with the SuperOffice CRM API, primarily for data enrichment and lead generation automation.
|
|
|
|
## 🚀 Production Deployment (March 2026)
|
|
|
|
**Status:** ✅ Live & Operational
|
|
**Environment:** `online3` (Production)
|
|
**Tenant:** `Cust26720`
|
|
|
|
### 1. Architecture & Flow
|
|
1. **Trigger:** SuperOffice sends a webhook (`contact.created`, `contact.changed`, `person.created`, `person.changed`) to `https://floke-ai.duckdns.org/connector/webhook`.
|
|
2. **Reception:** `webhook_app.py` (FastAPI) receives the event, validates the `WEBHOOK_TOKEN`, and pushes a job to the SQLite queue (`connector_queue.db`).
|
|
3. **Processing:** `worker.py` (v1.9.1) polls the queue, filters for relevance, fetches details from SuperOffice, and calls the **Company Explorer** for AI analysis.
|
|
4. **Sync:** Results (Vertical, Summary, hyper-personalized Openers) are patched back into SuperOffice `UserDefinedFields`.
|
|
|
|
### 2. Operational Resilience (Lessons Learned)
|
|
|
|
#### 🛡️ A. Noise Reduction & Loop Prevention
|
|
* **Problem:** Every data update (`PATCH`) triggers a new `contact.changed` webhook, creating infinite loops (Ping-Pong effect).
|
|
* **Solution 1 (Whitelist):** Only changes to `name`, `urladdress`, `urls`, `orgnr`, or `userdef_id` (UDFs) trigger processing.
|
|
* **Solution 2 (Circuit Breaker):** The system explicitly ignores any events triggered by its own API User (**Associate ID 528**). This stops the echo effect immediately.
|
|
|
|
#### 👥 B. Multi-Tenant Filtering (Roboplanet vs. Wackler)
|
|
* **Challenge:** The SuperOffice tenant is shared with Wackler. We must only process Roboplanet accounts.
|
|
* **Solution:** **Hybrid Whitelist Filtering**. We maintain a list of Roboplanet Associate IDs and Shortnames (e.g., `485`, `528`, `RKAB`, `RCGO`) in `config.py`.
|
|
* **Execution:** The worker fetches contact details and skips any account whose owner is not in this whitelist. This is faster and more reliable than querying group memberships via the API (which often returns 500 errors).
|
|
|
|
#### 📊 C. Dashboard Persistence & Prioritization
|
|
* **Challenge:** Webhooks often only contain IDs, leaving the dashboard with "Entity C123".
|
|
* **Solution:** **Late Name Resolution**. The worker persists the resolved Company Name and Associate Shortname to the SQLite database as soon as it fetches them from SuperOffice.
|
|
* **Status Priority:** Success (`COMPLETED`) now "outshines" subsequent ignored echos (`SKIPPED`). Once an account is green, it stays green in the dashboard cluster for 15 minutes.
|
|
|
|
#### 🛡️ D. Webhook De-Duplication (Contact Creation)
|
|
* **Problem:** SuperOffice occasionally sends *multiple* `contact.created` webhooks for the *same* new contact within a very short timeframe. This leads to duplicate processing jobs for the same entity.
|
|
* **Solution:** A **de-duplication shield** has been implemented in `worker.py` (March 2026).
|
|
* Before extensive processing, the worker now checks `connector_queue.db` via `queue_manager.check_for_recent_duplicate()`.
|
|
* If a `contact.created` event for the same company name is already `PROCESSING` or has been `COMPLETED` within the last 5 minutes, the current job is immediately `SKIPPED` as a duplicate.
|
|
|
|
#### 💊 E. Job Stability (Poison Pill)
|
|
* **Problem:** API errors or transient issues could cause a job to enter an infinite retry loop, blocking the entire queue indefinitely.
|
|
* **Solution:** A **"poison pill" mechanism** was added to the queue manager.
|
|
* A `retry_count` is now tracked for every job.
|
|
* If a job fails and is retried, this counter is incremented.
|
|
* If the counter exceeds a maximum threshold (currently 5), the job is automatically moved to the `FAILED` state and will not be retried again. This ensures that problematic jobs are cleared and do not block new incoming events.
|
|
|
|
### 3. Advanced API Handling (Critical Fixes)
|
|
|
|
* **OData Pagination:** We implemented `odata.nextLink` support (Manuel Zierl's advice) to correctly handle large result sets (>1000 records).
|
|
* **Case Sensitivity:** We learned that SuperOffice API keys are case-sensitive in responses (e.g., `contactId` vs `ContactId`) depending on the endpoint. Our code now handles both.
|
|
* **Robust Authentication:** The API client was hardened to raise exceptions on authentication failures instead of returning `None`. This ensures that auth errors are properly caught and jobs are failed or retried correctly.
|
|
* **Auth Consistency:** Always use `load_dotenv(override=True)` to prevent stale environment variables (like expired tokens) from lingering in the shell process.
|
|
* **Identity Crisis:** The `Associate/Me` and `Associate/{id}` endpoints return 500 errors if the API user lacks a linked Person record. This is a known configuration blocker for automated mailing.
|
|
|
|
### 4. Docker Optimization
|
|
|
|
* **Multi-Stage builds:** Reduced build time from 8+ minutes to seconds (for code changes) by separating the build environment (compilers) from the runtime environment.
|
|
* **Size reduction:** Removed `build-essential` and Node.js runtimes from final images, drastically shrinking the footprint.
|
|
|
|
### 5. Tooling & Diagnosis
|
|
Located in `connector-superoffice/tools/`:
|
|
|
|
* `verify_latest_roboplanet.py`: The "Ocular Proof" script that finds the youngest target account.
|
|
* `check_filter_counts.py`: Compares API counts with UI counts.
|
|
* `find_latest_roboplanet_account.py`: Diagnostic tool for group filtering.
|
|
* `cleanup_test_data.py`: Safely removes all session-created test objects.
|
|
|
|
### 6. Open Todos
|
|
* [x] **List ID Mapping:** Identified Production IDs (1613-1637). Hardcoded in `config.py`.
|
|
* [ ] **Mailing Identity:** Resolve 500 error on `Associate/Me`. (Meeting scheduled for Monday).
|
|
* [x] **Docker Build:** Multi-stage builds implemented.
|
|
* [x] **Dashboard:** Names and Associate shortnames now visible.
|
|
|
|
---
|
|
|
|
## Authentication (Legacy)
|
|
Authentication is handled via the `AuthHandler` class, which uses a refresh token flow to obtain access tokens. Ensure that the `.env` file in the project root is correctly configured. ALWAYS use `override=True` when loading settings.
|