Brancheneinstufung2/connector-superoffice/README.md

SuperOffice Connector README

Overview

This directory contains Python scripts designed to integrate with the SuperOffice CRM API, primarily for data extraction and analysis related to sales and customer product information.

🚀 Production Deployment (March 2026)

Status: ✅ Live (with active workarounds) Environment: online3 (Production) Tenant: Cust26720

1. Architecture & Flow

1. Trigger: SuperOffice sends a webhook (contact.created, contact.changed) to https://floke-ai.duckdns.org/connector/webhook.
2. Reception: webhook_app.py (FastAPI) receives the event, validates the token, and pushes a job to the SQLite queue (connector_queue.db).
3. Processing: worker.py polls the queue, fetches contact details from SuperOffice, and sends them to the Company Explorer.
4. Enrichment: Company Explorer analyzes the data and returns enrichment info (Vertical, Summary, etc.).
5. Sync: worker.py patches the data back into SuperOffice (UserDefinedFields).

2. Critical Issues & Workarounds (Lessons Learned)

🛑 A. The "Unhashable Dict" API Bug (Critical)

• Symptom: When fetching contact details (GET /Contact/{id}), Python crashes with TypeError: unhashable type: 'dict' when accessing UserDefinedFields.
• Cause: The SuperOffice Production API (online3) returns a malformed structure for UserDefinedFields for this tenant. It appears that one of the keys in the JSON response is being parsed as a dictionary object instead of a string, rendering the entire dictionary invalid for standard Python lookups. This behavior did not occur on the DEV (sod) environment.
• Workaround (Active): The worker.py implements a "Fail Open" strategy (safe_get_udfs).
  • It catches the TypeError.
  • It treats the existing UDFs as empty.
  • It proceeds with the enrichment and overwrites/patches the fields blindly.
  • Consequence: We cannot check if a field is already set before writing. We always write.

🔄 B. The "Ping-Pong" Loop (Resolved)

• Symptom: Accounts stuck in a loop (Processing -> Completed -> Processing -> ...).
• Cause: Due to Workaround A (Blind Write), the worker always sends a PATCH request to SuperOffice. Every PATCH triggers a new contact.changed webhook. Since we can't read the value to see "it's already done", we write again -> new webhook -> infinite loop.
• Solution: Implemented a Circuit Breaker in worker.py.
  • The worker checks the ChangedByAssociateId in the webhook payload.
  • If the ID matches our API User (528), the job is immediately marked as SUCCESS and skipped.

🔐 C. Environment Variables in Docker

• Lesson: docker-compose env_file directive makes variables available to the container, but python scripts run manually via docker exec do NOT see them unless load_dotenv is used explicitly.
• Fix: All scripts in tools/ now explicitly load the .env from the project root. config.py defaults to online3 to prevent accidental dev-connections.

3. Tooling (New)

Located in connector-superoffice/tools/:

• create_company.py: Creates a test company ("Bremer Abenteuerland") in Prod to trigger the webhook flow.
• verify_enrichment.py: Checks if the enrichment data (Vertical, Summary) actually landed in SuperOffice (bypassing the UDF crash).
• debug_raw_response.py: Saves the raw JSON response from SuperOffice to raw_api_response.json for debugging API structure issues.
• who_am_i.py: Attempts to identify the current API user (Associate ID).

4. Open Todos

• Support Ticket: Wait for SuperOffice to fix the UserDefinedFields JSON structure on Cust26720.
• Docker Optimization: The connector-superoffice build takes >8 minutes due to compiling C-extensions. Implement Multi-Stage Build.
• Hardcoded ID: Make the Circuit Breaker ID (528) configurable via .env (SO_API_ASSOCIATE_ID).

---

Authentication (Legacy)

Authentication is handled via the AuthHandler class, which uses a refresh token flow to obtain access tokens. Ensure that the .env file in the project root is correctly configured with SO_CLIENT_ID, SO_CLIENT_SECRET, SO_REFRESH_TOKEN, SO_REDIRECT_URI, SO_ENVIRONMENT, and SO_CONTEXT_IDENTIFIER.

Key SuperOffice Entities and Data Model Observations

During the development of reporting functionalities, the following observations were made regarding the SuperOffice data model:

1. Sale Entity

• Primary Source for Product Information: Contrary to initial expectations, product information is frequently stored as free-text within the Heading field of the Sale object (e.g., "2xOmnie CD-01 mit Nachlass"). This appears to be a common practice in the system, rather than utilizing structured product catalog items linked via quotes.
• Contact Association: A significant number of Sale objects (Sale?$filter=Contact ne null) are not directly linked to a Contact object (Contact: null), making it challenging to attribute sales to specific customers programmatically. Our reporting scripts specifically filter for sales where a Contact is present.
• No Direct Quote/QuoteLine Linkage: Attempts to retrieve Quote or QuoteLine objects directly via Sale/{saleId}/Quotes, Contact/{contactId}/Quotes, or Sale/{saleId}/Activities resulted in 500 Internal Server Errors or empty result sets. This indicates that direct, API-accessible linkages between Sales and structured QuoteLines are often absent or not exposed via these endpoints.

2. Product Information Storage (Hypothesis & Workaround)

• Free-Text in Heading: The primary source for identifying products associated with a sale is the Heading field of the Sale entity itself. This field often contains product codes, descriptions, and other relevant details as free-text.
• User-Defined Fields (UDFs): While UserDefinedFields were inspected for structured product data (e.g., RR-02-017-OMNIE), no such patterns were found in the sale_id=342243 example. This suggests that UDFs are either not consistently used for product codes or are named in a way that doesn't align with common product terminology.

Scripts

list_products.py

• Purpose: Fetches and displays a list of all defined product families from the SuperOffice product catalog (/List/ProductFamily/Items).
• Usage: python3 list_products.py

generate_customer_product_report.py

• Purpose: Generates a CSV report of customer sales, extracting product information from the Sale.Heading field using keyword matching.
• Methodology:
  1. Retrieves the latest SALE_LIMIT (e.g., 1000) Sale objects, filtering only those with an associated Contact ($filter=Contact ne null).
  2. Extracts SaleId, CustomerName, and SaleHeading for each relevant sale.
  3. Searches the SaleHeading for predefined PRODUCT_KEYWORDS (e.g., OMNIE, CD-01, Service).
  4. Outputs the results to product_report.csv.
• Usage: python3 generate_customer_product_report.py

Future Work

• Analyse der leeren product_report.csv: Untersuchen, warum die product_report.csv auch nach der Filterung nach Sale-Objekten mit Contact-Verknüpfung leer bleibt. Es ist entscheidend zu verstehen, ob es keine solchen Verkäufe gibt oder ob ein Problem mit der Datenabfrage oder -verarbeitung vorliegt.
• Manuelle Inspektion gefilterter Sale-Objekte: Wenn der Report leer ist, müssen wir einige Sale-Objekte, die die Bedingung Contact ne null erfüllen, manuell inspizieren, um ihre Struktur zu verstehen und festzustellen, ob das Heading-Feld oder andere Felder Produktinformationen enthalten.
• Verfeinerung der PRODUCT_KEYWORDS: Die Liste der Produkt-Schlüsselwörter muss möglicherweise erweitert werden, basierend auf einer detaillierteren manuellen Analyse der Verkaufsüberschriften.
• Erforschung alternativer API-Pfade: Falls der aktuelle Ansatz weiterhin Schwierigkeiten bereitet, müssen wir tiefer in die SuperOffice-API eintauchen, um strukturierte Produktdaten zu finden, auch wenn sie nicht direkt mit den Verkäufen verknüpft sind.