Brancheneinstufung2/GEMINI.md

Gemini Code Assistant Context

Wichtige Hinweise

• Projektdokumentation: Die primäre und umfassendste Dokumentation für dieses Projekt befindet sich in der Datei readme.md. Bitte ziehen Sie diese Datei für ein detailliertes Verständnis der Architektur und der einzelnen Module zu Rate.
• Git-Repository: Dieses Projekt wird über ein Git-Repository verwaltet. Alle Änderungen am Code werden versioniert. Beachten Sie den Abschnitt "Git Workflow & Conventions" für unsere Arbeitsregeln.

Project Overview

This project is a Python-based system for automated company data enrichment and lead generation. It focuses on identifying B2B companies with high potential for robotics automation (Cleaning, Transport, Security, Service).

The system architecture has evolved from a CLI-based toolset to a modern web application (company-explorer) backed by Docker containers.

Current Status (Jan 08, 2026) - Company Explorer (Robotics Edition v0.3.0)

1. Robotics Potential Analysis (v2.3)

• Chain-of-Thought Logic: The AI analysis (ClassificationService) now uses a multi-step reasoning process to evaluate companies based on their physical infrastructure (factories, warehouses) rather than just keywords.
• Provider vs. User: Strict logic implemented to distinguish between companies selling automation products and those needing them for their own operations.
• Configurable Settings: A database-driven configuration (RoboticsCategory) allows users to edit the definition and scoring logic for each robotics category directly via the frontend settings menu.

2. Deep Wikipedia Integration (v2.1)

• Extraction: The system extracts the first paragraph (cleaned of artifacts), industry, revenue (normalized to Mio €), employee count, and Wikipedia categories.
• Validation: Uses a "Google-First" strategy via SerpAPI, validating candidates by checking for domain matches and city/HQ location in the article.
• UI: The Inspector displays a dedicated Wikipedia profile section with visual tags.

3. Web Scraping & Legal Data (v2.2)

• Impressum Scraping:
  • 2-Hop Strategy: If no "Impressum" link is found on the landing page, the scraper automatically searches for for a "Kontakt" page and checks for the link there.
  • Root Fallback: If deep links (e.g. /about-us) fail, the scraper checks the root domain (/).
  • LLM Extraction: Unstructured legal text is parsed by Gemini to extract structured JSON (Legal Name, Address, CEO, VAT ID).
• Robustness:
  • JSON Cleaning: A helper (clean_json_response) strips Markdown code blocks from LLM responses to prevent parsing errors.
  • Aggressive Cleaning: HTML scraper removes <a> tags and cookie banners before text analysis to reduce noise.

4. User Control & Ops

• Manual Overrides: Users can manually correct the Wikipedia URL (locking the data) and the Company Website (triggering a fresh re-scrape).
• Polling UI: The frontend uses intelligent polling to auto-refresh data when background jobs (Discovery/Analysis) complete.
• Forced Refresh: The "Analyze" endpoint now clears old cache data to ensure a fresh scrape on every user request.

Lessons Learned & Best Practices

1. Numeric Extraction (German Locale):

   • Problem: "1.005 Mitarbeiter" was extracted as "1" (treating dot as decimal).
   • Solution: Implemented context-aware logic. If a number has a dot followed by exactly 3 digits (and no comma), it is treated as a thousands separator.
   • Revenue: For revenue (is_umsatz=True), dots are generally treated as decimals (e.g. "375.6 Mio") unless unambiguous multiple dots exist. Billion/Mrd is converted to 1000 Million.

2. LLM JSON Stability:

   • Problem: LLMs often wrap JSON in Markdown blocks (```json), causing json.loads() to fail.
   • Solution: ALWAYS use a clean_json_response helper that strips markers before parsing. Never trust raw LLM output.

3. Scraping Navigation:

   • Problem: Searching for "Impressum" only on the scraped URL (which might be a subpage found via Google) often fails.
   • Solution: Always implement a fallback to the Root Domain AND a 2-Hop check via the "Kontakt" page.

4. Frontend State Management:

   • Problem: Users didn't see when a background job finished.
   • Solution: Implementing a polling mechanism (setInterval) tied to a isProcessing state is superior to static timeouts for long-running AI tasks.

5. Impressum Extraction Debugging:

   • Problem: Impressum fields sometimes return empty/null even when the URL is correctly identified and the page exists.
   • Solution: Increased logging verbosity in _scrape_impressum_data to output the exact raw text sent to the LLM and the raw LLM response. This helps diagnose issues with LLM interpretation or JSON formatting during extraction.

Next Steps

• Data Import: Finalize the "List Matcher" to import and deduplicate Excel lists against the new DB.
• Export: Generate Excel/CSV exports of enriched leads for CRM import.