Brancheneinstufung2/GEMINI.md

Gemini Code Assistant Context

Wichtige Hinweise

• Projektdokumentation: Die primäre und umfassendste Dokumentation für dieses Projekt befindet sich in der Datei readme.md. Bitte ziehen Sie diese Datei für ein detailliertes Verständnis der Architektur und der einzelnen Module zu Rate.
• Git-Repository: Dieses Projekt wird über ein Git-Repository verwaltet. Alle Änderungen am Code werden versioniert. Beachten Sie den Abschnitt "Git Workflow & Conventions" für unsere Arbeitsregeln.

Project Overview

This project is a Python-based system for automated company data enrichment and lead generation. It uses a variety of data sources, including web scraping, Wikipedia, and the OpenAI API, to enrich company data from a CRM system. The project is designed to run in a Docker container and can be controlled via a Flask API.

The system is modular and consists of the following key components:

• brancheneinstufung_167.py: The core module for data enrichment, including web scraping, Wikipedia lookups, and AI-based analysis.
• company_deduplicator.py: A module for intelligent duplicate checking, both for external lists and internal CRM data.
• generate_marketing_text.py: An engine for creating personalized marketing texts.
• app.py: A Flask application that provides an API to run the different modules.
• company-explorer/: A new React/FastAPI-based application (v2.x) replacing the legacy CLI tools. It focuses on identifying robotics potential in companies.

Git Workflow & Conventions

• Commit-Nachrichten: Commits sollen einem klaren Format folgen:
  • Titel: Eine prägnante Zusammenfassung unter 100 Zeichen.
  • Beschreibung: Detaillierte Änderungen als Liste mit - am Zeilenanfang (keine Bulletpoints).
• Datei-Umbenennungen: Um die Git-Historie einer Datei zu erhalten, muss sie zwingend mit git mv alter_name.py neuer_name.py umbenannt werden.
• Commit & Push Prozess: Änderungen werden zuerst lokal committet. Das Pushen auf den Remote-Server erfolgt erst nach expliziter Bestätigung durch Sie.

Current Status (Jan 08, 2026) - Company Explorer (Robotics Edition)

• Robotics Potential Analysis (v2.3):

  • Logic Overhaul: Switched from keyword-based scanning to a "Chain-of-Thought" Infrastructure Analysis. The AI now evaluates physical assets (factories, warehouses, solar parks) to determine robotics needs.
  • Provider vs. User: Implemented strict reasoning to distinguish between companies selling cleaning products (providers) and those operating factories (users/potential clients).
  • Configurable Logic: Added a database-backed configuration system for robotics categories (cleaning, transport, security, service). Users can now define the "Trigger Logic" and "Scoring Guide" directly in the frontend settings.

• Wikipedia Integration (v2.1):

  • Deep Extraction: Implemented the "Legacy" extraction logic (WikipediaService). It now pulls the first paragraph (cleaned of references), categories (filtered for relevance), revenue, employees, and HQ location.
  • Google-First Discovery: Uses SerpAPI to find the correct Wikipedia article, validating via domain match and city.
  • Visual Inspector: The frontend Inspector now displays a comprehensive Wikipedia profile including category tags.

• Web Scraping & Legal Data (v2.2):

  • Impressum Scraping: Implemented a robust finder for "Impressum" / "Legal Notice" links.
    • Root-URL Fallback: If deep links (e.g., from /about-us) don't work, the scraper automatically checks the root domain (example.com/impressum).
    • LLM Extraction: Uses Gemini to parse unstructured Impressum text into structured JSON (Legal Name, Address, CEO).
  • Clean JSON Parsing: Implemented clean_json_response to handle AI responses containing Markdown (```json), preventing crash loops.

• Manual Overrides & Control:

  • Wikipedia Override: Added a UI to manually correct the Wikipedia URL. This triggers a re-scan and locks the record (is_locked flag) to prevent auto-overwrite.
  • Website Override: Added a UI to manually correct the company website. This automatically clears old scraping data to force a fresh analysis on the next run.

Lessons Learned & Best Practices

1. Numeric Extraction (German Locale):

   • Problem: "1.005 Mitarbeiter" was extracted as "1" (treating dot as decimal).
   • Solution: Implemented context-aware logic. If a number has a dot followed by exactly 3 digits (and no comma), it is treated as a thousands separator. For Revenue (is_umsatz=True), dots are generally treated as decimals (e.g. "375.6 Mio") unless multiple dots exist.
   • Rule: Always check for both , and . presence to determine locale.

2. LLM JSON Stability:

   • Problem: LLMs often wrap JSON in Markdown blocks, causing json.loads() to fail.
   • Solution: ALWAYS use a clean_json_response helper that strips ```json markers before parsing. Never trust raw LLM output for structured data.

3. Scraping Navigation:

   • Problem: Searching for "Impressum" only on the scraped URL (which might be a subpage found via Google) often fails.
   • Solution: Always implement a fallback to the Root Domain. The legal notice is almost always linked from the homepage footer.

Next Steps

• Frontend Debugging: Verify why the "Official Legal Data" block disappears in some states (likely due to conditional rendering checks on impressum object structure).
• Quality Assurance: Implement a dedicated "Review Mode" to validate high-potential leads.
• Data Import: Finalize the "List Matcher" to import and deduplicate Excel lists against the new DB.