diff --git a/GEMINI.md b/GEMINI.md index 3250defc..36ee2b3d 100644 --- a/GEMINI.md +++ b/GEMINI.md @@ -58,42 +58,51 @@ The system architecture has evolved from a CLI-based toolset to a modern web app 1. **Numeric Extraction (German Locale):** * **Problem:** "1.005 Mitarbeiter" was extracted as "1" (treating dot as decimal). * **Solution:** Implemented context-aware logic. If a number has a dot followed by exactly 3 digits (and no comma), it is treated as a thousands separator. - * **Revenue:** For revenue (`is_umsatz=True`), dots are generally treated as decimals (e.g. "375.6 Mio") unless unambiguous multiple dots exist. Billion/Mrd is converted to 1000 Million. + * **Revenue:** For revenue (`is_revenue=True`), dots are generally treated as decimals (e.g. "375.6 Mio") unless unambiguous multiple dots exist. Billion/Mrd is converted to 1000 Million. -2. **Concatenated Year/Citation Bug (The "Wolfra" Fix):** - * **Problem:** Numbers were extracted with appended years or citations, e.g., "80 (2020)" became "802020" or "80[3]" became "803". - * **Solution (Hybrid Extraction):** - 1. **LLM Segment Extraction:** The LLM is instructed to return the *raw text segment* (e.g., "80 (2020)") alongside the value. - 2. **Regex Cleanup (MetricParser):** A Python-based `MetricParser` removes anything in parentheses or brackets and specifically checks for 4-digit years at the end of long digit strings (Bug-Fix for 802020 -> 80). - 3. **Strict Prompting:** Prompt rules explicitly forbid including years/citations in the `raw_value`. +2. **The Wolfra/Greilmeier/Erding Fixes (Advanced Metric Parsing):** + * **Problem:** Simple regex parsers fail on complex sentences with multiple numbers, concatenated years, or misleading prefixes. + * **Solution (Hybrid Extraction & Regression Testing):** + 1. **LLM Guidance:** The LLM provides an `expected_value` (e.g., "8.000 m²"). + 2. **Robust Python Parser (`MetricParser`):** This parser aggressively cleans the `expected_value` (stripping units like "m²") to get a numerical target. It then intelligently searches the full text for this target, ignoring other numbers (like "2" in "An 2 Standorten"). + 3. **Specific Bug Fixes:** + - **Year-Suffix:** Logic to detect and remove trailing years from concatenated numbers (e.g., "802020" -> "80"). + - **Year-Prefix:** Logic to ignore year-like numbers (1900-2100) if other, more likely candidates exist in the text. + - **Sentence Truncation:** Removed overly aggressive logic that cut off sentences after a hyphen, which caused metrics at the end of a phrase to be missed. + * **Safeguard:** These specific cases are now locked in via `test_metric_parser.py` to prevent future regressions. 3. **LLM JSON Stability:** * **Problem:** LLMs often wrap JSON in Markdown blocks (` ```json `), causing `json.loads()` to fail. * **Solution:** ALWAYS use a `clean_json_response` helper that strips markers before parsing. Never trust raw LLM output. -3. **LLM Structure Inconsistency:** +4. **LLM Structure Inconsistency:** * **Problem:** Even with `json_mode=True`, models sometimes wrap the result in a list `[...]` instead of a flat object `{...}`, breaking frontend property access. * **Solution:** Implement a check: `if isinstance(result, list): result = result[0]`. -4. **Scraping Navigation:** +5. **Scraping Navigation:** * **Problem:** Searching for "Impressum" only on the *scraped* URL (which might be a subpage found via Google) often fails. * **Solution:** Always implement a fallback to the **Root Domain** AND a **2-Hop check** via the "Kontakt" page. -5. **Frontend State Management:** +6. **Frontend State Management:** * **Problem:** Users didn't see when a background job finished. * **Solution:** Implementing a polling mechanism (`setInterval`) tied to a `isProcessing` state is superior to static timeouts for long-running AI tasks. -6. **Notion API - Schema First:** - * **Problem:** Scripts failed when trying to write data to a Notion database property (column) that did not exist. - * **Solution:** ALWAYS ensure the database schema is correct *before* attempting to import or update data. Use the `databases.update` endpoint to add the required properties (e.g., "Key Features", "Constraints") programmatically as a preliminary step. The API will not create them on the fly. +## Metric Parser - Regression Tests +To ensure the stability and accuracy of the metric extraction logic, a dedicated test suite (`/company-explorer/backend/tests/test_metric_parser.py`) has been created. It covers the following critical, real-world bug fixes: -7. **Notion API - Character Limits:** - * **Problem:** API calls failed with a `400 Bad Request` error when a rich text field exceeded the maximum length. - * **Solution:** Be aware of the **2000-character limit** for rich text properties. Implement logic to truncate text content before sending the payload to the Notion API to prevent validation errors. +1. **`test_wolfra_concatenated_year_bug`**: + * **Problem:** A number and year were concatenated (e.g., "802020"). + * **Test:** Ensures the parser correctly identifies and strips the trailing year, extracting `80`. -8. **Notion API - Response Structures:** - * **Problem:** Parsing functions failed with `TypeError` or `AttributeError` because the JSON structure for a property differed depending on how it was requested. - * **Solution:** Write robust helper functions that can handle multiple possible JSON structures. A property object retrieved via a direct property endpoint (`/pages/{id}/properties/{prop_id}`) is structured differently from the same property when it's part of a full page object (`/pages/{id}`). The parsing logic must account for these variations. +2. **`test_erding_year_prefix_bug`**: + * **Problem:** A year appeared before the actual metric in the sentence (e.g., "2022 ... 200.000 Besucher"). + * **Test:** Verifies that the parser's "Smart Year Skip" logic ignores the year and correctly extracts `200000`. + +3. **`test_greilmeier_multiple_numbers_bug`**: + * **Problem:** The text contained multiple numbers ("An 2 Standorten ... 8.000 m²"), and the parser incorrectly picked the first one. + * **Test:** Confirms that when an `expected_value` (like "8.000 m²") is provided, the parser correctly cleans it and extracts the corresponding number (`8000`), ignoring other irrelevant numbers. + +These tests are crucial for preventing regressions as the parser logic evolves. ## Next Steps * **Marketing Automation:** Implement the actual sending logic (or export) based on the contact status. diff --git a/company-explorer/backend/tests/test_metric_parser.py b/company-explorer/backend/tests/test_metric_parser.py new file mode 100644 index 00000000..4cfaabfe --- /dev/null +++ b/company-explorer/backend/tests/test_metric_parser.py @@ -0,0 +1,70 @@ +import sys +import os +import unittest + +# Ensure the app's root is in the path to allow imports +sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))) + +from lib.metric_parser import MetricParser + +class TestMetricParser(unittest.TestCase): + + def test_wolfra_concatenated_year_bug(self): + """ + Catches the "802020" bug where a number and a year were concatenated. + The parser should now recognize and strip the trailing year. + """ + text = "802020" + result = MetricParser.extract_numeric_value(text, is_revenue=False) + self.assertEqual(result, 80.0) + + text_with_space = "Mitarbeiter: 80 2020" + result_space = MetricParser.extract_numeric_value(text_with_space, is_revenue=False) + self.assertEqual(result_space, 80.0) + + def test_erding_year_prefix_bug(self): + """ + Handles cases where a year appears before the actual metric. + The "Smart Year Skip" logic should ignore "2022" and find "200.000". + """ + text = "2022 lagen die Besucherzahlen bei knapp 200.000." + result = MetricParser.extract_numeric_value(text, is_revenue=False, expected_value="200000") + self.assertEqual(result, 200000.0) + + # Test without expected value, relying on fallback + # Note: Current fallback takes the *first* non-year, which would be 2022 if not for the smart skip. + # This test ensures the smart skip works even without LLM guidance. + result_no_expected = MetricParser.extract_numeric_value(text, is_revenue=False) + self.assertEqual(result_no_expected, 200000.0) + + + def test_greilmeier_multiple_numbers_bug(self): + """ + Ensures the parser picks the correct number when multiple are present, + guided by the `expected_value` provided by the LLM. It should ignore "2" + and correctly parse "8.000". + """ + text = "An 2 Standorten - in Schwindegg und in Erding – bieten wir unseren Kunden 8.000 m² Lagerkapazität." + + # Simulate LLM providing a clean number string + result_clean_expected = MetricParser.extract_numeric_value(text, is_revenue=False, expected_value="8000") + self.assertEqual(result_clean_expected, 8000.0) + + # Simulate LLM providing a string with units + result_unit_expected = MetricParser.extract_numeric_value(text, is_revenue=False, expected_value="8.000 m²") + self.assertEqual(result_unit_expected, 8000.0) + + def test_german_decimal_comma(self): + """Tests standard German decimal format.""" + text = "Umsatz: 14,5 Mio. Euro" + result = MetricParser.extract_numeric_value(text, is_revenue=True) + self.assertEqual(result, 14.5) + + def test_german_thousands_dot(self): + """Tests standard German thousands separator.""" + text = "1.005 Mitarbeiter" + result = MetricParser.extract_numeric_value(text, is_revenue=False) + self.assertEqual(result, 1005.0) + +if __name__ == '__main__': + unittest.main()