Rollback auf 1.3.5
Hier ist eine aktualisierte Version, die alle oben beschriebenen Anpassungen berücksichtigt. Ich habe Folgendes vorgenommen: Versionsupdate: Die Versionsnummer wurde auf v1.3.16 gesetzt. Neue Modi integriert: Modus 8 (Batch-Token-Zählung in Spalte AQ) Modus 51 (Verifizierung: Nur Wikipedia + Brancheneinordnung in einem Batch-Prozess) Die bestehenden Modi (1, 2, 3, 4, 5, 6, 7) bleiben erhalten. Verbesserte Header-Definitionen: Sowohl im Hauptblatt als auch im „Contacts“-Blatt. Verbesserte Fehlerbehandlung und Logging: Kleinere Anpassungen beim Logging und beim Warten auf Updates. Im Folgenden findest Du den vollständigen, aktualisierten Code (v1.3.16):
This commit is contained in:
@@ -18,7 +18,7 @@ except ImportError:
|
||||
|
||||
# ==================== KONFIGURATION ====================
|
||||
class Config:
|
||||
VERSION = "v1.3.16a" # v1.3.16: Neuer Modus 51 für gezielte Verifizierung (Branche & FSM)
|
||||
VERSION = "v1.3.16" # v1.3.16: Neuer Modus 8 (Batch-Token-Zählung in Spalte AQ) & Modus 51 (nur Verifizierung)
|
||||
LANG = "de"
|
||||
CREDENTIALS_FILE = "service_account.json"
|
||||
SHEET_URL = "https://docs.google.com/spreadsheets/d/1u_gHr9JUfmV1-iviRzbSe3575QEp7KLhK5jFV_gJcgo"
|
||||
@@ -176,8 +176,11 @@ def validate_article_with_chatgpt(crm_data, wiki_data):
|
||||
wiki_headers = "Wikipedia URL;Wikipedia Absatz;Wikipedia Branche;Wikipedia Umsatz;Wikipedia Mitarbeiter;Wikipedia Kategorien"
|
||||
prompt_text = (
|
||||
"Bitte überprüfe, ob die folgenden beiden Datensätze grundsätzlich zum gleichen Unternehmen gehören. "
|
||||
"Berücksichtige dabei leichte Abweichungen in Firmennamen und Ort. Wenn sie im Wesentlichen übereinstimmen, antworte mit 'OK'. "
|
||||
"Andernfalls nenne den wichtigsten Grund und eine kurze Begründung.\n\n"
|
||||
"Berücksichtige dabei, dass leichte Abweichungen in Firmennamen (z. B. unterschiedliche Schreibweisen, Mutter-Tochter-Beziehungen) "
|
||||
"oder im Ort (z. B. 'Oberndorf' vs. 'Oberndorf/Neckar') tolerierbar sind. "
|
||||
"Vergleiche insbesondere den Firmennamen, den Ort und die Branche. Unterschiede im Umsatz können bis zu 10% abweichen. "
|
||||
"Wenn die Daten im Wesentlichen übereinstimmen, antworte ausschließlich mit 'OK'. "
|
||||
"Falls nicht, nenne bitte den wichtigsten Grund und eine kurze Begründung, warum die Abweichung plausibel sein könnte.\n\n"
|
||||
f"CRM-Daten:\n{crm_headers}\n{crm_data}\n\n"
|
||||
f"Wikipedia-Daten:\n{wiki_headers}\n{wiki_data}\n\n"
|
||||
"Antwort: "
|
||||
@@ -203,16 +206,58 @@ def validate_article_with_chatgpt(crm_data, wiki_data):
|
||||
return "k.A."
|
||||
|
||||
def evaluate_branche_chatgpt(crm_branche, beschreibung, wiki_branche, wiki_kategorien):
|
||||
prompt_text = (
|
||||
"Du bist ein Experte im Field Service Management. Analysiere die folgenden Branchenangaben und ordne das Unternehmen "
|
||||
"einer der gültigen Branchen zu. Nutze ausschließlich die vorhandenen Informationen.\n\n"
|
||||
f"CRM-Branche: {crm_branche}\n"
|
||||
f"Beschreibung Branche extern: {beschreibung}\n"
|
||||
f"Wikipedia-Branche: {wiki_branche}\n"
|
||||
f"Wikipedia-Kategorien: {wiki_kategorien}\n\n"
|
||||
"Ordne das Unternehmen exakt einer der gültigen Branchen zu und gib aus:\n"
|
||||
target_branches = []
|
||||
try:
|
||||
with open("ziel_Branchenschema.csv", "r", encoding="utf-8") as csvfile:
|
||||
reader = csv.reader(csvfile)
|
||||
target_branches = [row[0] for row in reader if row]
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler beim Laden des Ziel-Branchenschemas: {e}")
|
||||
target_branches_str = "\n".join(target_branches)
|
||||
focus_branches = [
|
||||
"Gutachter / Versicherungen > Baugutachter",
|
||||
"Gutachter / Versicherungen > Technische Gutachten",
|
||||
"Gutachter / Versicherungen > Versicherungsgutachten",
|
||||
"Gutachter / Versicherungen > Medizinische Gutachten",
|
||||
"Hersteller / Produzenten > Anlagenbau",
|
||||
"Hersteller / Produzenten > Automaten (Vending, Slot)",
|
||||
"Hersteller / Produzenten > Gebäudetechnik Allgemein",
|
||||
"Hersteller / Produzenten > Gebäudetechnik Heizung, Lüftung, Klima",
|
||||
"Hersteller / Produzenten > Maschinenbau",
|
||||
"Hersteller / Produzenten > Medizintechnik",
|
||||
"Service provider (Dienstleister) > Aufzüge und Rolltreppen",
|
||||
"Service provider (Dienstleister) > Feuer- und Sicherheitssysteme",
|
||||
"Service provider (Dienstleister) > Servicedienstleister / Reparatur ohne Produktion",
|
||||
"Service provider (Dienstleister) > Facility Management",
|
||||
"Versorger > Telekommunikation"
|
||||
]
|
||||
focus_branches_str = "\n".join(focus_branches)
|
||||
additional_instruction = ""
|
||||
if wiki_branche.strip() == "k.A.":
|
||||
additional_instruction = (
|
||||
"Da keine Wikipedia-Branche vorliegt, berücksichtige bitte die Wikipedia-Kategorien mit erhöhter Gewichtung, "
|
||||
"insbesondere wenn Hinweise auf Personentransport oder öffentliche Verkehrsdienstleistungen vorliegen. "
|
||||
)
|
||||
system_prompt = (
|
||||
"Du bist ein Experte im Field Service Management. Deine Aufgabe ist es, ein Unternehmen basierend auf folgenden Angaben einer Branche zuzuordnen.\n\n"
|
||||
f"CRM-Branche (Spalte F): {crm_branche}\n"
|
||||
f"Branchenbeschreibung (Spalte G): {beschreibung}\n"
|
||||
f"Wikipedia-Branche (Spalte N): {wiki_branche}\n"
|
||||
f"Wikipedia-Kategorien (Spalte Q): {wiki_kategorien}\n\n"
|
||||
+ additional_instruction +
|
||||
"Das Ziel-Branchenschema umfasst ALLE gültigen Branchen, also sowohl Fokusbranchen als auch weitere, z. B. 'Housing > Sozialbau Unternehmen'.\n"
|
||||
"Das vollständige Ziel-Branchenschema lautet:\n"
|
||||
f"{target_branches_str}\n\n"
|
||||
"Falls das Unternehmen mehreren Branchen zugeordnet werden könnte, wähle bitte bevorzugt eine Branche aus der folgenden Fokusliste, sofern zutreffend:\n"
|
||||
f"{focus_branches_str}\n\n"
|
||||
"Gewichtung der Angaben:\n"
|
||||
"1. Wikipedia-Branche (Spalte N) zusammen mit Wikipedia-Kategorien (Spalte Q) (höchste Priorität, wenn verifiziert, ansonsten erhöhte Gewichtung der Kategorien)\n"
|
||||
"2. Branchenbeschreibung (Spalte G)\n"
|
||||
"3. CRM-Branche (Spalte F)\n\n"
|
||||
"Ordne das Unternehmen exakt einer der oben genannten Branchen zu (es dürfen keine zusätzlichen Branchen erfunden werden). "
|
||||
"Bitte antworte in folgendem Format (ohne zusätzliche Informationen):\n"
|
||||
"Branche: <vorgeschlagene Branche>\n"
|
||||
"Übereinstimmung: <OK oder X>\n"
|
||||
"Übereinstimmung: <ok oder X>\n"
|
||||
"Begründung: <kurze Begründung, falls abweichend, ansonsten leer>"
|
||||
)
|
||||
try:
|
||||
@@ -225,7 +270,7 @@ def evaluate_branche_chatgpt(crm_branche, beschreibung, wiki_branche, wiki_kateg
|
||||
try:
|
||||
response = openai.ChatCompletion.create(
|
||||
model="gpt-3.5-turbo",
|
||||
messages=[{"role": "system", "content": prompt_text}],
|
||||
messages=[{"role": "system", "content": system_prompt}],
|
||||
temperature=0.0
|
||||
)
|
||||
result = response.choices[0].message.content.strip()
|
||||
@@ -302,7 +347,9 @@ def evaluate_servicetechnicians_estimate(company_name, company_data):
|
||||
return "k.A."
|
||||
openai.api_key = api_key
|
||||
prompt = (
|
||||
f"Bitte schätze die Anzahl der Servicetechniker des Unternehmens '{company_name}' in einer der folgenden Kategorien: "
|
||||
f"Bitte schätze auf Basis öffentlich zugänglicher Informationen (vor allem verifizierte Wikipedia-Daten) "
|
||||
f"die Anzahl der Servicetechniker des Unternehmens '{company_name}' ein. "
|
||||
"Gib die Antwort ausschließlich in einer der folgenden Kategorien aus: "
|
||||
"'<50 Techniker', '>100 Techniker', '>200 Techniker', '>500 Techniker'."
|
||||
)
|
||||
try:
|
||||
@@ -327,7 +374,8 @@ def evaluate_servicetechnicians_explanation(company_name, st_estimate, company_d
|
||||
return "k.A."
|
||||
openai.api_key = api_key
|
||||
prompt = (
|
||||
f"Bitte erkläre, warum du für das Unternehmen '{company_name}' die Anzahl der Servicetechniker als '{st_estimate}' geschätzt hast."
|
||||
f"Bitte erkläre, warum du für das Unternehmen '{company_name}' die Anzahl der Servicetechniker als '{st_estimate}' geschätzt hast. "
|
||||
"Berücksichtige dabei öffentlich zugängliche Informationen wie Branche, Umsatz, Mitarbeiterzahl und andere relevante Daten."
|
||||
)
|
||||
try:
|
||||
response = openai.ChatCompletion.create(
|
||||
@@ -368,60 +416,98 @@ def wait_for_sheet_update(sheet, cell, expected_value, timeout=5):
|
||||
time.sleep(0.5)
|
||||
return False
|
||||
|
||||
# ==================== NEUE FUNKTION: LINKEDIN-KONTAKT-SUCHE MIT SERPAPI ====================
|
||||
def search_linkedin_contact(company_name, website, position_query):
|
||||
try:
|
||||
with open("serpApiKey.txt", "r") as f:
|
||||
serp_key = f.read().strip()
|
||||
except Exception as e:
|
||||
debug_print("Fehler beim Lesen des SerpAPI-Schlüssels: " + str(e))
|
||||
return None
|
||||
query = f'site:linkedin.com/in "{position_query}" "{company_name}"'
|
||||
debug_print(f"Erstelle LinkedIn-Query: {query}")
|
||||
params = {
|
||||
"engine": "google",
|
||||
"q": query,
|
||||
"api_key": serp_key,
|
||||
"hl": "de"
|
||||
}
|
||||
try:
|
||||
response = requests.get("https://serpapi.com/search", params=params)
|
||||
data = response.json()
|
||||
debug_print(f"SerpAPI-Response für Query '{query}': {data.get('organic_results', [])[:1]}")
|
||||
if "organic_results" in data and len(data["organic_results"]) > 0:
|
||||
result = data["organic_results"][0]
|
||||
title = result.get("title", "")
|
||||
if "–" in title:
|
||||
parts = title.split("–")
|
||||
elif "-" in title:
|
||||
parts = title.split("-")
|
||||
else:
|
||||
parts = [title]
|
||||
if len(parts) >= 2:
|
||||
name_part = parts[0].strip()
|
||||
pos = parts[1].split("|")[0].strip()
|
||||
name_parts = name_part.split(" ", 1)
|
||||
if len(name_parts) == 2:
|
||||
firstname, lastname = name_parts
|
||||
else:
|
||||
firstname = name_part
|
||||
lastname = ""
|
||||
return {"Firmenname": company_name, "Website": website, "Vorname": firstname, "Nachname": lastname, "Position": pos}
|
||||
else:
|
||||
return {"Firmenname": company_name, "Website": website, "Vorname": "", "Nachname": "", "Position": title}
|
||||
else:
|
||||
return None
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler bei der SerpAPI-Suche: {e}")
|
||||
return None
|
||||
|
||||
def count_linkedin_contacts(company_name, website, position_query):
|
||||
try:
|
||||
with open("serpApiKey.txt", "r") as f:
|
||||
serp_key = f.read().strip()
|
||||
except Exception as e:
|
||||
debug_print("Fehler beim Lesen des SerpAPI-Schlüssels: " + str(e))
|
||||
return 0
|
||||
query = f'site:linkedin.com/in "{position_query}" "{company_name}"'
|
||||
debug_print(f"Erstelle LinkedIn-Query (Count): {query}")
|
||||
params = {
|
||||
"engine": "google",
|
||||
"q": query,
|
||||
"api_key": serp_key,
|
||||
"hl": "de"
|
||||
}
|
||||
try:
|
||||
response = requests.get("https://serpapi.com/search", params=params)
|
||||
data = response.json()
|
||||
if "organic_results" in data:
|
||||
count = len(data["organic_results"])
|
||||
debug_print(f"Anzahl Kontakte für Query '{query}': {count}")
|
||||
return count
|
||||
else:
|
||||
debug_print(f"Keine Ergebnisse für Query: {query}")
|
||||
return 0
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler bei der SerpAPI-Suche (Count): {e}")
|
||||
return 0
|
||||
|
||||
# ==================== VERIFIZIERUNGS-MODUS (Modus 51) ====================
|
||||
def _process_verification_row(row_num, row_data):
|
||||
company_name = row_data[1] if len(row_data) > 1 else ""
|
||||
website = row_data[3] if len(row_data) > 3 else ""
|
||||
current_dt = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
if len(row_data) > 11 and row_data[11].strip() not in ["", "k.A."]:
|
||||
wiki_url = row_data[11].strip()
|
||||
try:
|
||||
wiki_data = WikipediaScraper().extract_company_data(wiki_url)
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler beim Laden des vorgeschlagenen Wikipedia-Artikels: {e}")
|
||||
article = WikipediaScraper().search_company_article(company_name, website)
|
||||
wiki_data = WikipediaScraper().extract_company_data(article.url) if article else {
|
||||
'url': 'k.A.', 'first_paragraph': 'k.A.', 'branche': 'k.A.',
|
||||
'umsatz': 'k.A.', 'mitarbeiter': 'k.A.', 'categories': 'k.A.',
|
||||
'full_infobox': 'k.A.'
|
||||
}
|
||||
else:
|
||||
article = WikipediaScraper().search_company_article(company_name, website)
|
||||
wiki_data = WikipediaScraper().extract_company_data(article.url) if article else {
|
||||
'url': 'k.A.', 'first_paragraph': 'k.A.', 'branche': 'k.A.',
|
||||
'umsatz': 'k.A.', 'mitarbeiter': 'k.A.', 'categories': 'k.A.',
|
||||
'full_infobox': 'k.A.'
|
||||
}
|
||||
wiki_values = [
|
||||
row_data[11] if len(row_data) > 11 and row_data[11].strip() not in ["", "k.A."] else "k.A.",
|
||||
wiki_data.get('url', 'k.A.'),
|
||||
wiki_data.get('first_paragraph', 'k.A.'),
|
||||
wiki_data.get('branche', 'k.A.'),
|
||||
wiki_data.get('umsatz', 'k.A.'),
|
||||
wiki_data.get('mitarbeiter', 'k.A.'),
|
||||
wiki_data.get('categories', 'k.A.')
|
||||
]
|
||||
gh = GoogleSheetHandler()
|
||||
gh.sheet.update(values=[wiki_values], range_name=f"L{row_num}:R{row_num}")
|
||||
# Branchenbewertung:
|
||||
crm_branch = row_data[6] if len(row_data) > 6 else "k.A."
|
||||
ext_branch = row_data[7] if len(row_data) > 7 else "k.A."
|
||||
wiki_branch = wiki_data.get('branche', 'k.A.')
|
||||
wiki_cats = wiki_data.get('categories', 'k.A.')
|
||||
branch_result = evaluate_branche_chatgpt(crm_branch, ext_branch, wiki_branch, wiki_cats)
|
||||
gh.sheet.update(values=[[branch_result["branch"]]], range_name=f"W{row_num}")
|
||||
gh.sheet.update(values=[[branch_result["consistency"]]], range_name=f"Y{row_num}")
|
||||
# Validierung mit ChatGPT:
|
||||
crm_data = ";".join(row_data[1:11])
|
||||
wiki_data_str = ";".join(row_data[11:18])
|
||||
valid_result = validate_article_with_chatgpt(crm_data, wiki_data_str)
|
||||
gh.sheet.update(values=[[valid_result]], range_name=f"R{row_num}")
|
||||
# Schreibe Timestamp, Version und Token Count:
|
||||
gh.sheet.update(values=[[current_dt]], range_name=f"AO{row_num}")
|
||||
gh.sheet.update(values=[[Config.VERSION]], range_name=f"AP{row_num}")
|
||||
# Für Batch-Token-Zählung wird später Spalte AQ aktualisiert.
|
||||
debug_print(f"Zeile {row_num} verifiziert: Antwort: {valid_result}")
|
||||
time.sleep(Config.RETRY_DELAY)
|
||||
crm_description = row_data[7] if len(row_data) > 7 else "k.A."
|
||||
wiki_url = row_data[11] if len(row_data) > 11 and row_data[11].strip() not in ["", "k.A."] else "k.A."
|
||||
wiki_absatz = row_data[12] if len(row_data) > 12 else "k.A."
|
||||
wiki_categories = row_data[16] if len(row_data) > 16 else "k.A."
|
||||
entry_text = (f"Eintrag {row_num}:\n"
|
||||
f"Firmenname: {company_name}\n"
|
||||
f"CRM-Beschreibung: {crm_description}\n"
|
||||
f"Wikipedia-URL: {wiki_url}\n"
|
||||
f"Wikipedia-Absatz: {wiki_absatz}\n"
|
||||
f"Wikipedia-Kategorien: {wiki_categories}\n"
|
||||
"-----\n")
|
||||
return entry_text
|
||||
|
||||
def process_verification_only():
|
||||
debug_print("Starte Verifizierungsmodus (Modus 51) im Batch-Prozess...")
|
||||
@@ -430,33 +516,27 @@ def process_verification_only():
|
||||
sh = gc.open_by_url(Config.SHEET_URL)
|
||||
main_sheet = sh.sheet1
|
||||
data = main_sheet.get_all_values()
|
||||
batch_size = Config.BATCH_SIZE
|
||||
batch_entries = []
|
||||
row_indices = []
|
||||
for i, row in enumerate(data[1:], start=2):
|
||||
if len(row) <= 25 or row[24].strip() == "":
|
||||
# Hier wird _process_verification_row genutzt
|
||||
entry_text = (f"Eintrag {i}:\n"
|
||||
f"Firmenname: {row[1] if len(row)>1 else 'k.A.'}\n"
|
||||
f"CRM-Beschreibung: {row[7] if len(row)>7 else 'k.A.'}\n"
|
||||
f"Wikipedia-URL: {row[11] if len(row)>11 else 'k.A.'}\n"
|
||||
f"Wikipedia-Absatz: {row[12] if len(row)>12 else 'k.A.'}\n"
|
||||
f"Wikipedia-Kategorien: {row[17] if len(row)>17 else 'k.A.'}\n"
|
||||
"-----")
|
||||
if len(row) <= 19 or row[18].strip() == "":
|
||||
entry_text = _process_verification_row(i, row)
|
||||
batch_entries.append(entry_text)
|
||||
row_indices.append(i)
|
||||
if len(batch_entries) == Config.BATCH_SIZE:
|
||||
if len(batch_entries) == batch_size:
|
||||
break
|
||||
if not batch_entries:
|
||||
debug_print("Keine Einträge für die Verifizierung gefunden.")
|
||||
return
|
||||
aggregated_prompt = ("Du bist ein Experte in der Verifizierung von Wikipedia-Artikeln für Unternehmen. "
|
||||
"Für jeden der folgenden Einträge prüfe, ob der vorhandene Wikipedia-Artikel plausibel passt. "
|
||||
"Für jeden der folgenden Einträge prüfe, ob der vorhandene Wikipedia-Artikel (URL, Absatz, Kategorien) plausibel passt. "
|
||||
"Gib für jeden Eintrag das Ergebnis im Format aus:\n"
|
||||
"Eintrag <Zeilennummer>: <Antwort>\n"
|
||||
"Antwortoptionen:\n"
|
||||
"- 'OK' wenn der Artikel passt\n"
|
||||
"- 'Kein Wikipedia-Eintrag vorhanden.'\n"
|
||||
"- 'Alternativer Wikipedia-Artikel vorgeschlagen: <URL> | X | <Begründung>'\n\n")
|
||||
"Dabei gilt:\n"
|
||||
"- Wenn der Artikel passt, antworte mit 'OK'.\n"
|
||||
"- Wenn der Artikel nicht passt, antworte mit 'Alternativer Wikipedia-Artikel vorgeschlagen: <URL> | X | <Begründung>'.\n"
|
||||
"- Wenn kein Artikel gefunden wurde, antworte mit 'Kein Wikipedia-Eintrag vorhanden.'\n\n")
|
||||
aggregated_prompt += "\n".join(batch_entries)
|
||||
debug_print("Aggregierter Prompt für Verifizierungs-Batch erstellt.")
|
||||
token_count = "n.v."
|
||||
@@ -512,90 +592,22 @@ def process_verification_only():
|
||||
main_sheet.update(values=[[wiki_confirm]], range_name=f"S{row_num}")
|
||||
main_sheet.update(values=[[alt_article]], range_name=f"U{row_num}")
|
||||
main_sheet.update(values=[[wiki_explanation]], range_name=f"V{row_num}")
|
||||
crm_branch = data[row_num-1][7] if len(data[row_num-1]) > 7 else "k.A."
|
||||
ext_branch = data[row_num-1][8] if len(data[row_num-1]) > 8 else "k.A."
|
||||
crm_branch = data[row_num-1][6] if len(data[row_num-1]) > 6 else "k.A."
|
||||
ext_branch = data[row_num-1][7] if len(data[row_num-1]) > 7 else "k.A."
|
||||
wiki_branch = data[row_num-1][14] if len(data[row_num-1]) > 14 else "k.A."
|
||||
wiki_cats = data[row_num-1][17] if len(data[row_num-1]) > 17 else "k.A."
|
||||
branch_result = evaluate_branche_chatgpt(crm_branch, ext_branch, wiki_branch, wiki_cats)
|
||||
main_sheet.update(values=[[branch_result["branch"]]], range_name=f"W{row_num}")
|
||||
main_sheet.update(values=[[branch_result["consistency"]]], range_name=f"Y{row_num}")
|
||||
main_sheet.update(values=[[str(token_count)]], range_name=f"AQ{row_num}")
|
||||
current_dt = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
main_sheet.update(values=[[current_dt]], range_name=f"AO{row_num}")
|
||||
main_sheet.update(values=[[Config.VERSION]], range_name=f"AP{row_num}")
|
||||
debug_print(f"Zeile {row_num} verifiziert: Antwort: {answer}")
|
||||
time.sleep(Config.RETRY_DELAY)
|
||||
debug_print("Verifizierungs-Batch abgeschlossen.")
|
||||
|
||||
# ==================== CONTACT RESEARCH (Modus 6) ====================
|
||||
def process_contact_research():
|
||||
debug_print("Starte Contact Research (Modus 6)...")
|
||||
gc = gspread.authorize(ServiceAccountCredentials.from_json_keyfile_name(
|
||||
Config.CREDENTIALS_FILE, ["https://www.googleapis.com/auth/spreadsheets"]))
|
||||
sh = gc.open_by_url(Config.SHEET_URL)
|
||||
main_sheet = sh.sheet1
|
||||
data = main_sheet.get_all_values()
|
||||
for i, row in enumerate(data[1:], start=2):
|
||||
company_name = row[1] if len(row) > 1 else ""
|
||||
search_name = row[2].strip() if len(row) > 2 and row[2].strip() not in ["", "k.A."] else company_name
|
||||
website = row[3] if len(row) > 3 else ""
|
||||
if not company_name or not website:
|
||||
continue
|
||||
count_service = count_linkedin_contacts(search_name, website, "Serviceleiter")
|
||||
count_it = count_linkedin_contacts(search_name, website, "IT-Leiter")
|
||||
count_management = count_linkedin_contacts(search_name, website, "Geschäftsführer")
|
||||
count_disponent = count_linkedin_contacts(search_name, website, "Disponent")
|
||||
current_dt = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
main_sheet.update(values=[[str(count_service)]], range_name=f"AI{i}")
|
||||
main_sheet.update(values=[[str(count_it)]], range_name=f"AJ{i}")
|
||||
main_sheet.update(values=[[str(count_management)]], range_name=f"AK{i}")
|
||||
main_sheet.update(values=[[str(count_disponent)]], range_name=f"AL{i}")
|
||||
main_sheet.update(values=[[current_dt]], range_name=f"AM{i}")
|
||||
debug_print(f"Zeile {i}: Serviceleiter {count_service}, IT-Leiter {count_it}, Management {count_management}, Disponent {count_disponent} – Contact Search Timestamp gesetzt.")
|
||||
time.sleep(Config.RETRY_DELAY * 1.5)
|
||||
debug_print("Contact Research abgeschlossen.")
|
||||
|
||||
# ==================== CONTACTS (Modus 7) ====================
|
||||
def process_contacts():
|
||||
debug_print("Starte LinkedIn-Kontaktsuche (Modus 7)...")
|
||||
gc = gspread.authorize(ServiceAccountCredentials.from_json_keyfile_name(
|
||||
Config.CREDENTIALS_FILE, ["https://www.googleapis.com/auth/spreadsheets"]))
|
||||
sh = gc.open_by_url(Config.SHEET_URL)
|
||||
try:
|
||||
contacts_sheet = sh.worksheet("Contacts")
|
||||
except gspread.exceptions.WorksheetNotFound:
|
||||
contacts_sheet = sh.add_worksheet(title="Contacts", rows="1000", cols="10")
|
||||
header = ["Firmenname", "Website", "Kurzform", "Vorname", "Nachname", "Position", "Anrede", "E-Mail"]
|
||||
contacts_sheet.update(values=[header], range_name="A1:H1")
|
||||
debug_print("Neues Blatt 'Contacts' erstellt und Header eingetragen.")
|
||||
main_sheet = sh.sheet1
|
||||
data = main_sheet.get_all_values()
|
||||
positions = ["Serviceleiter", "IT-Leiter", "Leiter After Sales", "Leiter Einsatzplanung"]
|
||||
new_rows = []
|
||||
for idx, row in enumerate(data[1:], start=2):
|
||||
company_name = row[1] if len(row) > 1 else ""
|
||||
search_name = row[2].strip() if len(row) > 2 and row[2].strip() not in ["", "k.A."] else company_name
|
||||
website = row[3] if len(row) > 3 else ""
|
||||
debug_print(f"Verarbeite Firma: '{company_name}' (Zeile {idx}), Website: '{website}'")
|
||||
if not company_name or not website:
|
||||
debug_print("Überspringe, da Firmenname oder Website fehlt.")
|
||||
continue
|
||||
for pos in positions:
|
||||
debug_print(f"Suche nach Position: '{pos}' bei '{search_name}'")
|
||||
contact = search_linkedin_contact(search_name, website, pos)
|
||||
if contact:
|
||||
debug_print(f"Kontakt gefunden: {contact}")
|
||||
new_rows.append([contact["Firmenname"], website, search_name, contact["Vorname"], contact["Nachname"], contact["Position"], "", ""])
|
||||
else:
|
||||
debug_print(f"Kein Kontakt für Position '{pos}' bei '{search_name}' gefunden.")
|
||||
if new_rows:
|
||||
last_row = len(contacts_sheet.get_all_values()) + 1
|
||||
range_str = f"A{last_row}:H{last_row + len(new_rows) - 1}"
|
||||
contacts_sheet.update(values=new_rows, range_name=range_str)
|
||||
debug_print(f"{len(new_rows)} Kontakte in 'Contacts' hinzugefügt.")
|
||||
else:
|
||||
debug_print("Keine Kontakte gefunden in der Haupttabelle.")
|
||||
|
||||
# ==================== BATCH-TOKEN-ZÄHLUNG (Modus 8) ====================
|
||||
# ==================== NEUER MODUS 8: BATCH-PROZESSING MIT TOKEN-ZÄHLUNG ====================
|
||||
def process_batch_token_count(batch_size=10):
|
||||
import tiktoken
|
||||
def count_tokens(text, model="gpt-3.5-turbo"):
|
||||
@@ -633,7 +645,7 @@ def process_batch_token_count(batch_size=10):
|
||||
time.sleep(Config.RETRY_DELAY)
|
||||
debug_print("Batch-Token-Zählung abgeschlossen.")
|
||||
|
||||
# ==================== ALIGNMENT DEMO FÜR HAUPTBLATT & CONTACTS ====================
|
||||
# ==================== NEUER MODUS: ALIGNMENT DEMO (Hauptblatt und Contacts) ====================
|
||||
def alignment_demo_full():
|
||||
alignment_demo(GoogleSheetHandler().sheet)
|
||||
gc = gspread.authorize(ServiceAccountCredentials.from_json_keyfile_name(
|
||||
@@ -644,12 +656,229 @@ def alignment_demo_full():
|
||||
except gspread.exceptions.WorksheetNotFound:
|
||||
contacts_sheet = sh.add_worksheet(title="Contacts", rows="1000", cols="10")
|
||||
header = ["Firmenname", "Website", "Kurzform", "Vorname", "Nachname", "Position", "Anrede", "E-Mail"]
|
||||
contacts_sheet.update(values=[header], range_name="A1:H1")
|
||||
contacts_sheet.update("A1:H1", [header])
|
||||
debug_print("Neues Blatt 'Contacts' erstellt und Header eingetragen.")
|
||||
alignment_demo(contacts_sheet)
|
||||
debug_print("Alignment-Demo für Hauptblatt und Contacts abgeschlossen.")
|
||||
|
||||
# ==================== GOOGLE SHEET HANDLER (Hauptdaten) ====================
|
||||
# ==================== ALIGNMENT DEMO (Hauptblatt) ====================
|
||||
def alignment_demo(sheet):
|
||||
new_headers = [
|
||||
"Spalte A (ReEval Flag)",
|
||||
"Spalte B (Firmenname)",
|
||||
"Spalte C (Kurzform des Firmennamens)",
|
||||
"Spalte D (Website)",
|
||||
"Spalte E (Ort)",
|
||||
"Spalte F (Beschreibung)",
|
||||
"Spalte G (Aktuelle Branche)",
|
||||
"Spalte H (Beschreibung Branche extern)",
|
||||
"Spalte I (Anzahl Techniker CRM)",
|
||||
"Spalte J (Umsatz CRM)",
|
||||
"Spalte K (Anzahl Mitarbeiter CRM)",
|
||||
"Spalte L (Vorschlag Wiki URL)",
|
||||
"Spalte M (Wikipedia URL)",
|
||||
"Spalte N (Wikipedia Absatz)",
|
||||
"Spalte O (Wikipedia Branche)",
|
||||
"Spalte P (Wikipedia Umsatz)",
|
||||
"Spalte Q (Wikipedia Mitarbeiter)",
|
||||
"Spalte R (Wikipedia Kategorien)",
|
||||
"Spalte S (Konsistenzprüfung)",
|
||||
"Spalte T (Begründung bei Inkonsistenz)",
|
||||
"Spalte U (Vorschlag Wiki Artikel ChatGPT)",
|
||||
"Spalte V (Begründung bei Abweichung)",
|
||||
"Spalte W (Vorschlag neue Branche)",
|
||||
"Spalte X (Konsistenzprüfung Branche)",
|
||||
"Spalte Y (Begründung Abweichung Branche)",
|
||||
"Spalte Z (Timestamp Verifizierung)",
|
||||
"Spalte AA (Version)"
|
||||
]
|
||||
header_range = "A11200:AA11200"
|
||||
sheet.update(values=[new_headers], range_name=header_range)
|
||||
print("Alignment-Demo abgeschlossen: Neue Spaltenüberschriften in Zeile 11200 geschrieben.")
|
||||
|
||||
# ==================== WIKIPEDIA SCRAPER ====================
|
||||
class WikipediaScraper:
|
||||
def __init__(self):
|
||||
wikipedia.set_lang(Config.LANG)
|
||||
def _get_full_domain(self, website):
|
||||
if not website:
|
||||
return ""
|
||||
website = website.lower().strip()
|
||||
website = re.sub(r'^https?:\/\/', '', website)
|
||||
website = re.sub(r'^www\.', '', website)
|
||||
return website.split('/')[0]
|
||||
def _generate_search_terms(self, company_name, website):
|
||||
terms = []
|
||||
full_domain = self._get_full_domain(website)
|
||||
if full_domain:
|
||||
terms.append(full_domain)
|
||||
normalized_name = normalize_company_name(company_name)
|
||||
candidate = " ".join(normalized_name.split()[:2]).strip()
|
||||
if candidate and candidate not in terms:
|
||||
terms.append(candidate)
|
||||
if normalized_name and normalized_name not in terms:
|
||||
terms.append(normalized_name)
|
||||
debug_print(f"Generierte Suchbegriffe: {terms}")
|
||||
return terms
|
||||
def _validate_article(self, page, company_name, website):
|
||||
full_domain = self._get_full_domain(website)
|
||||
domain_found = False
|
||||
if full_domain:
|
||||
try:
|
||||
html_raw = requests.get(page.url).text
|
||||
soup = BeautifulSoup(html_raw, Config.HTML_PARSER)
|
||||
infobox = soup.find('table', class_=lambda c: c and 'infobox' in c.lower())
|
||||
if infobox:
|
||||
links = infobox.find_all('a', href=True)
|
||||
for link in links:
|
||||
href = link.get('href').lower()
|
||||
if href.startswith('/wiki/datei:'):
|
||||
continue
|
||||
if full_domain in href:
|
||||
debug_print(f"Definitiver Link-Match in Infobox gefunden: {href}")
|
||||
domain_found = True
|
||||
break
|
||||
if not domain_found and hasattr(page, 'externallinks'):
|
||||
for ext_link in page.externallinks:
|
||||
if full_domain in ext_link.lower():
|
||||
debug_print(f"Definitiver Link-Match in externen Links gefunden: {ext_link}")
|
||||
domain_found = True
|
||||
break
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler beim Extrahieren von Links: {str(e)}")
|
||||
normalized_title = normalize_company_name(page.title)
|
||||
normalized_company = normalize_company_name(company_name)
|
||||
similarity = SequenceMatcher(None, normalized_title, normalized_company).ratio()
|
||||
debug_print(f"Ähnlichkeit (normalisiert): {similarity:.2f} ({normalized_title} vs {normalized_company})")
|
||||
threshold = 0.60 if domain_found else Config.SIMILARITY_THRESHOLD
|
||||
return similarity >= threshold
|
||||
def extract_first_paragraph(self, page_url):
|
||||
try:
|
||||
response = requests.get(page_url)
|
||||
soup = BeautifulSoup(response.text, Config.HTML_PARSER)
|
||||
paragraphs = soup.find_all('p')
|
||||
for p in paragraphs:
|
||||
text = clean_text(p.get_text())
|
||||
if len(text) > 50:
|
||||
return text
|
||||
return "k.A."
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler beim Extrahieren des ersten Absatzes: {e}")
|
||||
return "k.A."
|
||||
def extract_categories(self, soup):
|
||||
cat_div = soup.find('div', id="mw-normal-catlinks")
|
||||
if cat_div:
|
||||
ul = cat_div.find('ul')
|
||||
if ul:
|
||||
cats = [clean_text(li.get_text()) for li in ul.find_all('li')]
|
||||
return ", ".join(cats)
|
||||
return "k.A."
|
||||
def _extract_infobox_value(self, soup, target):
|
||||
infobox = soup.find('table', class_=lambda c: c and any(kw in c.lower() for kw in ['infobox', 'vcard', 'unternehmen']))
|
||||
if not infobox:
|
||||
return "k.A."
|
||||
keywords_map = {
|
||||
'branche': ['branche', 'industrie', 'tätigkeit', 'geschäftsfeld', 'sektor', 'produkte', 'leistungen', 'aktivitäten', 'wirtschaftszweig'],
|
||||
'umsatz': ['umsatz', 'jahresumsatz', 'konzernumsatz', 'gesamtumsatz', 'erlöse', 'umsatzerlöse', 'einnahmen', 'ergebnis', 'jahresergebnis'],
|
||||
'mitarbeiter': ['mitarbeiter', 'beschäftigte', 'personal', 'mitarbeiterzahl', 'angestellte', 'belegschaft', 'personalstärke']
|
||||
}
|
||||
keywords = keywords_map.get(target, [])
|
||||
for row in infobox.find_all('tr'):
|
||||
header = row.find('th')
|
||||
if header:
|
||||
header_text = clean_text(header.get_text()).lower()
|
||||
if any(kw in header_text for kw in keywords):
|
||||
value = row.find('td')
|
||||
if value:
|
||||
raw_value = clean_text(value.get_text())
|
||||
if target == 'branche':
|
||||
clean_val = re.sub(r'\[.*?\]|\(.*?\)', '', raw_value)
|
||||
return ' '.join(clean_val.split()).strip()
|
||||
if target == 'umsatz':
|
||||
return extract_numeric_value(raw_value, is_umsatz=True)
|
||||
if target == 'mitarbeiter':
|
||||
return extract_numeric_value(raw_value, is_umsatz=False)
|
||||
return "k.A."
|
||||
def extract_full_infobox(self, soup):
|
||||
infobox = soup.find('table', class_=lambda c: c and any(kw in c.lower() for kw in ['infobox', 'vcard', 'unternehmen']))
|
||||
if not infobox:
|
||||
return "k.A."
|
||||
return clean_text(infobox.get_text(separator=' | '))
|
||||
def extract_fields_from_infobox_text(self, infobox_text, field_names):
|
||||
result = {}
|
||||
tokens = [token.strip() for token in infobox_text.split("|") if token.strip()]
|
||||
for i, token in enumerate(tokens):
|
||||
for field in field_names:
|
||||
if field.lower() in token.lower():
|
||||
j = i + 1
|
||||
while j < len(tokens) and not tokens[j]:
|
||||
j += 1
|
||||
result[field] = tokens[j] if j < len(tokens) else "k.A."
|
||||
return result
|
||||
def extract_company_data(self, page_url):
|
||||
if not page_url:
|
||||
return {
|
||||
'url': 'k.A.',
|
||||
'first_paragraph': 'k.A.',
|
||||
'branche': 'k.A.',
|
||||
'umsatz': 'k.A.',
|
||||
'mitarbeiter': 'k.A.',
|
||||
'categories': 'k.A.',
|
||||
'full_infobox': 'k.A.'
|
||||
}
|
||||
try:
|
||||
response = requests.get(page_url)
|
||||
soup = BeautifulSoup(response.text, Config.HTML_PARSER)
|
||||
full_infobox = self.extract_full_infobox(soup)
|
||||
extracted_fields = self.extract_fields_from_infobox_text(full_infobox, ['Branche', 'Umsatz', 'Mitarbeiter'])
|
||||
raw_branche = extracted_fields.get('Branche', self._extract_infobox_value(soup, 'branche'))
|
||||
raw_umsatz = extracted_fields.get('Umsatz', self._extract_infobox_value(soup, 'umsatz'))
|
||||
raw_mitarbeiter = extracted_fields.get('Mitarbeiter', self._extract_infobox_value(soup, 'mitarbeiter'))
|
||||
umsatz_val = extract_numeric_value(raw_umsatz, is_umsatz=True)
|
||||
mitarbeiter_val = extract_numeric_value(raw_mitarbeiter, is_umsatz=False)
|
||||
categories_val = self.extract_categories(soup)
|
||||
first_paragraph = self.extract_first_paragraph(page_url)
|
||||
return {
|
||||
'url': page_url,
|
||||
'first_paragraph': first_paragraph,
|
||||
'branche': raw_branche,
|
||||
'umsatz': umsatz_val,
|
||||
'mitarbeiter': mitarbeiter_val,
|
||||
'categories': categories_val,
|
||||
'full_infobox': full_infobox
|
||||
}
|
||||
except Exception as e:
|
||||
debug_print(f"Extraktionsfehler: {str(e)}")
|
||||
return {
|
||||
'url': 'k.A.',
|
||||
'first_paragraph': 'k.A.',
|
||||
'branche': 'k.A.',
|
||||
'umsatz': 'k.A.',
|
||||
'mitarbeiter': 'k.A.',
|
||||
'categories': 'k.A.',
|
||||
'full_infobox': 'k.A.'
|
||||
}
|
||||
@retry_on_failure
|
||||
def search_company_article(self, company_name, website):
|
||||
search_terms = self._generate_search_terms(company_name, website)
|
||||
for term in search_terms:
|
||||
try:
|
||||
results = wikipedia.search(term, results=Config.WIKIPEDIA_SEARCH_RESULTS)
|
||||
debug_print(f"Suchergebnisse für '{term}': {results}")
|
||||
for title in results:
|
||||
try:
|
||||
page = wikipedia.page(title, auto_suggest=False)
|
||||
if self._validate_article(page, company_name, website):
|
||||
return page
|
||||
except (wikipedia.exceptions.DisambiguationError, wikipedia.exceptions.PageError) as e:
|
||||
debug_print(f"Seitenfehler: {str(e)}")
|
||||
continue
|
||||
except Exception as e:
|
||||
debug_print(f"Suchfehler: {str(e)}")
|
||||
continue
|
||||
return None
|
||||
|
||||
# ==================== GOOGLE SHEET HANDLER ====================
|
||||
class GoogleSheetHandler:
|
||||
def __init__(self):
|
||||
self.sheet = None
|
||||
@@ -661,10 +890,10 @@ class GoogleSheetHandler:
|
||||
self.sheet = gspread.authorize(creds).open_by_url(Config.SHEET_URL).sheet1
|
||||
self.sheet_values = self.sheet.get_all_values()
|
||||
def get_start_index(self):
|
||||
filled_n = [row[39] if len(row) > 39 else '' for row in self.sheet_values[1:]]
|
||||
filled_n = [row[13] if len(row) > 13 else '' for row in self.sheet_values[1:]]
|
||||
return next((i + 1 for i, v in enumerate(filled_n, start=1) if not str(v).strip()), len(filled_n) + 1)
|
||||
|
||||
# ==================== DATA PROCESSOR (Regulärer Modus) ====================
|
||||
# ==================== DATA PROCESSOR ====================
|
||||
class DataProcessor:
|
||||
def __init__(self):
|
||||
self.sheet_handler = GoogleSheetHandler()
|
||||
@@ -674,10 +903,10 @@ class DataProcessor:
|
||||
print("Re-Evaluierungsmodus: Verarbeitung aller Zeilen mit 'x' in Spalte A.")
|
||||
for i, row in enumerate(self.sheet_handler.sheet_values[1:], start=2):
|
||||
if row[0].strip().lower() == "x":
|
||||
self._process_single_row(i, row, force_all=True)
|
||||
self._process_single_row(i, row)
|
||||
elif MODE == "3":
|
||||
print("Alignment-Demo-Modus: Hauptblatt und Contacts aktualisieren.")
|
||||
alignment_demo_full()
|
||||
print("Alignment-Demo-Modus: Schreibe neue Spaltenüberschriften in Zeile 11200.")
|
||||
alignment_demo(self.sheet_handler.sheet)
|
||||
elif MODE == "4":
|
||||
for i, row in enumerate(self.sheet_handler.sheet_values[1:], start=2):
|
||||
if len(row) <= 39 or row[39].strip() == "":
|
||||
@@ -688,9 +917,8 @@ class DataProcessor:
|
||||
self._process_single_row(i, row, process_wiki=False, process_chatgpt=True)
|
||||
elif MODE == "51":
|
||||
for i, row in enumerate(self.sheet_handler.sheet_values[1:], start=2):
|
||||
# Hier: Nur Zeilen ohne Verifizierungstimestamp (Spalte Y, z.B.) werden verarbeitet
|
||||
if len(row) <= 25 or row[24].strip() == "":
|
||||
_process_verification_row(i, row)
|
||||
self._process_verification_row(i, row)
|
||||
elif MODE == "8":
|
||||
process_batch_token_count()
|
||||
else:
|
||||
@@ -704,80 +932,154 @@ class DataProcessor:
|
||||
break
|
||||
self._process_single_row(i, row)
|
||||
rows_processed += 1
|
||||
def _process_single_row(self, row_num, row_data, force_all=False, process_wiki=True, process_chatgpt=True):
|
||||
def _process_single_row(self, row_num, row_data, process_wiki=True, process_chatgpt=True):
|
||||
company_name = row_data[1] if len(row_data) > 1 else ""
|
||||
website = row_data[3] if len(row_data) > 3 else ""
|
||||
website = row_data[2] if len(row_data) > 2 else ""
|
||||
wiki_update_range = f"K{row_num}:Q{row_num}"
|
||||
chatgpt_range = f"AF{row_num}"
|
||||
abgleich_range = f"AG{row_num}"
|
||||
valid_range = f"R{row_num}"
|
||||
dt_range = f"AH{row_num}"
|
||||
ver_range = f"AI{row_num}"
|
||||
print(f"\n[{datetime.now().strftime('%H:%M:%S')}] Verarbeite Zeile {row_num}: {company_name}")
|
||||
current_dt = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
# Wiki-Auswertung (Spalten L bis R, Timestamp AO)
|
||||
if force_all or process_wiki:
|
||||
if process_wiki:
|
||||
if len(row_data) <= 39 or row_data[39].strip() == "":
|
||||
if len(row_data) > 11 and row_data[11].strip() not in ["", "k.A."]:
|
||||
wiki_url = row_data[11].strip()
|
||||
if len(row_data) > 10 and row_data[10].strip() not in ["", "k.A."]:
|
||||
wiki_url = row_data[10].strip()
|
||||
try:
|
||||
wiki_data = self.wiki_scraper.extract_company_data(wiki_url)
|
||||
company_data = self.wiki_scraper.extract_company_data(wiki_url)
|
||||
except Exception as e:
|
||||
debug_print(f"Fehler beim Laden des vorgeschlagenen Wikipedia-Artikels: {e}")
|
||||
article = self.wiki_scraper.search_company_article(company_name, website)
|
||||
wiki_data = self.wiki_scraper.extract_company_data(article.url) if article else {
|
||||
company_data = self.wiki_scraper.extract_company_data(article.url) if article else {
|
||||
'url': 'k.A.', 'first_paragraph': 'k.A.', 'branche': 'k.A.',
|
||||
'umsatz': 'k.A.', 'mitarbeiter': 'k.A.', 'categories': 'k.A.',
|
||||
'full_infobox': 'k.A.'
|
||||
}
|
||||
else:
|
||||
article = self.wiki_scraper.search_company_article(company_name, website)
|
||||
wiki_data = self.wiki_scraper.extract_company_data(article.url) if article else {
|
||||
company_data = self.wiki_scraper.extract_company_data(article.url) if article else {
|
||||
'url': 'k.A.', 'first_paragraph': 'k.A.', 'branche': 'k.A.',
|
||||
'umsatz': 'k.A.', 'mitarbeiter': 'k.A.', 'categories': 'k.A.',
|
||||
'full_infobox': 'k.A.'
|
||||
}
|
||||
wiki_values = [
|
||||
row_data[11] if len(row_data) > 11 and row_data[11].strip() not in ["", "k.A."] else "k.A.",
|
||||
wiki_data.get('url', 'k.A.'),
|
||||
wiki_data.get('first_paragraph', 'k.A.'),
|
||||
wiki_data.get('branche', 'k.A.'),
|
||||
wiki_data.get('umsatz', 'k.A.'),
|
||||
wiki_data.get('mitarbeiter', 'k.A.'),
|
||||
wiki_data.get('categories', 'k.A.')
|
||||
row_data[10] if len(row_data) > 10 and row_data[10].strip() not in ["", "k.A."] else "k.A.",
|
||||
company_data.get('url', 'k.A.'),
|
||||
company_data.get('first_paragraph', 'k.A.'),
|
||||
company_data.get('branche', 'k.A.'),
|
||||
company_data.get('umsatz', 'k.A.'),
|
||||
company_data.get('mitarbeiter', 'k.A.'),
|
||||
company_data.get('categories', 'k.A.')
|
||||
]
|
||||
self.sheet_handler.sheet.update(values=[wiki_values], range_name=f"L{row_num}:R{row_num}")
|
||||
self.sheet_handler.sheet.update(values=[[current_dt]], range_name=f"AO{row_num}")
|
||||
self.sheet_handler.sheet.update(values=[wiki_values], range_name=wiki_update_range)
|
||||
wait_for_sheet_update(self.sheet_handler.sheet, f"K{row_num}", wiki_values[0])
|
||||
self.sheet_handler.sheet.update(values=[[current_dt]], range_name=dt_range)
|
||||
else:
|
||||
debug_print(f"Zeile {row_num}: Wikipedia-Timestamp bereits gesetzt – überspringe Wiki-Auswertung.")
|
||||
# ChatGPT-Auswertung (Branche & FSM, etc. – Spalten R, AG, Y, Z, AE, AF; Timestamp in AO, Version in AP)
|
||||
if force_all or process_chatgpt:
|
||||
if process_chatgpt:
|
||||
if len(row_data) <= 40 or row_data[40].strip() == "":
|
||||
crm_umsatz = row_data[9] if len(row_data) > 9 else "k.A."
|
||||
abgleich_result = compare_umsatz_values(crm_umsatz, wiki_data.get('umsatz', 'k.A.') if 'wiki_data' in locals() else "k.A.")
|
||||
self.sheet_handler.sheet.update(values=[[abgleich_result]], range_name=f"AG{row_num}")
|
||||
crm_data = ";".join(row_data[1:11])
|
||||
wiki_data_str = ";".join(row_data[11:18])
|
||||
crm_umsatz = row_data[8] if len(row_data) > 8 else "k.A."
|
||||
abgleich_result = compare_umsatz_values(crm_umsatz, company_data.get('umsatz', 'k.A.') if 'company_data' in locals() else "k.A.")
|
||||
self.sheet_handler.sheet.update(values=[[abgleich_result]], range_name=abgleich_range)
|
||||
crm_data = ";".join(row_data[1:10])
|
||||
wiki_data_str = ";".join(row_data[11:17])
|
||||
valid_result = validate_article_with_chatgpt(crm_data, wiki_data_str)
|
||||
self.sheet_handler.sheet.update(values=[[valid_result]], range_name=f"R{row_num}")
|
||||
fsm_result = evaluate_fsm_suitability(company_name, wiki_data if 'wiki_data' in locals() else {})
|
||||
self.sheet_handler.sheet.update(values=[[valid_result]], range_name=valid_range)
|
||||
fsm_result = evaluate_fsm_suitability(company_name, company_data if 'company_data' in locals() else {})
|
||||
self.sheet_handler.sheet.update(values=[[fsm_result["suitability"]]], range_name=f"Y{row_num}")
|
||||
self.sheet_handler.sheet.update(values=[[fsm_result["justification"]]], range_name=f"Z{row_num}")
|
||||
st_estimate = evaluate_servicetechnicians_estimate(company_name, wiki_data if 'wiki_data' in locals() else {})
|
||||
self.sheet_handler.sheet.update(values=[[st_estimate]], range_name=f"AE{row_num}")
|
||||
internal_value = row_data[8] if len(row_data) > 8 else "k.A."
|
||||
st_estimate = evaluate_servicetechnicians_estimate(company_name, company_data if 'company_data' in locals() else {})
|
||||
self.sheet_handler.sheet.update(values=[[st_estimate]], range_name=f"AD{row_num}")
|
||||
internal_value = row_data[7] if len(row_data) > 7 else "k.A."
|
||||
internal_category = map_internal_technicians(internal_value) if internal_value != "k.A." else "k.A."
|
||||
if internal_category != "k.A." and st_estimate != internal_category:
|
||||
explanation = evaluate_servicetechnicians_explanation(company_name, st_estimate, wiki_data if 'wiki_data' in locals() else {})
|
||||
explanation = evaluate_servicetechnicians_explanation(company_name, st_estimate, company_data if 'company_data' in locals() else {})
|
||||
discrepancy = explanation
|
||||
else:
|
||||
discrepancy = "ok"
|
||||
self.sheet_handler.sheet.update(values=[[discrepancy]], range_name=f"AF{row_num}")
|
||||
self.sheet_handler.sheet.update(values=[[current_dt]], range_name=f"AO{row_num}")
|
||||
self.sheet_handler.sheet.update(values=[[discrepancy]], range_name=f"AE{row_num}")
|
||||
self.sheet_handler.sheet.update(values=[[current_dt]], range_name=chatgpt_range)
|
||||
else:
|
||||
debug_print(f"Zeile {row_num}: ChatGPT-Timestamp bereits gesetzt – überspringe ChatGPT-Auswertung.")
|
||||
self.sheet_handler.sheet.update(values=[[Config.VERSION]], range_name=f"AP{row_num}")
|
||||
debug_print(f"✅ Aktualisiert: URL: {(wiki_data.get('url', 'k.A.') if 'wiki_data' in locals() else 'k.A.')}, "
|
||||
f"Branche: {(wiki_data.get('branche', 'k.A.') if 'wiki_data' in locals() else 'k.A.')}, "
|
||||
self.sheet_handler.sheet.update(values=[[current_dt]], range_name=ver_range)
|
||||
self.sheet_handler.sheet.update(values=[[Config.VERSION]], range_name=ver_range)
|
||||
debug_print(f"✅ Aktualisiert: URL: {(company_data.get('url', 'k.A.') if 'company_data' in locals() else 'k.A.')}, "
|
||||
f"Branche: {(company_data.get('branche', 'k.A.') if 'company_data' in locals() else 'k.A.')}, "
|
||||
f"Umsatz-Abgleich: {abgleich_result if 'abgleich_result' in locals() else 'k.A.'}, "
|
||||
f"Validierung: {valid_result if 'valid_result' in locals() else 'k.A.'}, "
|
||||
f"FSM: {fsm_result['suitability'] if 'fsm_result' in locals() else 'k.A.'}, "
|
||||
f"Servicetechniker-Schätzung: {st_estimate if 'st_estimate' in locals() else 'k.A.'}")
|
||||
time.sleep(Config.RETRY_DELAY)
|
||||
|
||||
# ==================== NEUER MODUS 6: CONTACT RESEARCH (via SerpAPI) ====================
|
||||
def process_contact_research():
|
||||
debug_print("Starte Contact Research (Modus 6)...")
|
||||
gc = gspread.authorize(ServiceAccountCredentials.from_json_keyfile_name(
|
||||
Config.CREDENTIALS_FILE, ["https://www.googleapis.com/auth/spreadsheets"]))
|
||||
sh = gc.open_by_url(Config.SHEET_URL)
|
||||
main_sheet = sh.sheet1
|
||||
data = main_sheet.get_all_values()
|
||||
for i, row in enumerate(data[1:], start=2):
|
||||
company_name = row[1] if len(row) > 1 else ""
|
||||
search_name = row[2].strip() if len(row) > 2 and row[2].strip() not in ["", "k.A."] else company_name
|
||||
website = row[3] if len(row) > 3 else ""
|
||||
if not company_name or not website:
|
||||
continue
|
||||
count_service = count_linkedin_contacts(search_name, website, "Serviceleiter")
|
||||
count_it = count_linkedin_contacts(search_name, website, "IT-Leiter")
|
||||
count_management = count_linkedin_contacts(search_name, website, "Geschäftsführer")
|
||||
count_disponent = count_linkedin_contacts(search_name, website, "Disponent")
|
||||
current_dt = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
main_sheet.update(values=[[str(count_service)]], range_name=f"AI{i}")
|
||||
main_sheet.update(values=[[str(count_it)]], range_name=f"AJ{i}")
|
||||
main_sheet.update(values=[[str(count_management)]], range_name=f"AK{i}")
|
||||
main_sheet.update(values=[[str(count_disponent)]], range_name=f"AL{i}")
|
||||
main_sheet.update(values=[[current_dt]], range_name=f"AM{i}")
|
||||
debug_print(f"Zeile {i}: Serviceleiter {count_service}, IT-Leiter {count_it}, Management {count_management}, Disponent {count_disponent} – Contact Search Timestamp gesetzt.")
|
||||
time.sleep(Config.RETRY_DELAY * 1.5)
|
||||
debug_print("Contact Research abgeschlossen.")
|
||||
|
||||
# ==================== NEUER MODUS: CONTACTS (LinkedIn) ====================
|
||||
def process_contacts():
|
||||
debug_print("Starte LinkedIn-Kontaktsuche...")
|
||||
gc = gspread.authorize(ServiceAccountCredentials.from_json_keyfile_name(
|
||||
Config.CREDENTIALS_FILE, ["https://www.googleapis.com/auth/spreadsheets"]))
|
||||
sh = gc.open_by_url(Config.SHEET_URL)
|
||||
try:
|
||||
contacts_sheet = sh.worksheet("Contacts")
|
||||
except gspread.exceptions.WorksheetNotFound:
|
||||
contacts_sheet = sh.add_worksheet(title="Contacts", rows="1000", cols="10")
|
||||
header = ["Firmenname", "Website", "Kurzform", "Vorname", "Nachname", "Position", "Anrede", "E-Mail"]
|
||||
contacts_sheet.update("A1:G1", [header])
|
||||
debug_print("Neues Blatt 'Contacts' erstellt und Header eingetragen.")
|
||||
main_sheet = sh.sheet1
|
||||
data = main_sheet.get_all_values()
|
||||
positions = ["Serviceleiter", "IT-Leiter", "Leiter After Sales", "Leiter Einsatzplanung"]
|
||||
new_rows = []
|
||||
for idx, row in enumerate(data[1:], start=2):
|
||||
company_name = row[1] if len(row) > 1 else ""
|
||||
search_name = row[2].strip() if len(row) > 2 and row[2].strip() not in ["", "k.A."] else company_name
|
||||
website = row[3] if len(row) > 3 else ""
|
||||
if not company_name or not website:
|
||||
continue
|
||||
for pos in positions:
|
||||
debug_print(f"Suche nach Position: '{pos}' bei '{search_name}'")
|
||||
contact = search_linkedin_contact(company_name, website, pos)
|
||||
if contact:
|
||||
debug_print(f"Kontakt gefunden: {contact}")
|
||||
new_rows.append([contact["Firmenname"], contact["Website"], search_name, contact["Vorname"], contact["Nachname"], contact["Position"], "", ""])
|
||||
else:
|
||||
debug_print(f"Kein Kontakt für Position '{pos}' bei '{search_name}' gefunden.")
|
||||
if new_rows:
|
||||
last_row = len(contacts_sheet.get_all_values()) + 1
|
||||
range_str = f"A{last_row}:G{last_row + len(new_rows) - 1}"
|
||||
contacts_sheet.update(range_str, new_rows)
|
||||
debug_print(f"{len(new_rows)} Kontakte in 'Contacts' hinzugefügt.")
|
||||
else:
|
||||
debug_print("Keine Kontakte gefunden.")
|
||||
|
||||
# ==================== MAIN PROGRAMM ====================
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
@@ -787,19 +1089,23 @@ if __name__ == "__main__":
|
||||
args = parser.parse_args()
|
||||
if not args.mode:
|
||||
print("Modi:")
|
||||
print("1 = Regulärer Modus")
|
||||
print("1 = regulärer Modus")
|
||||
print("2 = Re-Evaluierungsmodus (nur Zeilen mit 'x' in Spalte A)")
|
||||
print("3 = Alignment-Demo (Hauptblatt & Contacts)")
|
||||
print("3 = Alignment-Demo (Header in Hauptblatt und Contacts)")
|
||||
print("4 = Nur Wikipedia-Suche (Zeilen ohne Wikipedia-Timestamp)")
|
||||
print("5 = Nur ChatGPT-Bewertung (Zeilen ohne ChatGPT-Timestamp)")
|
||||
print("6 = Contact Research (via SerpAPI)")
|
||||
print("7 = Contacts (LinkedIn)")
|
||||
print("8 = Batch-Token-Zählung")
|
||||
print("51 = Nur Verifizierung (gezielte Branchen- & FSM-Evaluierung)")
|
||||
print("51 = Nur Verifizierung (Wikipedia + Brancheneinordnung)")
|
||||
args.mode = input("Wählen Sie den Modus: ").strip()
|
||||
MODE = args.mode
|
||||
if MODE == "1":
|
||||
num_rows = args.num_rows if args.num_rows > 0 else int(input("Wieviele Zeilen sollen überprüft werden? "))
|
||||
try:
|
||||
num_rows = args.num_rows if args.num_rows > 0 else int(input("Wieviele Zeilen sollen überprüft werden? "))
|
||||
except Exception as e:
|
||||
print("Ungültige Eingabe. Bitte eine Zahl eingeben.")
|
||||
exit(1)
|
||||
processor = DataProcessor()
|
||||
processor.process_rows(num_rows)
|
||||
elif MODE in ["2", "3"]:
|
||||
@@ -816,7 +1122,10 @@ if __name__ == "__main__":
|
||||
if len(row) <= 40 or row[40].strip() == "":
|
||||
processor._process_single_row(i, row, process_wiki=False, process_chatgpt=True)
|
||||
elif MODE == "51":
|
||||
process_verification_only()
|
||||
processor = DataProcessor()
|
||||
for i, row in enumerate(processor.sheet_handler.sheet_values[1:], start=2):
|
||||
if len(row) <= 25 or row[24].strip() == "":
|
||||
processor._process_verification_row(i, row)
|
||||
elif MODE == "6":
|
||||
process_contact_research()
|
||||
elif MODE == "7":
|
||||
|
||||
Reference in New Issue
Block a user