This commit addresses the issue of duplicate jobs being created by the SuperOffice connector.
The root cause was identified as a race condition where SuperOffice would send multiple webhooks in quick succession for the same entity, leading to multiple identical jobs in the queue.
The solution involves several layers of improvement:
1. **Ingress De-duplication:** The now checks for existing jobs for the same entity *before* adding a new job to the queue. This is the primary fix and prevents duplicates at the source.
2. **DB Schema Enhancement:** The table schema in was extended with an column to allow for reliable and efficient checking of duplicate entities.
3. **Improved Logging:** The log messages in for job retries (e.g., when waiting for the Company Explorer) have been made more descriptive to avoid confusion and false alarms.
SuperOffice sends two 'contact.created' webhooks for a single new contact. This caused the connector to process the same entity twice, leading to duplicate entries and logs.
This commit introduces a de-duplication shield in the worker:
- A new method is added to to check for jobs with the same company name that are either 'PROCESSING' or 'COMPLETED' within the last 5 minutes.
- The worker now fetches the company name upon receiving a job, updates the job record with the name, and then calls the new de-duplication method.
- If a duplicate event is detected, the job is skipped, preventing redundant processing.
