<turbo-stream action="update" target="wiki_card"><template>

<h1>Duplicates in Integration: Why They Happen and How to Actually Handle Them</h1>
<div class="trix-content"><p>Before you can fix duplicates, you need to understand how they get created in the first place. There are a few usual suspects:<br><br></p><p><strong>Race conditions.</strong> System A creates a record and immediately syncs it to System B. System B processes the sync and creates the record. But before it finishes, a user in System B has already created the same record manually. Now you have two.<br><br></p><p><strong>Missing or mismatched keys.</strong> Your integration uses Email to match contacts, but System A stores john.doe@company.com and System B stores John.Doe@Company.com. The lookup fails, a new record gets created, and now you have twins.<br><br></p><p><strong>Bidirectional sync without a lock.</strong> Record gets updated in System A → syncs to System B → triggers an update event in System B → syncs back to System A → repeat. If you're not careful, this loop also creates phantom records along the way.<br><br></p><p><strong>Initial load collisions.</strong> You're migrating historical data into a new system that already has some records. Nobody mapped the existing data to the incoming data. Duplicates at scale.<br><br></p><p><strong>System restarts and retries.</strong> A sync fails halfway through, the job retries, and the record gets written twice. Especially common when your integration platform doesn't track what was already committed.<br><br><br></p><h3><strong>The Matching Strategy</strong></h3><p><br></p><p>Before you write a single mapping or toggle a single field, you need to define how your integration will recognize that two records in different systems represent the same real-world entity.<br><br></p><p>This is your <strong>matching key</strong> and it's the single most important design decision you'll make on a duplicate-prone integration.<br><br></p><p>Common options:<br><br></p><ul><li><strong>External ID / Cross-reference ID</strong> : System A stores System B's ID (and vice versa). This is the gold standard. Once a record is linked, it will never be confused for another one.</li><li><strong>Business key</strong> : A natural identifier that both systems share: email address, VAT number, order number, SKU. Works well when the field is truly unique and consistently formatted.</li><li><strong>Composite key</strong> : A combination of fields (e.g., Company Name + Order Number + Line Number).</li><li><strong>Fuzzy matching</strong> : Similarity algorithms that catch near-matches and can be implemented in the systems that should be integrated.&nbsp;</li></ul><p><br></p><h3><strong>The Architecture Choices That Prevent Duplicates</strong></h3><p><br></p><p>Duplicate handling isn't just a data problem. It's an architecture problem. These patterns help:<br><br></p><p><strong>1. Upsert over Insert<br></strong><br></p><p>Never blindly insert. Always try to match first, then update if found, insert if not. Most integration platforms and APIs support upsert natively.<br><br></p><p><strong><br>2. Write Back the ID<br></strong><br></p><p>When System A creates a record in System B, System B returns an ID. Write that ID back into System A immediately. This creates a permanent, unambiguous link between the two records. Future syncs follow the link instead of trying to re-match by field values. This is especially important in bidirectional integrations. If you skip this step, every sync is a potential duplicate.<br><br></p><p><strong><br>3. Single Direction of Truth (at least per entity)<br></strong><br></p><p>Decide which system <em>owns</em> each record type. Customers might be created in the CRM and pushed to the ERP. Orders might flow the other way. Pick a direction and enforce it. When both systems can create the same type of record independently and sync bidirectionally without guardrails, duplicates are almost inevitable.<br><br></p><p><strong><br>4. Deduplication at the Gate<br></strong><br></p><p>If you have a system that can receive records from multiple sources, build a deduplication check at the entry point before the record is written. Check for an existing match. If found, merge or reject with a clear log message. Don't let the record through and deal with the mess later.<br><br></p><p><strong><br>5. Idempotent Sync Jobs<br></strong><br></p><p>Design your integration jobs so that running them twice produces the same result as running them once. This means: if a record was already synced, re-syncing it updates it in place rather than creating a new one. Idempotency is your safety net for retries, failures, and reruns.<br><br></p><p><strong><br>The Summary<br></strong><br></p><p>Duplicates in integrations are a design problem more than a technical one. They appear when there's ambiguity about identity, ownership, or sequencing and they tend to compound over time.<br><br></p><p>The fundamentals:<br><br></p><ul><li>Define your matching key before anything else</li><li>Use upsert, write back IDs, and keep sync jobs idempotent</li><li>Establish a single system of record per entity type</li><li>Build exception handling for the cases you can't match</li><li>Have the business ownership conversation early</li></ul><p>Get those right, and duplicates become the exception rather than the default.<br><br></p></div>
</template></turbo-stream>

