Duplicate Management: Keeping Your List Clean
Where duplicates come from, the normalisation rules that prevent most of them, and how to merge without losing history.
On this page
The short version
- Most duplicates are preventable at capture. Normalising the address on the way in removes the majority for the cost of three lines.
- The hard cases are the same person with genuinely different addresses, and those cannot be merged automatically without guessing.
- Merging loses history unless you plan for it. Decide what happens to engagement, consent date and source before you merge anything.
Duplicates cost money on contact-based pricing, distort every engagement rate, and occasionally produce the visible embarrassment of one person receiving the same email twice.
They divide into two kinds with completely different remedies: the same address stored twice in different forms, which is preventable, and the same person with two addresses, which is not.
Normalise at capture
The majority of duplicates are one address in several forms. Different capitalisation, a trailing space, a stray character from a paste.
Three operations at write time remove nearly all of them: trim whitespace, lowercase the whole address, and reject anything that does not contain a plausible at-sign structure. Applied consistently across every capture point — forms, imports, integrations — they prevent the problem rather than cleaning it up later.
Consistently is the operative word. A site with a normalised main form and an unnormalised checkout will accumulate duplicates through the checkout indefinitely.
Where duplicates come from
| Cause | Preventable? | Fix |
|---|---|---|
| Case differences | Yes | Lowercase at write time |
| Leading or trailing spaces | Yes | Trim at write time |
| Multiple capture points, no normalisation | Yes | Apply the same rules everywhere |
| Imports without a dedupe step | Yes | Match against existing before insert |
| Plus-tagged variants | Partly | A deliberate decision either way |
| Provider dot variants | Partly | Provider-specific; usually leave alone |
| Genuinely different addresses | No | Cannot be merged safely by rule |
Merging without losing history
A merge has to decide what happens to four things, and doing it by default usually loses the ones that matter.
Consent date: keep the earliest, because it is the one that evidences permission. Source: keep the earliest for the same reason, and record the second somewhere rather than discarding it.
Engagement history: combine, do not overwrite. A record showing the later signup's two weeks of history, when the earlier one had two years, will misclassify the person in every engagement segment.
Preferences and suppression: take the most restrictive. If either record unsubscribed, the merged record is unsubscribed. This one is not a judgement call — merging an opt-out into an active record and mailing the result is a genuine compliance failure.
What duplicates cost
Three things, and only the first is usually counted. Platform pricing is generally based on contacts stored, so every duplicate is paid for monthly.
The second is that every engagement rate is diluted. A subscriber counted twice who opens once appears as one open across two records, which drags down the ratios that decide placement — and since list hygiene is largely about protecting those ratios, duplicates undo part of the work a cleaning pass does.
The third is the visible one. A subscriber receiving the same email twice concludes that nobody is paying attention, and it is the kind of error that gets screenshotted. Deduplication before every send is the cheapest defence, and most platforms do it automatically within a single send but not across two overlapping segments.
The same person, two addresses
A work address and a personal one, an old employer and a new one. These are not duplicates in any technical sense and no rule can identify them reliably.
Automated matching on name is the tempting approach and it is wrong often enough to be dangerous — common names collide, and merging two different people's records is worse than any duplicate.
The workable answer is to leave them as two records and suppress both from acquisition messaging if either has bought. Where a customer tells you they have two addresses, merge on their instruction rather than on inference.
Duplicate check
- Trim and lowercase applied at every capture point
- Imports match against existing records before inserting
- The plus-tag and dot policy is decided and written down
- It is applied consistently everywhere
- Merges keep the earliest consent date and source
- Engagement history is combined, not overwritten
- The most restrictive suppression wins on merge
- Name-based automatic matching is not used
Frequently asked questions
How many duplicates are normal?
With normalisation at capture, very few — a fraction of a percent. A materially higher rate points at a capture point that is not applying the rules rather than at a cleaning problem.
Should I merge or delete the duplicate?
Merge, keeping the earliest consent date and the combined history. Deleting one loses whichever record held the better evidence of permission.
What about duplicates across systems?
That is a joining problem rather than a duplicate one. Normalise on both sides before matching, and measure the match rate — an unmeasured join quietly failing on a fifth of records persists for years.