What lead deduplication is
Lead deduplication, usually shortened to dedupe, is the check that asks one question before a lead is sold: have we seen this person before? If the answer is yes, within a time window and a scope you choose, the lead is treated as a duplicate rather than sold as new.
It can look like housekeeping, but it does a lot to decide whether buyers trust your leads. A buyer who is sold the same person twice will notice, and will usually want their money back.
Why duplicate leads happen
Duplicates are normal, and most of them are not fraud. They usually come from one of the causes below.
Repeat form fills
Someone submits a form, hears nothing within the hour and submits it again. Or they fill in two comparison sites that both send to you.
Several suppliers, same person
If you buy traffic from more than one publisher, the same consumer can arrive through two of them, sometimes minutes apart.
Retries and resubmits
A supplier's system times out waiting for your answer and posts the lead again. From your side, that is a second lead.
Overlapping files
A spreadsheet that overlaps with last week's, older data re-worked into a new campaign, or the same file loaded twice by mistake.
What duplicates cost
The direct cost is the return. A buyer who receives the same person twice will reject or return the second one and, if your terms allow it, claw back what they paid. If you also paid your supplier for that lead, you are out of pocket on both sides.
The indirect cost is larger. Buyers watch their duplicate rate. One who keeps finding repeats will lower the price they pay, tighten their caps or move their budget elsewhere. And a consumer who gets two calls from the same company about the same enquiry is more likely to complain.
There is a cost to deduping too hard as well. Every lead you mark as a duplicate is a lead you do not sell. A window that is too long, or a scope that is too wide, turns away people who have come back into the market. Set the rule by what your buyers will and will not pay for.
Choosing a matching key
A dedupe rule needs something to compare. The usual choices each have a weak spot.
Email address
The most common key. Fairly unique to a person, present on most online leads and easy to normalise. It misses people who use a second address.
Phone number
What call-centre buyers care about most, since it is the number they dial. It needs normalising: 07700 900123, 447700900123 and +447700900123 are the same number.
Name and postcode or address
Useful in property-based verticals such as energy and home improvement, where the property matters as much as the person. Harder to match reliably because of spelling and formatting.
Either email or phone
Catches more repeats, but also catches households that share a landline or an email account.
Whatever key you choose, normalise it the same way every time. Two values that differ only by capital letters or a trailing space should match. If they do not, your dedupe is quietly letting duplicates through, and nothing will tell you.
Time windows
A lookback window sets how far back the check looks. A lead that only matches something older than the window counts as new.
Short windows, a day or a few days, are mainly there to catch repeat form fills and resubmits. Longer windows of 30, 60 or 90 days reflect how long a buyer considers someone "theirs" after buying them. Some buyers want a permanent rule: once they have bought a person, they never want them again.
The right window usually comes from the buyer's contract or returns policy. If a buyer will return any lead they already bought in the last 30 days, a 30-day window for that buyer is the obvious place to start.
Scope: campaign, buyer or account
Scope decides which earlier leads count as "seen before". It matters as much as the key.
Campaign scope asks whether this person has already come into this campaign. It is good for catching repeat submissions from suppliers, and for telling the supplier at once so they do not expect to be paid.
Buyer scope asks whether this buyer has already had this person. A lead that is a duplicate for one buyer may be new to another, so a buyer-level rule lets you still sell it elsewhere instead of throwing it away.
Across campaigns asks whether the buyer has had this person from any of your campaigns. It matters when the same buyer takes leads from several campaigns, such as separate landing pages for the same product.
Most operators use more than one. A short campaign-level window catches resubmits at the door, and longer buyer-level windows protect each buyer's own history.
When to run the check
Dedupe at intake when you want the supplier to know immediately. If your API answers "duplicate" in the same response the supplier receives, there is no argument later about whether you owe them for that lead.
Dedupe at the buyer when the same lead may be new to some buyers and not to others. That check has to run during routing, buyer by buyer, because the answer depends on who is about to receive the lead.
Clean files before you import them. Removing the obvious duplicates from a spreadsheet first saves you processing leads you already know you cannot sell.
How it works in Hawfinch
Hawfinch dedupes on email address. The match ignores capital letters, so [email protected] and [email protected] count as the same person. A lead with no email address has nothing to match on, so the dedupe checks do not apply to it. There are three levels, each with its own lookback.
Campaign dedupe runs when a supplier posts a lead to the campaign's API endpoint. If the same email has already come into that campaign within the lookback, the supplier gets a "duplicate" status in the response. Hawfinch still stores the lead, marked as a duplicate, with when the earlier lead arrived and which buyer bought it if it sold. The lookback is set in days, or with no time limit.
Buyer dedupe is set on each buyer in a delivery group. It checks whether that buyer has already bought a lead with the same email, either across all your campaigns or just this one, within a lookback in days or with no time limit. If it has, that buyer is skipped and the lead carries on down the waterfall to the next buyer, so it can still be sold elsewhere.
Route dedupe applies the same check to a single buyer link, with its own lookback in days.
The checks fail closed. If a dedupe check cannot run, Hawfinch does not guess: at intake the lead is not accepted and the supplier is told, and during routing that buyer is skipped. Every outcome is recorded against the lead, so you can see which check stopped it and why.
When you need to override a rule for one lead, for example when a buyer asks for a resend, you can edit the lead and redistribute it with buyer dedupe skipped for that one send. For spreadsheets, Files can find and remove duplicates before you send the rows into a campaign. There is more on intake and checks on the lead management page, and dedupe is one of the features included on every plan, as shown on pricing.