AI Lead Scoring: Where It Helps and Where It Misleads

Lead scoring is an easy sell. Sales has more leads than time, a model ranks them, everyone works the top of the list. The mechanics are simple and the tooling is mature.

The failure mode is subtler than a wrong score. A scoring model trained on who you have closed will confidently reproduce who you have closed — including the biases in how leads were worked in the first place. If a segment was historically neglected, it converted poorly, and the model learns to deprioritise it further. The feedback loop closes and nobody notices, because the metrics improve.

Where scoring genuinely helps

• High volume with limited sales capacity, where triage is already happening informally and inconsistently.
• Clear disqualification signals — wrong geography, wrong company size, a service you do not offer.
• Urgency detection from what the prospect wrote, which is a classification task models do well.
• Routing by topic, so an enquiry reaches the person who can answer it.

Notice these are mostly about ordering and routing, not about deciding who is worth a conversation.

Where it misleads

Thin training data

If you close forty deals a year, you do not have enough examples to learn a reliable pattern. A model fitted on that data will be confident and arbitrary. Below a few hundred outcomes, a simple written rule set is more honest and easier to correct.

Proxy variables

Models find whatever correlates. I have seen scoring effectively keyed to time of submission, because leads that arrived during working hours were called quickly and therefore converted. The model learned the sales team's schedule, not lead quality.

The unworked-lead blind spot

Your data only records outcomes for leads someone contacted. Leads that were ignored have no outcome — and are usually treated as negatives. The model then learns to suppress exactly the segment it has never seen tested.

The remedy is to work a random sample of low-scoring leads regularly. It costs a small amount of capacity and it is the only way to find out whether the model is wrong.

A structure that holds up

The approach I recommend separates the score from the decision, which keeps behaviour explainable.

1. 1Start with explicit rules for hard disqualification. These are business decisions and should never be probabilistic.
2. 2Use a model for classification and urgency, producing a label with a confidence value.
3. 3Keep routing rules deterministic and readable, driven by the label rather than replacing it.
4. 4Show the reasoning in the CRM — which signals raised or lowered the score — so a salesperson can override with judgement.
5. 5Hold out a random sample of low scores for manual work, permanently.
6. 6Review scoring accuracy quarterly against actual closed revenue, not against pipeline created.

What to measure

The metric that matters is not model accuracy. It is whether total closed revenue improved after prioritisation changed — and whether the mix of customers stayed healthy. A model can improve conversion rate while narrowing your customer base into a segment that stops growing.

Track both. If conversion is up and new-segment acquisition is down, the model is optimising you into a corner, and that is worth knowing before it takes a year of pipeline with it.