Why do ATS checkers give the same resume different scores?
ATS checkers give different scores because the number is not measuring your resume. A 2026 audit held the evidence constant and a third of decisions reversed.
ATS checkers give the same resume different scores because the number is not a measurement of your resume. Each checker compares your file against a job description using its own rubric, on its own servers, and prints the result as a percentage. Two rubrics, two numbers, one unchanged resume.
That part has never been a secret. What changed in September 2026 is that somebody measured how stable this kind of judgement is when the evidence is held still, and the finding is worse than "imprecise".
If you want the anatomy of the number first, what an ATS score actually measures covers the two different things that go by that name. This page is about why two of them disagree.
What the two audits did
Qiangju Chen and Yang Xiao published a controlled audit on 15 September 2026. The design is the reason it carries weight. They built candidate profiles at set competence levels, rendered each profile into several different resume presentations, and ran what they describe as "a deterministic validation gate [that] excludes variants that alter the underlying evidence before scoring".
That gate is the whole argument. Any version of a resume that changed what the candidate had actually done was discarded before anything was scored. What survived differed only in wording, structure, stylistic polish and document extraction quality.
Then they scored what was left. Llama-3.1-8B was the strongest of the set on validity, at 0.781, and it still "reverses 29.6% of matched pairwise decisions under competence-preserving presentation changes". Mistral-7B-v0.3 reached validity of 0.644 with a 41.4% flip rate. Formatting the prompt the way each model expects improved validity for several of them and, in the authors' words, "does not remove this instability".
A second paper approaches it from the opposite direction. Jane Castleman, Zeyu Shen, Blossom Metevier, Max Springer and Aleksandra Korolova constructed a resume corpus in which the better-qualified candidate is known by design, then asked models to rank. They report that "many models are unable to consistently select the resumes describing more qualified candidates", and that models "do not reliably abstain when ranking equally-qualified candidates".
| Chen and Xiao, 15 September 2026 | Castleman and colleagues, 20 February 2026 | |
|---|---|---|
| The question | Does the decision change when only the presentation changes? | Does the model pick the better candidate at all? |
| The method | Competence held fixed, and a validation gate drops any variant that alters the evidence | A corpus built so the stronger candidate is known in advance |
| The finding | 29.6% to 41.4% of matched pairwise decisions reverse | Many models cannot consistently choose the stronger resume, and do not abstain on ties |
| What was tested | Six open instruction-tuned model conditions | General-purpose models used as screeners |
Imprecision and instability are different problems
Nearly every page answering this question says that scores differ because each tool weights keywords differently. That is true, and it leaves the number's legitimacy intact. If the disagreement were only about weighting, two checkers would be two thermometers in different units, and you could convert between them.
A reversal rate between 29.6% and 41.4% on matched pairs is a different animal. A reversal is not a smaller number or a larger one. It is the opposite decision, reached about the same evidence, because the presentation moved while the evidence sat still.
A number that changes when the evidence does not is not measuring the evidence. It is measuring the document's surface, which is the one thing a percentage is least useful for, because a list would tell you what to fix and a percentage does not.
Where this evidence stops
Earlier than the figures make it sound, and saying so is the point of citing them properly.
Both papers test language models used as screeners. Chen and Xiao ran six open instruction-tuned conditions, models in the 7B and 8B parameter range that anyone can download. Neither paper tested Workday, Greenhouse or iCIMS, and none of those systems exposes a candidate-facing score that an outside auditor could examine in the first place.
So this is not "your score at a named vendor is 41% arbitrary". Nobody outside those companies can say what their ranking does, and anybody who quotes you a figure for it is guessing. That is the same objection this page is making about checkers, and it applies to us as well.
What makes the audits transfer anyway is a sentence in the Castleman paper about who is doing the screening: "many entities rely on general purpose LLMs without further adapting them to the task". The model being audited is frequently the same class of model that ends up in the loop, without fine-tuning, without validation, and without anybody measuring whether its decisions hold still.
Whether a particular employer's pipeline flips on 5% of pairs or 40% of them is not knowable from outside it. Nobody knows. Write that down next to any number a tool gives you.
What does not move when the presentation moves
Three things in this process are checkable, and none of them is a score.
Automatic rejection is documented, and it does not read your resume. Greenhouse states that with auto-reject, an organisation sets up application rules so that "based on an applicant's answer to a question, they will automatically be rejected as a potential candidate", on its Plus and Pro subscription tiers. The same page adds that these rules "can only be configured for questions of the following type: Yes / No, Single-select, Multi-select". That is a rule on the application form, and none of those three types is a resume. The instant rejection people remember is usually a question they answered, not a document that scored badly. The mechanics are in rejected minutes after applying.
The parse is the same every time, because it is extraction rather than judgement. Either your most recent employer landed in the employer field or it did not. Either your dates survived or a two-column layout ran two jobs together. That is a fact about your file, it takes about twenty seconds to look at, and the method is in is my resume ATS friendly.
Coverage is stable for the same reason. Which requirements in the posting your resume answers, which it answers weakly, and which it never mentions at all is a list, not a verdict. Whether a given word appears in two documents is a fact about those documents, so it does not depend on whose rubric you use. The percentage built on top of it does.
Four things worth doing instead of moving the number
- Read the parse before you read any score. Upload your file somewhere that shows you the extracted fields. Fix what landed in the wrong place. This is the only part of the pipeline you fully control.
- Turn the percentage into a list. Ask which stated requirements your resume never mentions, then decide which of them describe work you actually did. The twenty-minute version is in how to tailor your resume to a job description.
- Keep one resume per job, not one per checker. Editing to satisfy a second rubric after you satisfied the first is how a resume ends up written for software instead of for the person who reads it.
- Stop where the next point would need an invented number. A vague true line survives the interview. A specific false one does not, and you are the one who has to sit in the room with it.
Upload your resume if you want to see the parse rather than a score. No account and no card.
The short version
Two ATS checkers give different scores because each one is reporting agreement with its own private rubric, and no rubric in this category has a published validation against a hiring outcome. The September 2026 audit adds the part that should settle it: hold the evidence constant, change only how it is presented, and between 29.6% and 41.4% of matched decisions reverse. Instability that large is not a precision problem. It means the quantity being reported is not the quality being judged.
Rzume does not calculate a score. It shows you the parse first, field by field, including what it could not place, and then names the requirements your resume does not answer. The parse and the full review are free and need no account.
Upload your resume, or read what each plan includes first.
Sources
- Chen, Q. and Xiao, Y., Competence-Preserving Resume Perturbations Expose Presentation Sensitivity in LLM Screening, submitted 15 September 2026. Six open instruction-tuned model conditions. The validation gate, the 0.781 and 0.644 validity figures and the 29.6% and 41.4% flip rates are quoted from the abstract.
- Castleman, J., Shen, Z., Metevier, B., Springer, M. and Korolova, A., Measuring Validity in LLM-based Resume Screening, submitted 20 February 2026. A constructed corpus with known ground-truth superiority.
- Greenhouse, Auto-reject support documentation, page last updated 30 January 2026. Primary vendor documentation for what automatic rejection acts on, and for the three question types a rule can be built from.
All accessed 26 September 2026.
