Recent findings suggest that an AI system flags a problem in a mortgage file. The reviewer looks at the documents and disagrees. During an evaluation, this can become a discussion about whether the technology works. In production, it becomes a decision someone has to make while the file is waiting.
According to recent reports, the system may have identified a genuine problem. The reviewer may have information the system did not receive. Both may be looking at the same evidence and interpreting a requirement differently. Putting a person in the process does not settle any of this. The team still needs a reliable way to establish what is correct.
Industry observers note that in my recent HousingWire article, I argued that lenders should measure how much human attention a loan consumes. Disagreements between AI and reviewers deserve particular attention because resolving them can take far more effort than identifying the original issue.
I believe lenders should agree on how to resolve these cases before an evaluation begins. Otherwise, the evaluation can end with a long list of overrides and very little clarity about what the results mean.
The original finding should remain available after a reviewer challenges it. That record should include the document and page involved, the data used, the requirement being tested and the relevant system or rule version. The reviewer’s response should sit alongside it, with the evidence supporting the disagreement.
Recent findings suggest that consider an illustrative example in which a system flags a missing signature, but the reviewer claims it is present on a later version of the document. Before deciding whether the finding was wrong, the team needs to establish which version the system reviewed and when the later version became available.
If the document arrived after the review, the finding may have been correct when it was raised. If the document was already available but the system failed to select it, there is a different problem to fix. A final status of “cleared” would hide that distinction. Preserving the original evidence gives the team something concrete to investigate and prevents a later reviewer from having to reconstruct the entire discussion.
Industry observers note that A finding often contains more than one conclusion. In the signature example, one question is whether a signature appears in the relevant location. Another is whether that signature is required for the document, transaction and stage of the process being reviewed. The first is an observation, while the second depends on the applicable requirement.
A system could get the observation right and apply the wrong requirement. A reviewer could correctly question the requirement but overlook that the document being reviewed is incomplete. These should be recorded as separate complications.
I would ask the team to identify exactly where the disagreement sits: document selection, extracted information, applicable requirement or judgment about the problem’s significance. That distinction helps the team choose the corrective action. Improving document recognition will not resolve an unclear policy, and more reviewer training will not repair a system that consistently selects an outdated document. It also keeps the discussion focused on evidence rather than on whether the reviewer or the technology should be trusted more.
Industry observers note that every disputed finding needs an owner who can make the relevant decision, with that authority established before the disagreement occurs. A straightforward document-selection error may be resolved within the review team. A question about how a requirement applies may need a designated policy owner, while a recurring technical error should reach the team responsible for the system.
For material cases, I would use a second qualified reviewer rather than leaving the original reviewer to be the sole judge of their own decision. The same principle applies to the technology provider. A provider can explain its output, but should not be the only party deciding whether that output meets the lender’s requirements.
In a recent development, the resolution should state what was decided, the evidence relied upon and who approved it. “Reviewer disagreed” tells the next person very little. The process should also be proportionate. An obvious extraction error should not require a committee. A disputed concern that could affect a consequential decision deserves more scrutiny.
Sometimes the available evidence will not support a decision because a document is unreadable, a required page is missing or the relevant requirement needs clarification. Forcing the case into “AI correct” or “reviewer correct” in those circumstances creates certainty the team has not earned. The case should remain unresolved, with the missing evidence or clarification identified and the next action assigned.
It has been revealed that that status should have an owner and a review date. It should also have a defined consequence for the workflow, based on the nature of the matter and the lender’s existing controls. An unresolved disagreement should not silently disappear because the file has moved to another team.
During an evaluation, unresolved cases should remain visible in the results. They should not be counted as successes or quietly removed from the reported sample. This matters when comparing systems. A strong accuracy figure is difficult to interpret if the hardest cases have been excluded because no one could establish the correct answer.
The resolution itself is only part of the result. Lenders should also examine how much time the reviewer spent finding the evidence, whether another person had to repeat the review and whether the file was reopened. They should track whether the same concern returned on the next loan. These questions reveal whether the technology is making the work easier.
A finding that is ultimately correct can still be poorly presented. If the reviewer has to search hundreds of pages to understand it, there is a workflow problem worth fixing. A high override rate also needs investigation rather than an immediate conclusion. It could reflect poor system performance, unclear requirements, inconsistent reviewer decisions or several of these together.
Sources indicate that I would review disputed cases by cause and severity, alongside the time and rework required to resolve them. I would also sample cases where the reviewer and system agreed. Agreement alone does not establish that both were correct.
In a recent development, the findings should inform changes to the technology, the instructions or the review process. Those changes need testing before they are applied more broadly. An individual override should not automatically become a fresh rule or a training label.
Lenders should ask technology providers to demonstrate a disputed case, including how the evidence is retained, who can challenge the finding and how the final decision is recorded. That demonstration may be more revealing than another clean file moving through the system without intervention.
Industry observers note that mortgage operations will continue to need experienced people who can question an output and explain their judgment. Those people need evidence, authority and a workable route to resolution. When the next reviewer opens the file, they should be able to understand what was questioned, how it was resolved and why the decision can be relied upon. If they have to start the investigation again, the disagreement was never properly closed.
Industry observers note that alok Bansal is Founder and CEO of Logikality.This column does not necessarily reflect the opinion of HousingWire’s editorial department and its owners. To contact the editor responsible for this piece: [email protected].
