Automating Judgment Out
In an Amazon warehouse in Baltimore, roughly 300 full-time workers were fired between August 2017 and September 2018 for being too slow at picking, packing and moving items. That was close to a tenth of the building’s workforce.
Not one firing required a human manager’s approval.
Amazon’s system tracked “time off task,” the gap between one barcode scan and the next. When a worker crossed a threshold too often, the software generated warnings and, eventually, a termination notice.
Did someone stop to help a co-worker? Deal with a jammed machine? Take a bathroom break?
The system did not ask.
Amazon said managers were free to intervene. Yet documents filed with the US National Labor Relations Board described a system designed to operate without human intervention.
Amazon had built a system that followed rules faithfully, but struggled with exceptions.
Air Canada built nearly the opposite.
In November 2022, a passenger named Jake Moffatt visited Air Canada’s website after his grandmother died. He asked its chatbot about bereavement rates.
The chatbot informed him he could buy a regular ticket and claim the discount afterward, within 90 days. That was not Air Canada’s policy. The airline required passengers to receive approval before traveling.
Moffatt flew, submitted his claim, and was denied.
Air Canada later argued that the chatbot was effectively a separate entity and that the airline should not be held responsible for something the chatbot had said. A Canadian tribunal disagreed and ordered Air Canada to compensate Moffatt.
The two systems failed in opposite directions.
Amazon’s system followed its rules too rigidly and ignored exceptions. Air Canada’s chatbot wandered beyond the rules and hallucinated an answer.
Both failures point toward a larger problem facing companies racing to automate decisions. As organizations automate more, judgment becomes more important, not less.
Good judgment is more than reaching a conclusion. It means deciding what question needs answering, identifying which facts matter, determining which rules apply, recognizing exceptions and uncertainty, reaching a conclusion, and explaining why.
Most experienced employees perform those steps almost invisibly. AI forces companies to make them explicitly.
That is what I mean by engineering judgment.
Engineering Judgment
Engineering judgment means breaking a decision into its underlying mental operations, then deciding which parts should be locked down to run the same way every time, which ones require real judgment, human or AI, and where a human specifically has to be the one exercising it.
The goal is not maximum automation. The goal is a trustworthy decision process.
Sompo Holdings, one of Japan’s largest insurance groups, demonstrated this idea at Google Cloud Next Tokyo conference on July 31, 2026.
AI engineer Takuya Gotoh described what happens if an insurer simply feeds an underwriting manual into a generative AI model and asks for a decision.
Suppose an underwriting process involves four reasoning steps, and the AI performs each step correctly 90 percent of the time, which sounds pretty good. Unfortunately, the final answer would be correct only about two-thirds of the time. Errors compound over steps.
To avoid that, Sompo did something more effective than asking a bigger model to reason harder. It broke judgment into steps.
Instead of giving one general-purpose AI model the entire decision, Sompo divided the process among specialized agents.
It sounds technical. It isn’t. It’s what a careful person already does before reaching a conclusion. Here’s a case from Gotoh’s presentation.
A manufacturing company applies for JPY120 million in coverage. Under Sompo’s original rule, that is comfortably under the threshold. Approval was only required above JPY200 million, so the application clears automatically.
Then the rule changed. Senior approval is now required whenever coverage is JPY100 million or more and the applicant’s revenue is under JPY1 billion. The application didn’t change but the rules did.
- One agent reads the new rule and recognizes that revenue now matters for this case, something nobody had to check previously.
- Another agent pulls the two numbers that now matter, JPY120 million in coverage, JPY800 million in revenue, from defined fields, so there’s a clean record of exactly where each figure came from.
- Another agent looks up the actual wording of the updated rule.
- Another agent checks both numbers against both thresholds: JPY120 million clears UPY100 million, but JPY800 million comes in under JPY1 billion. The case is now routed to a human.
- Another agents writes up why, citing the specific rule and the specific numbers.
- An orchestrator agent keeps the five other agents running in a determined order, every time this situation comes up.
Sompo management decided that even if revenue for a corporate applicant comes below their targeted threshold, they will still seek out human guidance.
And if required information is missing, or two rules are in conflict, this multi-agent system doesn’t guess. It sends the case to a human for input.
In this case, humans are firmly in the loop.
The Hidden Risk: Answering the Wrong Question
But even when you have a logically sound agentic AI system, and humans are in the loop, there is a failure mode that becomes more important as companies build increasingly autonomous systems.
A system might have the right facts. It might apply the rules correctly. It might produce a perfectly coherent answer. And still be wrong.
Because it framed the problem incorrectly.
Remember Watergate?
Five men were arrested breaking into the Democratic National Committee’s offices in June 1972. Most newsrooms ran it as a burglary story, and a burglary story is exactly what it looked like: an arrest described briefly off the front page.
Frame it as a burglary, and the investigation is complete once the five men are charged.
Frame it as the reporters Woodward and Bernstein of the Washington Post did, a cover-up reaching into the campaign to re-elect President Nixon, and teh same five arrests become the first thread of a two-year investigation that ends with a presidential resignation.
Both framings would have produced a complete, defensible story. Thankfully, the Post picked the right one.
A system could therefore produce a complete, well-supported answer to the wrong question.
This is more dangerous than a visible hallucination. If you see the problem, you can correct it. But a framing error might go unnoticed for a long time, by all except for the super curious.
A person doesn’t need to approve every step. Placing human review at random points, just to say there’s a “human in the loop,” doesn’t help either. What matters is whether a bad framing would even be visible.
Watergate wasn’t. A reporter who ran with the burglary framing would never have hit a missing fact or contradiction. That particular story was complete on its own terms. Nothing downstream would have caught the mistake, which is exactly why framing itself needed a second set of eyes, not more diligence inside the chose frame.
The principle extends beyond journalism. A hiring system has to decide what it’s actually evaluating before it looks at a single resume.
If you frame the question as “does this candidate match the current job description,” the system will look for matching titles, years of experience, specific tools listed in the CV.
If you frame that same opening as “who has the highest potential for growth or future leadership,” the system will look for something else – complexity of problems solved by the candidate, opportunities to take on more responsibility, or work outside their formal role.
A candidate who is a strong match on the second framing may look like a mismatch on the first, with nothing missing and nothing contradicted anywhere in their file.
If leadership wants future potential but the system has been engineered to optimize current-job fit, no amount of resume review inside that system will expose the mistake.
The system is working perfectly, but is solving the wrong problem.
In fact, when judgment is badly engineered with AI, you get wrong answers faster, at scale.
Designing Human Judgment Back In
That’s precisely why automation doesn’t eliminate the need for human judgment. It concentrates human attention around the cases where judgment matters most.
Dan Shipper, CEO of Every, describes the paradox more provocatively: “The more you automate, the more human work there is to do.”
Engineering judgment is therefore a management problem before it is an engineering problem.
Leaders who own a workflow need to understand how competent people make decisions before engineers automate it: what questions are being answered, which facts matter, where exceptions arise, which errors are tolerable, which uncertainties require escalation, and who ultimately gets to decide.
Sompo’s architecture makes this explicit. A human enters when required evidence does not exist or when rules conflict. in other systems, a human might enter when the initial framing materially changes what evidence the system seeks.
None of the moments are arbitrary. They sit where uncertainty changes the nature of the decision.
Leaders need to understand these moments of uncertainty and how to manage them within workflows. If leaders want to leverage the potentially massive lifts in productivity that automation brings, they need to make sure they automate with good judgment.
Software searches. Agents retrieve. Models summarize. Rules engines check. AI drafts.
Leaders want to take all that work off of human hands. But not at the risk of optimizing bad automation.
Amazon automated a decision with no room for exceptions. Air Canada gave too much freedom to invent an answer. The next generation of enterprise AI has to operate between those two extremes, identifying where judgment actually lives in the work and reserving human attention for exactly those moments.
Take the searching, checking, and drafting off someone’s hands. What’s left isn’t an empty job. It’s the human part that mattered most all along.
Determining what matters.
Making the decision.
Taking responsibility for what happens next.
