When the Algorithm Cancels Your Coverage Before You Know It
Key quote:
Consumers deserve transparency and fairness throughout the insurance process, especially when losing coverage can leave someone unknowingly uninsured and vulnerable to penalties and financial risk.
Why it matters:
Since 2021, the insurance sector has chased AI efficiency gains only to get stuck in a cycle of lawsuits and settlements. The latest example is from Pennsylvania, where Attorney General Dave Sunday forced GEICO into a settlement after an automated underwriting review left a West Philadelphia driver on the road without coverage. This wasn’t some abstract error either; a customer submitted documents she thought were sufficient, received no confirmation that those weren’t good enough, and had her policy cut mid-cycle. She didn’t know she was uninsured until it turned out she was driving uninsured (both awkward and against the law), proving that algorithmic underwriting might be faster, but not better.
The real story here isn’t just that an algorithm messed up again, but that insurance regulators are starting to draw meaningful distinctions between technical glitches and genuine consumer harm. The settlement demands GEICO follow the Pennsylvania Insurance Department’s guidance on AI, which is based on the National Association of Insurance Commissioners’ Model Bulletin. That framework mandates strict due diligence on third-party vendors, detailed contracts covering data sourcing and intellectual property, and ongoing validation for model drift. Admittedly, those are normal controls that should be expected across industries, not just insurance. But GEICO couldn’t fob off the blame by pointing to its AI vendor, since insurers remain fully responsible for their supply chain decisions. If you buy a predictive model that makes the wrong decision and cancels a policy without human oversight, that is your liability, not the software company’s. And I anticipate we’ll see more of this as companies lean into deploying agents for standard workflows.
The settlement also forces operational changes that sound simple, but will require some substantial changes to insurance application forms and processing, like requiring GEICO to extend the document submission window by one week, and accepting a single form of residency proof instead of two. These aren’t just tweaks, but direct responses to a system that confused low-income customers with overly complex demands and didn’t really think through basic human behavior. When the third-largest insurer in the state settles, the rest of the insurers probably noticed that regulators won’t punish companies for using AI, but will pursue legal action when there are opaque processes and poor communication that affects people.
We’ve seen this before in a 2023 class action against State Farm claiming their AI rating system wrongly terminated policies, or a 2022 suit against Progressive alleging similar flaws with automated risk scoring. Even Lemonade faced claims that its AI engine denied coverage in violation of contracts. This most recent agreement sets a new benchmark by moving beyond what’s slowly moving through the courts to enforce specific, measurable fixes. Executives and IT leaders need to stop treating AI outputs as mere data points, because in these cases they’re regulatory decisions with real-world consequences which should have human oversight. If your system flags a customer for review, it’d be a good idea to have a human in the loop to confirm the failure and communicate that clearly.
As usual, the way forward for companies is boring but necessary: build governance frameworks, make sure that your compliance is implemented in code (and not just a neat-looking PDF), train your staff on compliance, and ensure every significant automated decision has human validation. Or run the risk of increased oversight, mandatory compliance reporting, and so many lawsuits.
