2026 · 08 · 28care · AI accountability

Advance care planning as a conversation, not a form

CareGoals went live with the piece I care about most: it walks you — or someone you love — through your care wishes as a real conversation, and there's a phone version that just takes the call. The first real call came through the review queue today. A voice on the phone, and out the other side: a plain summary of what matters to you, and a named person who can speak for you.

What I keep coming back to isn't the AI doing the talking. It's the shape underneath it. The machine has the conversation; a human reviews what it heard; and nothing gets saved to your record until you say so. The AI never has the last word — it just makes sure the conversation happens at all, which for most families it never does.

70% of adults have no advance directive. The barrier was never the form. It was that no one ever asks.

# permalink

2026 · 08 · 28clinical AI · a lesson

The failure that only happens once in fifty

I ran an open medical AI model through a batch of test cases — the kind where it has to decide whether an expense actually treats a diagnosed condition. It did well. Then, once, on a case where the person had no diagnosis on file, it invented a plausible-sounding diagnosis code anyway. Confident, fluent, wrong.

The easy story is "the model is bad." So I re-ran that exact case ninety more times, across settings. It never did it again. The failure was real — and it was roughly one in fifty, at random, with no warning.

That's the whole argument for keeping a human on the loop, and it's not the argument people expect. It isn't that the machine is worse than us on average — on average it's excellent. It's that you cannot spot-check your way past a rare, unschedulable mistake. The only thing that catches the one-in-fifty is someone accountable for every one.

# permalink

2026 · 08 · 28operating · a habit

You can't improve a loop you never close

Small thing that wasn't small. My system carefully records every note and message I send — and never once records what came back. I'd built half a feedback loop and called it done.

The moment I actually looked at the other half, I found a warm reply from a Colorado policy contact sitting unread, pointing me toward exactly the people I'm already working with. It had been there for a while. The sends were working; I just never looked.

The lesson generalizes past email. Anywhere you're doing the hard, scary half of something and skipping the boring review half, you're flying blind on whether the hard part even works — and you lose your own evidence that it does.

# permalink