I'm typically pretty critical of PM oriented pieces, but I found this to be a decent overview of how to reason about building these systems from first principles + some of the non-tech pain points + how to address them.
> Confidence calibration: When your agent says it's 60% confident, it should be right about 60% of the time. Not 90%, not 30%. Actual 60%.
With current technology (LLM), how can an agent ever be sure about its confidence?
We’ve found through extensive trial and error that it’s absolutely best not to let anyone with a pm title read anything whatsoever.
So far our projects and products are doing slightly better than before we hid their glasses.
Stop trying to treat these things as more than they are. Stop trying to be clever. These models are the single most complex things ever created by humans; the summation of decades of research, trillions in capex, and the untold countless hours of thousands of people smarter than you and I. You will not meaningfully add to their capabilities with some hacked together reasoning workflows. Work within the confines of what they can actually do; anything else is complete delusion.
What does the PM title even mean at this point? Ideally they would not be diving this deep into technical architecture (though there is massive value in understanding what's involved) and focus on understanding scope, user needs and how to measure success.