Last year I tried this experiment with ChatGPT, where I dropped a bunch of old client reports into it, along with the CSVs, the source data, and I said, ‘ write the new report based on these models,’ and it spat out this pretty perfect looking report. And I went through and I checked the numbers and everything was totally wrong, just fabricated.
I must have forgotten that I’d done this previously, ’cause I’d tried it again about a week ago with, uh, the latest versions of ChatGPT and Claude. Dropped the report in, gave it the samples, gave the data, and it gave me perfect reports. Sounded a lot like me and all the numbers based on this new data were right.
Incredible.
This highlights this core AI concept that Ethan Mollick, Wharton professor, calls the Jagged Frontier. That you don’t know just how weird AI can be, how good it can be at some things, and it how just surprisingly dumb it can be at other things. And you can only develop an intuitive sense by experimenting with AI yourself.
He gives the challenge of try to use AI for every single task you do for work for a single day. Just try it on everything and that you’ll be astonished at what it can pull off and surprised and disappointed at what it can’t. So I took this advice to heart and I pushed my experiment a little bit further.
I asked the AI find me some insights that I’m not seeing, and it did some amazing stuff. It pulled together all these disparate rows and, gave me these averages and really useful information that I wouldn’t have even thought to do myself. and I used these insights in my client report.
A jagged frontier: high Peaks, very low troughs. As Mollick says about AI, if you’re not totally freaked out, you’re not pushing the boundaries far enough. So keep experimenting with AI. Find your own jagged frontier.