Each time a new AI model comes out, I have my own set of tests. One is a health puzzle. I have a bunch of hormonal data and life style data and i put it into the new model to get it's diagnosis and plan to fix the issue with a mechanistic explanation. 

Then I have a novel I've been writing. I've got about about a dozen chapters done. The first chapter has a subtle twist in the last couple sentences. It's subtle but profound for understanding what's going on in the story. I have the new AI model read this incomplete manuscript draft and I judge it on how it judges my manuscript. I see what it picked up on, what it missed, where it has some retarded politically correct "suggestions", what recommendations it has for plot, and if it can understand the larger mission of the book. 

These two exercises tell me almost everything I need to know about a model. I don't really care about whether it can solve a math problem or code an app. There is far more information about it's performance when you throw a very noisy, very complex system at it. Biology is such a system - mostly due to our lack of comprehensive knowledge, and a novel, with all its twists and rabbitholes, it's contextual richness. Well, lets just say that the limits of a SOTA model show up quickly and obviously.

There is another test that I like to throw at SOTA models. I give it my latest book WHITE MIRROR,¹ and then I ask it to solve the puzzle.

WHITE MIRROR has within it a puzzle that I designed and embedded into the text. It is not an easy puzzle. It requires thinking across multiple domains and multiple dimensions of each of those domains. It can't say too much about it because, well, naturally, I don't want to take away from anyone who aims to cherish the challenge. 

While I'm sure there will come a time and an AI system that will be able to solve my little puzzle in a handful of seconds, I can't deny that a smug and satisfied smile emerges when I see the results of the latest Fable model put to this task. What's the verdict?

Well, when I tried this with Fable, it failed. Spectacularly. The thing had no idea what it was doing, what to look at, what to pay attention to, where to draw connections, what to drill down into. To be completely honest the experience was like that of David Attenborough watching an orangutang bang a saw against a piece of wood, hammer at horizontal nails and get distracted by a banana in the distance.

Don't worry, I'm aware that my puzzle will not remain immune to the possibilities of compute. But that's not the point. It was never the point. I did seek to make something that was fairly robust to AI systems, and given my fairly extensive usage of AI systems I had a bit of an intuition about what would and wouldn't work.

Today, I worked on the code for Tinkered Thinking. It's been on my to-do list for a while and I finally had time this past weekend to lay out some plans. And then of course AI just plowed through the work. Work that I used to do by hand. Tedious work. Mind numbing work. Sure there was a lot in the process of hand-coding that was interesting and challenging. But fiddling with css to get some nice effect to work was never in that category. Now I have better problems. Like, what should I write about? What books should I release? What else can I offer the world through this brand, this. . . experiment. 

This is the point of AI, and its under appreciated by the doomers and even the accelerationists. As with all things in the public discourse, any loudly touted position lacks the nuance that would quell such protestations with a more thoughtful contemplation of the topic. Unfortunately most people do a job that requires solving a bunch of mundane and tedious provlems of questionable necessity. And if suddenly relieved of those problems, the concern isn't to get better problems, the concern is where the paycheck will come from. 

In Korea for example, the AI-wariness of the states doesn't make any sense. This is because it's apparently quite difficult to terminate positions, and so everyone is welcoming the computer's ability to do a bunch of their work. I'm sure the same would be true in the states. The computer can now do my job, but I still get the paycheck? Sweet.

Unfortunately the USA is a different breed of economics. And this essay about puzzle problems and models isn't the place to solve the disparity between what's going on and what might be needed.

Think about it in terms of your time off. Your weekends are often littered with annoying problems: sign up for car insurance, cancel that subscription, fix the broken drawer in the bureau, clean the bathroom. These are all shitty problems that you don't get paid for. And if you woke up tomorrow and there was suddenly a system capable of taking all these trivial problems off your hands. . . would you object? Or would you instantly think about how you'd be able to spend your time more freely, doing things you actually wanted to do? Do those trivial bullshit problems give meaning to your life? I don't think so. And if you do think so, then I think there's a bowl of psychedelic mushrooms with your name on it, or a tab of acid at the very least. Which is to say: get real. The only reason people attach meaning to mundane jobs is because it's a euphemism for survival by way of a paycheck. The real fear isn't a loss of meaning, it's an insecurity about the possibility of getting another job, a better job, with better problems. This is a very real concern, and for many it's going to be quite bumpy.

The ladder of agency remains: once you've mastered one level, it's a mistake to keep rerunning that level over and over. You need to level up, and dive into something that's beyond your ability. It's the only way your ability expands: with better problems.

And now we have an incredible and dynamic tool that can blast through a lot of those trivial and time consuming problems that used to eat up your hours.

Regardless, you can be sure that each time a new SOTA model drops, I'll be throwing my own puzzle at it: