What Feels Hard

Why AI can write a sonnet but miss a word count, and why that matters when you hand over work.

A man in a cardigan hands a slip of paper to a robot that holds a heavy trunk overhead with one arm, while a blue slip falls from the robot's other hand. The words the easy end sit at the left and a count at the right.

What we call a hard task is a task that is hard for a person.

A sonnet takes craft. A 50-word poem adds a count. In 2023, Ethan Mollick described AI doing well at the sonnet, yet missing the poem's exact word limit.

LIFEBench later found GPT-4o missed exact word-count targets by 31% on average.

LIFEBench also found that models cannot accurately count the words they have written so far. In 2023, Mollick attributed this to AI working in tokens, not words.

In 2024, TechCrunch reported GPT-4o and Claude counting two r's in strawberry. Matthew Guzdial explained the role of tokenization: the models process chunks of text rather than individual letters.

The models process chunks of text rather than individual letters.

The models process chunks of text rather than individual letters. Dots, in order: the robot, the closed boxes, the loose beads, the person.

Click a numbered dot to see what it stands for. Start with 1.

In a test on 10,000 random words, Fu and colleagues found that even GPT-4o, the best model tested, miscounted letters in 17% of the words. Errors correlated most strongly with repeated letters; most models couldn't count letters appearing more than twice in words.

You trust the count. Counting feels like the part that should take care of itself.

That is what bothers me about the failure. You are tempted to give it less scrutiny precisely because it seems simple.

The easy end

You rank the work by how hard it feels, then hand over the easy end. You keep the difficult parts close, where you can watch them.

That gives you a tidy division of labor, if your sense of difficulty matches the model's limits.

In 2023, Fabrizio Dell'Acqua and colleagues ran a field experiment testing GPT-4 with Boston Consulting Group consultants. The researchers behind the study with BCG consultants had trouble designing a task outside those limits. They wanted work that skilled consultants could do well, but GPT-4 would get wrong.

Designing that task took several attempts. They eventually settled on a task based on an existing business case, built so GPT-4 would make an error during the analysis.

Even the team that named the jagged frontier had to test where it stopped. The jagged frontier means tasks that look equally hard, like the sonnet and the 50-word poem, differ in whether AI can succeed. They could not simply choose something that felt difficult and expect failure.

You are using familiarity with your own work to judge the model's ability to do it.

Those iterations matter to me. Yet a published score offers what looks like a way to skip them.

Compared with whom?

OpenAI reported GPT-4 scoring 298 out of 400 on the Uniform Bar Exam, around the 90th percentile. Eric Martínez's 2024 re-analysis questioned what that comparison meant.

The comparison group was heavily skewed toward repeat takers who had failed before.

The comparison group was heavily skewed toward repeat takers who had failed before. Dots, in order: the robot, the repeat takers, the first-time takers.

Click a numbered dot to see what it stands for. Start with 1.

The comparison group was heavily skewed toward repeat takers who had failed before. Against first-time takers, Martínez estimated GPT-4's result at about the 62nd percentile overall.

On essays, his estimate fell to about the 42nd percentile against first-time takers. Against people who passed, he estimated its essay performance at about the 15th.

These were estimates against different groups, not new versions of GPT-4.

The essays were also its weakest section. Martínez pointed out that these tasks more closely resembled what a lawyer does.

Care about that weakness more than the headline rank.

I expect that problem to matter beyond this model. I still need to know which tested tasks resemble my own.

Keep the misses

Take one task from your week whose output reaches a customer, rather than a whole domain. Run 100 real cases through the model.

Have the person who owns that task check every output, not a sample.

That person decides which of the 100 outputs are good enough to reach the customer.

For each miss, keep the output beside the case that produced it, with the review written down. Look at what went wrong, not just whether the result passed.

That record needs to sit outside your head. For each decision, I wrote down my choice, how confident I felt and my prediction, then checked them later against what happened. My confidence journal showed that calls I marked 90% confident were right 58% of the time. The feeling had not told me which calls would fail.

Don't let your impression of a model become another unchecked call.

Mollick advises using AI on enough work tasks to begin seeing the frontier's shape. Add a written record to that practice.

I expect the failures to gather in patterns I would not have predicted. The record lets you check that expectation, rather than remember only the surprise.

When the model changes, run the cases again. Dell'Acqua and his co-authors urge professionals to adjust their understanding as the frontier moves. Keep the old misses beside the new results, not a fresh feeling that the model has improved.

Are 100 cases the answer?

Treat 100 as a starting batch, not a certificate. The point is to check real work closely enough to learn from the failures.

There is a fair reason not to do all this yourself. Published benchmarks exist so each person need not rebuild the same test. They offer evidence beyond what you could gather alone.

Use them to help choose what to try. They show what a model can do under tested conditions. Do not expect them to settle what you can trust in your own work. A published score is a report on someone else's cases, so it cannot decide what reaches your customer.

My expectation is that misses in my own cases will not follow the published ranking. If those misses do follow it, I would be wrong. The leaderboard would have earned more of my trust.

I know what feels hard. I want to know what fails.

I'll send what I find at the edge of my own work by email.

Part of The First Move