The LLM Peter Principle
The Peter Principle stated simply is that "People get promoted to the level of their incompetence".
The way it works is that if you are incredible at shuffling papers. Your boss might think things like: "what a waste that this person is just shuffling papers, it seems like they could do it in their sleep, we aren't giving them a chance to use their full potential, surely they are bored."
To try to help this master paper shuffler, they will be promoted to Manager of Paper Shufflers. This might go well, they might be able to coach everybody on the team in arcane shuffling techniques, they might have deep insight into the psychological realities of shuffling papers. It can also easily go horribly, none of the skills of paper shuffling are the same as management, there is no reason to assume that they would be good at this new task.
The Peter Principle emerges from the idea that this promotion process will happen in a loop. If they are good at their job, to uncap their potential, they will take on bigger challenges. The terminal condition of this loop is that eventually they will be terrible at the new job. Because they are aren't succeeding anymore, they will not get promoted, and they are trapped doing something they are bad at.
LLMs?
In ye olde days of mid 2024, Cursor launched a "Composer" mode. You would select a bunch of files, give a few instructions and it would make changes for you. There were some issues, but it worked pretty well. You had to manage the list of files that were in context, but once you built a solid context the LLM would do a good job of writing your code.
Eventually this feature got pretty good, and as a user I would think "why do I have to add all these files to the context, surely the LLM should be able to figure this out." The experience was so magical that it really felt like it could do a lot more.
Shortly thereafter Cursor launched "Agent mode". In addition to making edits, the LLM could now search the codebase, add files to context, and run commands to test things. The LLM took on bunch of the easy work I was doing, broadening the scope of the problems it could tackle without my help. It could really stretch its legs and get stuff done without me holding its hand. It was still 2024, it was VERY cool.
The flip side of letting the LLM do more is that it must do more. It has more points where it needs to make the right decision (what file to read, what command to run). Making more decisions means more opportunities to make bad decisions, and boy howdy does the LLM sometimes make bad decisions.
So in Composer (which was still available as "Manual" for a while), you had to do a bunch of work to make the task easy, but once the task was easy the outputs were great. The LLM got promoted, now it does all that work for you.
But it had a lot of trouble with all this new work, and so nobody was looking very hard at new responsibilities to add to the agent loop. Most of the focus was on tools (todos, subagents) to give the LLM a handle on its new responsibilities. We spent a while in this place, but now the LLM and the tooling has gotten to the point where agent mode in tools like Claude Code and Codex works incredibly well.
So what happened? We gave them more responsibility. Cursor and Codex now push you to not even look at your code, and everybody is having their agents orchestrate swarms of agents doing work. These are big promotions, but needless to say they can cause a lot to go wrong.
We weren't content to stay in composer, and when you see how good agents are I guarantee your next thought will be about how to orchestrate them into a swarm. It is an obvious promotion given how good they are at coding right now.
You can see how this adds together to a sort of Peter Principle for LLM applications.
The only problem is that if we demote the AI they perform much better, but tooling providers keep pushing for the promotion. I kept returning to composer while agents were unreliable, but it was soon disabled. Agentic coding tools delegate more and more to the LLM, and don't give an easy way to pull back to the smaller skillset where LLMs don't make mistakes. (It's pretty easy to build new tools now though, so I am working on it)
It shows up everywhere in AI. Before ChatGPT came out, the goal was to have AI write coherent language. But once that was solved we immediately became more concerned with whether an AI would answer with perfect accuracy. We could not pause and just marvel at how cool it was that a computer could write sentences.