"Tokenmaxxing" Is Not Productivity

We All Want to Be Productive

Everyone wants to be productive. As a manager, we don’t produce much ourselves, so our productivity is the vicarious productivity of our whole team. It’s an important part of how we measure ourselves and how we are measured. If there’s a secret unlock that will increase the return on our investment in people, processes, tools, etc. - of course we’re going to want to use it. And heaven forbid someone else figures it out before you and you end up looking like a doofus trying to keep up with race cars speeding along the road while you’re plodding along riding a donkey.

Tokenmaxxing

I've been trying to avoid writing too much that's directly about AI and how it pertains to the software development world and the management thereof. The "AI is good/bad" discourse is already everywhere and maybe you'd like to read about something else once in a while. That said, the fact that it's having an impact on the industry is unavoidable. I can't stop thinking about this piece in the New York Times (gift link). Specifically this bit:

And at tech companies like Meta and Shopify, managers have started to factor A.I. use into performance reviews, rewarding workers who make heavy use of A.I. tools and chastening those who don’t.
...
But it has also created an expensive new status game, known as “tokenmaxxing,” among A.I.-obsessed workers who are desperate to prove how productive they are.

If you care about the practice of management and the goals of your team, this hopefully has provoked a similar reaction in you as it has in me: 🤦‍♂️

Set aside for a second whether or not you think using LLMs for software development at all is a good idea in the first place, that's not what this is about. This is about managing outcomes. Obviously, (well, let's say 'hopefully') nobody's getting promoted for using a lot of AI while shipping nothing. Shipping working software is the point. That's the thing that matters. How you get there is incidental and certainly not something to give more than a passing glance at in a performance review.

Tool Use Is Not A Goal

Imagine two developers. One uses a full-featured IDE and one uses a basic text editor and command line tools. You might be curious about why the text editor user doesn't prefer the IDE. You might even inquire. You might not even agree with their reasoning but come performance review time, if they got their work done and their sum total of their contributions to the team was on par or better than everyone else, who cares? They've found a way to work that works for them so don't mess with it!

If your assessment of that person is "did a great job, but stubbornly refuses to do things the way I would so I'm scoring them 'does not meet expectations'" you need to adjust your expectations. Your goals for your team should focus on outcomes, which boils down to shipped software, personal growth, and team health. The specifics of how you get there are going to vary by person, team, circumstance and other factors and should not be goals in and of themselves.

Managers who negatively judge other people for doing things in a way that's different from how they themselves would do that same thing are suffering from a counter-productive form of narcissism that will inevitably lead to worse overall outcomes. Either people will get tired of it and quit or they'll acquiesce and be less efficient than they otherwise would be doing things in a way that works for them. In any case, the overall outcomes will be worse.

Don't Make Gameable Metrics Part of the Performance Process

This is engineering management 101. If you make a game, people will play it. I've touched on this before, but it bears repeating. Counting lines of code, tickets closed, or now, AI token use are all things that a person incentivized to increase will find a way to do so even if the overall outcomes you actually care about, like completed features, more sales, increased profits, etc. don't change. Maximizing LLM token use for its own sake is trivially simple for any capable programmer to do. Additionally, they cost money. Financially, this is equivalent of incentivizing a delivery driver to burn more gasoline on the presumption that more gas used -> more deliveries made.

Metrics are useful as measures but they're only a small part of the performance picture and should never be turned into goals.

Productivity Is A Feeling - But Feelings Can Mislead

The output of knowledge work is not truly countable. It has countable aspects but those aspects are not the outcome. A good novel has a countable number of words and pages, and those impact the cost of printing the book itself, but the number is not what makes a novel good or bad or profitable to publish. The costs of building a house are knowable, as is the bill of materials needed to build it but they don't tell you the whole story of whether or not the house is well designed and delightful to live in.

Unless you're a machine screwing tops on jars, most productivity is not directly measurable. You feel it when you have a good day, you were focused, you got a lot done. Maybe it was one thing off your list or twenty, but you know it was a 'productive' day. When it's your team's productivity you rely on what you can observe but the set is limited, which produces anxiety. Our productivity-through-others might be sub-optimal! That's why it's tempting to focus on the measurable things like lines of code or tokens consumed, if the number goes up it gives us comfort and something tangible we can point to that indicates that we're in the race-car lane, not the donkey-riding lane.

But our feelings can fool us. Some studies have apparently shown that LLM use can make you feel faster while actually slowing you down but, again, that presumes an objective measure of productivity exists and that one can successfully control for LLM use or not alone. Also everything is changing daily so what was true last month might not be anymore.

Nevertheless, it's certainly true that we, as humans, are capable of feeling in a way that doesn't match reality and that's doubly true when those feelings are about other people. Don't mistake the trappings of productivity for the real thing. Be mindful if you're optimizing for true productivity or the feeling of productivity. Focus on outcomes, trust your team’s methods, and let the chaos of creativity drive your success.

Image:"tokens casino" by Token Company is licensed under CC BY 2.0 .