Systems Need Change Every Few Cycles

There is a pattern I keep running into. Someone builds a system with good intentions, and it works well for a while, until people figure out how to game it and it slowly turns into a parody of what it was meant to do.

The exam that became an industry

JEE was supposed to find bright students regardless of where they came from (the SAT pitched the same idea in the US). Today there is an entire industry built around "cracking" it, with coaching fees that rival college fees, and kids who have drilled so many problem types that they can solve anything they've seen before but freeze on anything they haven't.

While the test did not get any easier, we actually got better at beating it. Once something meant to measure aptitude became the target, it stopped measuring much of anything. Having written about knowledge vs grades before, I will admit I played that game too.

Open source, now with resume points

This one hurts, because open source is how I got into all of this. I spent my early years porting Sailfish OS to a phone and teaching Git as a GitHub Campus Expert, mostly because it was fun and the community was kind to people who didn't know much yet.

Google Summer of Code was meant to bring new contributors into projects they cared about, but for a lot of students it is now mainly a line on a resume. Some still do it for the love of the project, while plenty treat the project as a means to that line.

Hacktoberfest had it worse. In 2020, maintainers were flooded with pull requests that changed a word in a README, sent by people chasing a free T-shirt, and it got bad enough that the organizers changed the rules and made it opt-in for repositories. Both programs still exist, but the incentive ate away a chunk of the spirit.

Search, optimized into the ground

Google used to surface the best page for what you asked, and then we all learned SEO. Now you get recipe blogs with a life story before the ingredients, and product reviews written by people who have never touched the product.

In cracking the algorithm, we broke the thing that made it useful.

Then there is the uncomfortable one: democracy. In theory, you do good work and people vote for you to keep doing it. Although campaigning well and governing well are different skills in practice, the system rewards the first one every election. I went down this road in Systems of Governance, so I will not repeat the argument here.

Goodhart's law, everywhere

"When a measure becomes a target, it ceases to be a good measure."

That is Goodhart's law, and once you notice it, it shows up everywhere. Universities chase rankings, companies hit the quarter and damage the year, and social media optimizes for engagement and ends up optimizing for outrage. Followers can be bought, and so can "engagement."

I should admit something here. I am privileged enough to care about the spirit of a thing more than what it does for my resume. Since I do not need open source contributions to get a job and this blog does not depend on ad revenue, all of this is easy to say from where I sit.

That is also why the gaming bothers me. When a system gets gamed, the people who needed it most lose it first: the student from a small town who would have shown up on a fair test, the new developer who would have found a community, the small business that can't outspend an SEO farm. Everyone pays for the gaming, just not equally.

So, change the game

I have come to accept that the perfect system does not exist. Every system eventually gets gamed, because someone always finds the exploit.

So perhaps, instead of building a better system and defending it forever, we should expect every system to wear out and change it every few cycles. The new one need not be better, since the change itself resets the game and breaks the exploits that grew around the old one.

Imagine exams that change format every few years, so that coaching for the format stops paying, or contribution programs that reinvent themselves before the spam catches up. It would be painful, and people would complain (I would complain), but it might hurt less than watching a good system rot from the inside.

What this looks like with a team

This is the part I actually get to think about day to day, as CTO at Metafic. Engineering metrics are the classic trap: if you lean too hard on tickets closed, story points or velocity, you soon end up measuring how good people are at hitting the number.

What I have come to believe, while I am still figuring this out, is that a few things help more than any perfect scorecard. Rotate what you celebrate, so that the bug fix gets noticed one time and the documentation or the mentoring the next. Ask for the story of what changed, because "I helped three teammates get unstuck" tells you more than a ticket count. And be open about the whole thing. When you tell people that a process is changing because the old one started getting gamed, they tend to become partners in the change rather than people looking for the next loophole.

None of it is bulletproof, but a team that is proud of real work is harder to game than any dashboard.

I do not have all the answers. Maybe systems are less like monuments and more like gardens, which need tending, pruning, and the occasional replanting.

Or maybe I'm just gaming the system by writing a think-piece about gaming the system.

P.S. At least I'm not doing it for a T-shirt.