Truth-telling systems and incentive systems

There are two kinds of measurement systems: Truth-telling systems and incentive systems. A system can be one, the other, or both.

Kirk Golsdberry’s maps of expected value for NBA shots completely changed how teams thought about the game and their roster.

I’ve developed a bit of a niche expertise over the last decade: Multi-touch digital marketing attribution. It’s not something I’ve sought out. In fact, it’s not even something I’m particularly interested in per se. I’ll never make it the core of all my work. But going deep on marketing attribution has proven, time and again, to be fertile ground for sussing out how the people running a company understand the business they’re in. Attribution is like a tiny porthole: Look through and you can see all the way to the horizon. I can’t help but press my face against the glass.

Every company I work with has an attribution system. It’s easy to understand why: They market to their customers and want to know what’s working and what’s not, and “attribution” broadly speaking is the way to do it. Each company collects signals — impressions, clicks, coupons, survey answers, A/B tests, etc. — tracking their marketing activity, and then their attribution system processes these signals to report on which activities drive growth.

If a prospect clicks on a Facebook ad and then buys a product, the attribution system will give Facebook a little credit. If a prospect uses a coupon from a podcast, the attribution system will give the podcast a little credit. If a prospect clicks on a Facebook ad and then uses a coupon from a podcast, the attribution system will work out how much credit to give to each.

Younger companies measure their marketing simply. That’s as it should be — most of their activities are simple, so measurement should be too.

I usually get pulled in when things start to get a little more complicated. Instead of marketing in one or two channels, they’re marketing in four or five; instead of spending millions, they’re spending tens of millions.

The CEO or the head of marketing asks, “What’s the right way to do attribution?” What they mean is, “What system will tell us The Answer?” They imagine a machine that can fill out the perfect budget, a forecast that can tell them exactly where they’ll be in a year, a magic eight ball that will tell them where to spend and how much and what they’re get for it.

In short, they imagine a truth-telling system. But that’s not what they need.

What these companies need, most of all, is an incentive system. The Answer they imagine doesn’t exist. They need, simply, to decide heading and speed. Do more, or do less. Warmer, or colder.

Where should we spend more tomorrow? Where should we invest less? How will we know when it’s time to adjust course? (The answer to that last question is to carefully measure how you got to where you are — dead reckoning.)

The attribution system that tells you, “Give a little credit to Facebook, and a little to that podcast” isn’t telling you an objective truth. It’s not giving you The Answer. It’s just communicating a simple, directional opinion: “Do a little more of that, whatever it was.” The system is producing dopamine to course through your organization. Again: Incentives, not truth.

Of course, the right incentives will help you approach the truth over time. You’ll get closer and closer, like a game of Marco Polo. And like Marco Polo, you’re aiming at a moving target. So warmer or colder, closer or further, more or less, are all better signals than Right or Wrong.

---

Over time every measurement system becomes an incentive system.This happens even if the systems aren’t particularly good at telling the truth. “What gets measured gets done” is, as often as not, uttered with a resigned sigh.

The fun is in figuring out when a system is approaching the truth and when it is not.

In basketball, the score is both an incentive system and a truth-telling system. It’s almost tautologically a truth-telling system — the team that scores the most points wins, period. And it definitely acts an incentive system — seeing points on the board is what keeps teams striving.

But individual points? Rebounds? Shooting percentage? They’re relationship to the truth is less clear. Is a player who scores 20 points a game better than one who scores 15? Does that tenth rebound, which might come at the expense of a teammate, really improve the team’s odds of winning?

“Counting stats,” as they’re called, are definitely an incentive system. Players will go to extreme lengths to grab their tenth rebound or win the scoring title. Whether these stats tell the truth, though, is another question entirely.

To “solve” this problem, statisticians developed advanced measures that have a more provable relationship to winning. That’s how we’ve ended up with the alphabet soup of VORP and WAR and PER and OBPM and so many others. These are, in theory, better truth-telling measurement systems. Unlike rebounds, typically a player with a higher WAR contributes more to winning than a player with a lower WAR.

But do these advanced stats actually work as incentive systems?

Not on the players, for the most part. Most players don’t know what their VORP is, and even if they did, they wouldn’t know what to do to improve it. The signal’s too complex to function as an incentive.

The few advanced measurement systems that do change player behavior are those that can be distilled to utter simplicity. The most influential distillation is this one: “Three is more than two.” That basic insight — that the expected value of a three point shot is often higher than a two point shot — revolutionized basketball starting about a decade ago. The result is teams take way more three pointers than they used to. (So much so that it’s driving some people crazy and leading to all kinds of fun proposals to fix things.)

Advanced metrics have been much more effective incentive systems for managers than for players. Coaches run totally different plays than they used to, designed to produce more three pointers. And general managers used advanced measurement systems to decide which players to sign and which to let go. Over time, the league’s player mix has rebalanced toward those who do things rewarded by advanced stats. Individual players have had a harder time adapting, but the market for their talents has changed, and fast.

Here’s where that leaves us: Advanced stats make for a good truth-telling system because they’re aligned with winning, which is the desired outcome. But their effectiveness as an incentive system depends on the actor. Advanced stats are often ineffective incentives for players, but very effective as an incentives for managers.

Most interestingly, from the perspective of the league — which doesn’t care about winning or losing, but instead about entertainment — the advanced stats are not a good truth-telling system at all. More three pointers is not necessarily more enjoyable for the fan, or for TV viewership, or for revenue. There’s constant chatter about the league changing rules to short-circuit the incentives that advanced stats created for managers, which some suspect have made basketball less entertaining.

---

Whether it’s marketing or basketball or practically any other goal-oriented activity, measurement systems are simply tools.

When it comes to approximating the truth, you will invariably be left to ask, “What is it I actually want?” For business and basketball that may be easy, but for many other areas it’s hard. And it’s always harder than meets the eye.

And when it comes to generating incentives, you must first answer, “Incentives for whom?” You won’t just have to decide what you want to ultimately incentivize, but how you want to distill that into simple signals they’ll understand. The simple signals are what produce action. “More of this, less of that” is better than complex statistical output. “Three is more than two” is better than “the expected value of this shot will increase your VORP by 0.036.”

These two lines of questioning — “What do I want?” and “Who do I want to incentivize and how?” — are what I find myself coming back to again and again. Getting clear answers to both is the only way to design the system you want.

---

---

---

Because it’s so consistent with the themes of this post, this week’s recommendation is a short article in NY Mag about a decision coach. It’s amazing how often people are looking for simple clarity and decisiveness. “‘One thing I’m good at is getting to the heart of a problem, seeing what a person really wants to do, and figuring out how they should do that,’ Wulfhart says. ‘I’ll give you a real opinion, no waffling.’”