Reminds me also of Goodhart’s Law: any metric that becomes a target ceases to be a good a good metric.
A PERFECT EXAMPLE was our use of metrics in Afghanistan. SIGAR reports have shown over and over and over that the US military *consistently* used BS metrics in a vain attempt to demonstrate success. They were using the amount of money they deployed as a metric of success!!!! Insanity!!!!
I think your post is generally right on the money.
I would push back with on school test scores, however. To your point, though, failing schools will try to find all kinds of metrics to distract from low test scores, which cannot be manipulated or re-framed. Good schools produce students who do well on standardized tests.
There is a reason that teacher’s unions attack test scores—they shine a spotlight on failure.
Try to find a school with below average test scores but whose students thrive after graduation. They may exist, but it’s probably for reasons unrelated to the school—like most students work in the local factory and do fine after graduation.
I don’t think all data is the enemy. I think it can be part of the “map.” But it’s about finding simple and relevant metrics that aren’t easily manipulated—they are a real measurement of effectiveness, not an easily manipulated measure of performance. For a business this is probably something like simple margins or free cash flow. Again to your point, business that stress about increasing social media engagement and NPS are probably chasing the wrong things.
In conclusion, I largely agree with some important caveats. Great work, Mark!
Such good food for thought and practical application. I'm working on a metric for my lead generation system and I need to keep this in mind....particularly that I don't allow the metric to become the measure of success.
I used to lead a seminar at a military college. The seminar was qualitative in nature, we didn't have metrics. A new boss came in at the top level and wanted to measure the course's impact.
I pushed back and suggested they (and the folks designing the survey) spend a day with me on the obstacle course we used to facilitate the seminar. I wanted them to "feel" the impact, to see the "aha" moments the/my cadets experienced.
They refused. "We have your curriculum," they said.
The last meeting on this I was invited to (my Commander told me my presence was no longer required after this one), I told them, "You cannot measure this impact, particularly without experiencing it."
I lost the battle but told my CC, "Sir, there's going to be a day I'm called on the carpet for a low rating due to factors outside my control."
I was assured that wouldn't happen.
The survey was created and issued to each student.
One day about two months later, I was called on the carpet to explain a lower than average rating.
"Weather," I said, "that day was about two degrees above our knock it off threshold."
"I can't sell that to the Boss" was the answer back.
"It's the facts," I said.
I rotated out of the position not long after that (not early, time was up). The course died on the vine after I left, as a sycophant replaced me. I know that's probably cliche to say, but it's true.
Wherever there’s a system, there will be people trying to game it—and this often leads to perverse incentives. Take polling, for example. Polls are only as accurate as their sample: the broader and more representative the group of respondents, the better the data.
However, with very few people answer phone calls or respond to unsolicited text messages these days. Those who do are statistical outliers, and this self-selection introduces significant bias—often in unpredictable but meaningful ways.
Despite this, poll results drive news cycles and shape campaign strategies. They influence donors and voters alike.
In practice, polling has shifted from offering usable insights about a representative public to reinforcing existing narratives about popular support.
Well said. Polling no longer measures opinion, it manufactures momentum.
When response bias becomes the norm, polls stop reflecting the public and start shaping it. The map replaces the terrain, and campaigns steer toward the mirage.
Reminds me also of Goodhart’s Law: any metric that becomes a target ceases to be a good a good metric.
A PERFECT EXAMPLE was our use of metrics in Afghanistan. SIGAR reports have shown over and over and over that the US military *consistently* used BS metrics in a vain attempt to demonstrate success. They were using the amount of money they deployed as a metric of success!!!! Insanity!!!!
I think your post is generally right on the money.
I would push back with on school test scores, however. To your point, though, failing schools will try to find all kinds of metrics to distract from low test scores, which cannot be manipulated or re-framed. Good schools produce students who do well on standardized tests.
There is a reason that teacher’s unions attack test scores—they shine a spotlight on failure.
Try to find a school with below average test scores but whose students thrive after graduation. They may exist, but it’s probably for reasons unrelated to the school—like most students work in the local factory and do fine after graduation.
I don’t think all data is the enemy. I think it can be part of the “map.” But it’s about finding simple and relevant metrics that aren’t easily manipulated—they are a real measurement of effectiveness, not an easily manipulated measure of performance. For a business this is probably something like simple margins or free cash flow. Again to your point, business that stress about increasing social media engagement and NPS are probably chasing the wrong things.
In conclusion, I largely agree with some important caveats. Great work, Mark!
Afghan would be a garden of case studies, I’m sure.
Goodhart and Campbell strategic cousins.
Goodhart shows why the metric stops working; Campbell shows how the system bends itself to protect the numbers.
Using test scores:
Campbell explains why schools chase scores over real learning (corruption).
Goodhart explains why those scores stop meaning anything (breakdown).
For 5GW:
Campbell reveals how metrics enable deception.
Goodhart reveals how metrics fail under pressure.
Campbell exposes the theater 🎭
Goodhart other diagnoses the collapse 📉
You need both to know both.
Such good food for thought and practical application. I'm working on a metric for my lead generation system and I need to keep this in mind....particularly that I don't allow the metric to become the measure of success.
I used to lead a seminar at a military college. The seminar was qualitative in nature, we didn't have metrics. A new boss came in at the top level and wanted to measure the course's impact.
I pushed back and suggested they (and the folks designing the survey) spend a day with me on the obstacle course we used to facilitate the seminar. I wanted them to "feel" the impact, to see the "aha" moments the/my cadets experienced.
They refused. "We have your curriculum," they said.
The last meeting on this I was invited to (my Commander told me my presence was no longer required after this one), I told them, "You cannot measure this impact, particularly without experiencing it."
I lost the battle but told my CC, "Sir, there's going to be a day I'm called on the carpet for a low rating due to factors outside my control."
I was assured that wouldn't happen.
The survey was created and issued to each student.
One day about two months later, I was called on the carpet to explain a lower than average rating.
"Weather," I said, "that day was about two degrees above our knock it off threshold."
"I can't sell that to the Boss" was the answer back.
"It's the facts," I said.
I rotated out of the position not long after that (not early, time was up). The course died on the vine after I left, as a sycophant replaced me. I know that's probably cliche to say, but it's true.
Wherever there’s a system, there will be people trying to game it—and this often leads to perverse incentives. Take polling, for example. Polls are only as accurate as their sample: the broader and more representative the group of respondents, the better the data.
However, with very few people answer phone calls or respond to unsolicited text messages these days. Those who do are statistical outliers, and this self-selection introduces significant bias—often in unpredictable but meaningful ways.
Despite this, poll results drive news cycles and shape campaign strategies. They influence donors and voters alike.
In practice, polling has shifted from offering usable insights about a representative public to reinforcing existing narratives about popular support.
Well said. Polling no longer measures opinion, it manufactures momentum.
When response bias becomes the norm, polls stop reflecting the public and start shaping it. The map replaces the terrain, and campaigns steer toward the mirage.