Discussion about this post

User's avatar
Austin Caroe's avatar

Reminds me also of Goodhart’s Law: any metric that becomes a target ceases to be a good a good metric.

A PERFECT EXAMPLE was our use of metrics in Afghanistan. SIGAR reports have shown over and over and over that the US military *consistently* used BS metrics in a vain attempt to demonstrate success. They were using the amount of money they deployed as a metric of success!!!! Insanity!!!!

I think your post is generally right on the money.

I would push back with on school test scores, however. To your point, though, failing schools will try to find all kinds of metrics to distract from low test scores, which cannot be manipulated or re-framed. Good schools produce students who do well on standardized tests.

There is a reason that teacher’s unions attack test scores—they shine a spotlight on failure.

Try to find a school with below average test scores but whose students thrive after graduation. They may exist, but it’s probably for reasons unrelated to the school—like most students work in the local factory and do fine after graduation.

I don’t think all data is the enemy. I think it can be part of the “map.” But it’s about finding simple and relevant metrics that aren’t easily manipulated—they are a real measurement of effectiveness, not an easily manipulated measure of performance. For a business this is probably something like simple margins or free cash flow. Again to your point, business that stress about increasing social media engagement and NPS are probably chasing the wrong things.

In conclusion, I largely agree with some important caveats. Great work, Mark!

Expand full comment
Rob |GroktheWorld/OODAAnalyst's avatar

I used to lead a seminar at a military college. The seminar was qualitative in nature, we didn't have metrics. A new boss came in at the top level and wanted to measure the course's impact.

I pushed back and suggested they (and the folks designing the survey) spend a day with me on the obstacle course we used to facilitate the seminar. I wanted them to "feel" the impact, to see the "aha" moments the/my cadets experienced.

They refused. "We have your curriculum," they said.

The last meeting on this I was invited to (my Commander told me my presence was no longer required after this one), I told them, "You cannot measure this impact, particularly without experiencing it."

I lost the battle but told my CC, "Sir, there's going to be a day I'm called on the carpet for a low rating due to factors outside my control."

I was assured that wouldn't happen.

The survey was created and issued to each student.

One day about two months later, I was called on the carpet to explain a lower than average rating.

"Weather," I said, "that day was about two degrees above our knock it off threshold."

"I can't sell that to the Boss" was the answer back.

"It's the facts," I said.

I rotated out of the position not long after that (not early, time was up). The course died on the vine after I left, as a sycophant replaced me. I know that's probably cliche to say, but it's true.

Expand full comment
4 more comments...

No posts