Submitted by Matthew Bowles | Director of Insight & Evidence @ Brown Hound
A health economics webinar grabbed my attention recently. An economist very competently took us through how decisions are navigated with healthcare budgets and patient outcomes. I started thinking about my world, research. Specifically, the different times I have moved towards metrics that help us make decisions in a changing industry. In the middle of this was one rockstar metric: the QALY.
The QALY (Quality-Adjusted Life Year) is defined as one year lived in perfect health for a single person. Market researchers and others too spot an immediate problem with this: an individual’s quality of life is distinct, qualitative, and deeply personal. You cannot fit my perfect health experience into a single metric.
But, putting ourselves aside for a second, the QALY allows us to make very practical and evidenced choices in a complicated environment. A practical example: If a health provider decides to buy into supplying a specific insulin pump, that decision is based on an evidenced expectation of improved patient outcomes. If those outcomes then don’t materialise, for any number of reasons, we can find out why and identify the direct cost of that to the service. The evidence then informs a different approach next time.
There is a potentially uncomfortable usefulness to the QALY: something that can feel analytical, and perhaps even cold at the level of individual experience, but it can still help us find the maximum good for a wider population. This is probably the best health outcome to spend ratio, and the seemingly impossible choices of where our limited health budgets get spent becomes a little bit easier.
It all made me reflect on my life in research so far, quite literally by the metrics.
LIFE BY THE METRICS
Flashback to 2010, a lone consultant, undertaking old school brand tracking for a major grocer. The task on its face: giving various people the certainty to make decisions and help them understand what value our insights could deliver. Over time we collectively weathered challenges like the UK horsemeat scandal, quantifying the public losing and recovering faith in their supermarkets.
As for the metrics, it was classic brand tracking stuff. We had mean scores (at 1 decimal place) that fluctuated inside normal tolerances. This still managed to cause the odd stir, despite our best efforts to focus on robust shifts. My sense was that data sometimes led but often needed to fit the team’s existing narrative, we followed the prevailing winds in the business.
For someone new to all this, it felt at odds with the vision of a young lone maverick analyst breaking down “strategy-changing insights” at the coalface of research. But in hindsight and taking my own personal ambitions out of focus for a beat, this is preferable and more realistic than a twisting vortex of wholly new ideas. Likewise, if our healthcare providers jumped on every new product based on a new forecast or a predicted marginal benefit it would quickly get impractical. There’s a benefit to the metrics being useful, but wisdom runs alongside them, telling us when to run with it.
A few years later, data science is the industry’s hot topic. Our agency pivots a part of mass brand tracking to double down, invest and chase this. The idea of already having the answer, rather than waiting for research timelines to catch up. A lot went on.
But measurement-wise, so much came down to the not so humble Z-score. A typically backend metric thrust into the limelight of client use. Honestly, it did a great deal of the lifting. Fulfilling a promise of automatic insight, our product sorted thousands of comparisons in seconds. Overall, we found the largest and most robust differences fast. Like the QALY, this helps economists perform like for like comparisons and find the top benefit.
Naturally, at every workshop we gave an increasingly condensed stats primer, but once users got it, they were flying. It felt semi-automated for the user. Do all great insight narratives solely come from the biggest sample differences sorted by a single metric? Clearly not! But with a good source, some enthusiasm and curiosity, you can certainly go a long way starting there.
I took a pause, then carried all this into medical device research, and several things happened at once. In metric terms: an explosion of choice, doing more qualitative and developing in applied statistics. We also moved to the countryside and embraced a different way of living.
But it hit differently too. Yes, I was helping manufacturers navigate optimum price and device ranging decisions in developing markets. Yes, I was talking to physicians who could not find the treatment information they needed to do their jobs. We collectively knew, these are real people and lives. It’s not reducing media agency ad spend or solving a theoretical targeting problem. It’s getting closer to decisions that change health outcomes.
Let’s not kid ourselves, before I get carried away. It’s MR, not ER. I am no Clooney, nor a high-flying health economist saving lives or finding and speaking up for groundbreaking policy shifts. But as individuals in insight evidence, we do each quietly manage the fact that what we do lives adjacent to informing these decisions. Research with vulnerable people needs to follow ethical guidelines of course, but it can also go further to respect the fact that they can be vulnerable too.
So, what was the metric explosion? Well, this was the era of triangulation, no longer single source or banging the drum of a specific solution. Not exactly postmodernism, but certainly complex solution building. Multi-method work negotiated internally and sometimes debated across verticals and countries, with clients. The wild west compared to the harmony conjured by a single metric world, and entirely appropriately so.
How do single metrics like our QALY survive in this environment? Well, it is a composite, it requires various measurements to produce it. There are different geographic groups that find consensus on how it is done, but the fundamental identity and purpose remains consistent.
All too quickly, it is 2023. We have clients and many agency side, simultaneously and occasionally literally asking the question: “Is there something we can do with AI?” And the answer is frequently yes. So, we all dived into this and keep on learning based on the outcomes.
From my perspective we shifted focus from metrics. There were increasing quantities of data and an increasing capacity to produce and analyse more of it. Our computers can talk to us; it is in our faces. So really, we started focusing on trust and the quality of our sources. It is still an act of triangulation, and the pre-AI methods, thinking and frameworks for that are thankfully alive and well.
This also happens to be around the time that I went independent and opened our consultancy Brown Hound with my wife. We both do healthcare from different specialisms and know the need for humans at the centre of what is going on, trying to understand it together.
WHEN WILL THIS END?!?
Have we found a fundamental teaching, singular route forward, or an omni-metric that feels as seemingly elegant as the QALY to guide us in today’s nuanced world? Not a chance! And I do not expect to; this field evolves and changes and it is usually for good reasons. But just because things change does not mean we should stop placing faith in where a good metric can take us either.
So, if you will permit me a final anecdote: I am reminded of my first health study years ago, I came from London ad-agency decision making. I am in a backroom in Boston, staring at a tiny plastic patient safety device. The device engineer almost fondly told me of the highs and lows of their 5-year journey on the manufacturing process. In the palm of my hand, the result seemed understated relative to how it was created.
And sometimes I think that about research and metrics too: fundamentally, they seem like a small element, but collectively they add a piece to the stories that we each work so hard to tell.
