A few things have recently prompted me to question the various claims I hear at work, on television or in the media. Firstly, I’ve been reading the fabulous “I think you’ll find its a little bit more complicated than that” by Ben Goldacre (Bad Science is also an excellent read) and Super Freakonomics, the sequel to .. yes you’ve guessed it Freakonomics , but Stephen J Dubner and Steven Levitt. Both these books look at that way data is presented, how data shapes the decisions we make and generally questions how we design data collection, approach its interpretation and how human behaviour comes into play.
I’m currently designing a significant transformation program and so starting with accurate data is crucial. I often hear wild claims of the benefits that can be achieved, because a particular area is allegedly very inefficient, or how much better the business would be if “we just fixed [insert hot topic of the week]”. Usually when I dig into these types of claims, they rarely have any data to back them up and are based on anecdote or instinct. Both of these sources are notoriously unreliable bases on which to spend substantial sums of money.
Being in question everything mode, two particular claims jumped out this week, firstly because they are both ambiguous and one of them seemed particularly misleading.
The first I have seen a few times an appears on the back of (i think) arriva buses in Leeds and proudly states Buses up to every [x] minutes between A and B, the one i saw this morning was “Buses up to every 30 minutes between Leeds and Keighley” (although why anyone would want to go to Keighley every 30 minutes is beyond me).
The second was on a TV advert for Smart Meters, this proudly claimed “If we all got a smart meter it would save enough energy to light every home in Aberdeen, Manchester and Cardiff”
The problem I have with the Arriva claim is its of no use at all . “Every 30 minutes” is a measure of frequency, and so “up to” suggests this is the maximum frequency of buses and so the claim would be valid if there were only 2 buses a day, separated by 30 minutes. I don’t get how its helpful to proudly display such information on the back of a bus in big letters – Arriva, I’d love to hear from you the customer experience design that said this was a good idea and the mathematician who signed off the phrasing.
The problem with the smart meter claim is another classic sleight of hand. On the surface, it sounds like a good thing – that must be a lot of energy to light all those homes – but enter lies, damn lies and statistics – apart from the details like – what assumptions were made about how much energy each house uses – there are two glaring flaws.
The first is .. over what period … If we all get a smart meter do we save that much energy every year, or does it take 100 years to get the pay back.
But that point, whilst annoying is still irrelevant to the fact that getting a smart meter doesn’t save you energy . Using less energy saves energy. So, getting a smart meter and turning off all the lights, not leaving the TV, Computer, Games console etc. on standby, which are all things you can do without having a smart meter by the way would save energy.
But humouring the smart meter people for a second, lets have a guess at how much energy we’d need to save.
I reckon there are probably 3 million people in Manchester, Cardiff and Aberdeen out of a UK population of 70 million – so 5% in round numbers. Now the advert explicitly said “save enough energy to light all the homes in” . The average home in the UK uses 10 KWh per day of electricity (of which about 10% is lighting ) and 35 KWh of gas. In total 45KWh.
So to light every home in the aforementioned cities, we would need to save 3 million x 1 KWh – or 3,000,000 KWh per day.
But this saving is made from the entire populations entire energy usage of 70 million x 45 KW/h or 3.1 billion KWh . In other words, if we all reduced our energy usage by a whopping 0.09 % .. no thats not 9% thats 9 hundredths of a percent. Or if you gave me one hundred quid, I’d give you 9p back.
Looking at it another way, each household needs to save 0.04kw hour per day. If we use 1KWh for lighting and we have 20 light bulbs in our house we can make the far simpler to deliver claim
“If we all took out one light bulb, we would save enough energy to light every home in Manchester, Aberdeen and Cardiff”
And that’s without the overhead of making the smart meters and driving to my house to install them.
I’m sure (i hope) there’s a flaw in my argument some where as there must be lots of bright people gainfully employed in smart meters.. so please embarrass me and point out where I’ve messed up the maths.