Back in the last “lockdown” I linked to various pieces of work by other economists attempting to make sense of, evaluate etc, choices the government was making. There was Ian Harrison’s work challenging some of the modelling estimates the Prime Minister liked to wave around and some aspects of the “Level 4” restrictions. There was an early attempt at a cost-benefit analysis by Bryce Wilkinson of the New Zealand Initiative, and another exercise looking at a similar question in a different way by John Gibson at Waikato University. There was another exercise that I never wrote about, but which is reported and linked to here, by Martin Lally, a consultant economist and former Victoria University academic.
What was striking, even at the time, was that there was no sign that the government had commissioned from officials, or officials had undertaken anyway, any sort of serious cost-benefit analysis of the sorts of intervention they were looking at and imposed. It always seemed likely at the time that there was nothing of the sort – the public sector had, after all, been woefully underprepared, sluggish in getting any serious planning underway, and complacent for too long that this was largely someone else’s (PRC’s) problem Anyway, when the government finally got round to publishing the relevant documents, sure enough there was no serious structured attempt to cost and evaluate alternative policy options. (It is not, I hasten to add, that any cost-benefit analysis can give one “the” answers, but it provides a disciplined framework to analyse the options, assumpions and sensitivities.) But there was nothing – even though the New Zealand authorities had the best part of two months of lead time.
These issues take on a fresh salience with this week’s out-of-the-blue partial lockdown of Auckland, and the government decision later today. It prompted me to finally go and take a look at an exercise undertaken by an economist at the Productivity Commission in early May, illustrating for the benefit of The Treasury – who we used to assume were the champions of robust cost-benefit analysis – how the decision in late April on whether to extend “Level 4” for another five days might have been rigorously analysed in a careful cost-benefit framework, looking only at the marginal costs and benefits of the two options the government had had in front of it. The author concluded that, with the information available at the time, the extension was probably not justified, but that is less relevant than the fact that an economist at another agency was having to do this for The Treasury after the event. Apparently neither The Treasury nor ministers had been interested in getting such analysis done when the decisions were being made.
Restrictions – border restrictions – have remained in place, but there seems to have been relatively little interest in evaluating the costs and benefits of those choices. But this week’s restrictions have brought the issue back into focus. There have been a couple of newspaper articles, notably in today’s Herald: this by Kate MacNamara, and a column by (newly returned from working for the National Party) Matthew Hooton. MacNamara explicitly ends her piece with the argument
“There will be a time when the best option is to ease border restrictions, abandon lockdowns, and let our health system, including tracking and tracing, do the heavy lifting. We need credible analysis to help us know if that time is now.”
I’d say “perhaps” to the first sentence – and it remains troubling that there is no identified or championed (by the government) credible exit strategy from our current eliminationist/closed-borders model – but would strongly echo the call for serious, open, analysis on the issue and options.
Martin Lally’s latest paper on a cost-benefit approach isn’t that analysis – we need proper marginal analysis on the costs and benefits from here, with what has happened to now in principle largely irrelevant (sunk costs and all that). But Martin’s paper, which he has given me permission to share
Martin Lally cost-benefit assessment of Covid lockdown August 2020
is still a useful look back at the merits of choices made over recent months, and probably sheds at least some light – poses some questions – on how the choices going forward might look.
His conclusion is as follows (QALY = “quality-adjusted life year”)
This paper considers the effect of the New Zealand government adopting a suppression policy versus a milder mitigation policy, with the actions of other governments taken as given. The cost per QALY saved from doing so would seem to have been vastly in excess of the currently used value for a QALY of $45,000. Consideration of alternative parameter values and recognition of factors omitted from the analysis would not likely reverse this imbalance in cost per QALY saved versus currently accepted figures for the value of a QALY. The suppression policy was therefore dramatically inconsistent with long-established views about the value of a QALY.
The broad approach is to look at lives saved by the government’s elimination approach and the (primarily) economic costs of that strategy. Neither is necessarily straightforward. On the economic side, one sometimes hears champions of the government touting a view that there is no such economic cost – in fact, I heard former Labour leader Phil Goff make exactly that claim this morning. Locking down hard, while costly initially, is – these champions conveniently claim – its own reward; initial losses more than outweighed by the subsequent gains (faster sustained recovery etc). But there is no actual evidence for these claims – at best such an outcome could be considered as one scenario. (In the early days, the PM was claiming support from 1918, suggestions I looked at here.)
Perhaps that line might have seemed more plausible to some just a few days ago. But then, with essentially no notice, our largest city was flung back into a partial-lockdown, and whatever choices the government announces today, we are told to expect more of these events, timing and size of course unknown and unknowable. So we take further real output losses now and – perhaps at least importantly – fresh huge uncertainty (affecting all manner of firms, and households too). Perhaps the government can finally fix up border testing – isn’t it just staggering that two-thirds of people working at aiports/MIQ facilities etc haven’t been tested at all? – reducing the chance of further outbreaks/lockdowns. But even if that were done as best as humanly possible, it wouldn’t change the limitations of the closed border itself.
And the difficulty for champions of the “own reward” model is the absence of a compelling exit strategy. If we could count on the virus simply dying out, going away, by some clearly defined date next year, the calculations change quite a lot. There is a credible exit strategy then, and we just have to hold on til then. Similarly it we could count on a highly effective vaccine being generally available by some clearly defined date next year, again things look more encouraging for the “own reward” story. Perhaps those too are scenarios to add into a serious evaluation of the strategy. Along with scenarios in which there is never a very effective vaccine and/or the virus remains much as it is indefinitely.
In any case, what Lally does is to assume that some – quite moderate – proportion of the difference between the Treasury’s GDP forecasts from last December and those from this year’s Budget should be treated as the cost of the elimination approach. His central case assumes 25 per cent. That may be too high.
The other side of the equation is, of course, lives saved (and reductions in impairments to the quality of life, of those with serious but non-fatal Covid). Of course, some of that early modelling suggested catastrophic losses if we hadn’t gone to a fairly severe lockdown. But if, as Harrison suggested, those numbers didn’t look that plausible at the time, they look much less so now. Lally focuses on the case of Sweden, which has pursued – not always well – something closer to a mitigation policy.
To date Sweden has suffered 570 deaths per 1m of population and the increase in the rate is tailing away to zero. New Zealand’s population of 5m implies 2,850 deaths under a Sweden-style mitigation policy. The QALYs saved would then be (2,850 – 22)*5*0.5 = 7,070.
It is a sample of one, but again he illustrates that you can assume a materially higher numbers of QALYs saved and the calculations still don’t end up very favourable to the New Zealand approach. A further caveat is that, although he notes the point, Lally does not explicitly allow for the QALYs saved in respect of the people with serious non-fatal Covid cases. The Productivity Commission piece does include some estimates, and if I’ve read document correctly, the effect is to double the overall QALYs saved.
Lally is very conscious of the sensitivities in his analysis. This is the last extract I’m going to quote.
The parameters used in this analysis are debatable. The death rate under a mitigation policy may be much larger than estimated here. If it is doubled, the cost per QALY saved would halve to $4.25m, but would still be 94 times the usually accepted figure. The GDP loss from the current path relative to that if there is no curtailment in economic activity could be smaller. If it were halved, in addition to the death rate being doubled, the cost per QALY saved would fall further to $2.12m but this would still be 47 times the usually accepted figure. The remaining parameter is the proportion of the GDP loss due to lockdown rather than mitigation, which is unknown. However, any reasonable proportion will produce a cost per QALY saved well in excess of the usual figure of $45,000.
(Incidentally, I prefer a high number for the value of a QALY – the Productivity Commission paper discusses some of the options.)
My point in this post is not to articulate a strong personal view on what the government should have done, or should do now. As I’ve said in past posts, my visceral reactions tends to be more cautious than my analytical one, and one shouldn’t discount visceral reactions. And in the last lockdowns, my bigger concerns were about the overreach in many of the non-economic restrictions – remember the government that totally banned funerals, or a solitary swim at a quiet suburban beach.
But I reckon there is crying need for more analysis – open and transparent, disciplined analysis, exploring a wide range of asssumptions and scenarios. As I noted, Lally’s paper isn’t that for the period ahead – we need marginal analysis from here, that explicitly takes account of the uncertainty of the relevant end dates – but it is still worth reading, perhaps especially so in conjunction with the (slightly longer, more detailed, and better-tabulated) Productivity Commission piece, which represents the sort of analysis we should be expecting from our core government officials – notably The Treasury – were they adequately (well, excellently) doing their job. And as the government ploughs on – apparently supported by all other parties – with their eliminationist approach, we deserve a credible, carefully evaluated, exit strategy. At present, there is none.
UPDATE: Lally has responded to my point that his paper is not a marginal approach (costs and benefits from here) and so can’t shed light on choices from here, and has added a paragraph (in this version) offering one way of looking at that question concluding that
“Switching to a Sweden-style approach is therefore clearly warranted.”
Those who believe that virtue is its own reward (as above) will certainly not be persuaded. My own reaction is that – as per my final paragraph – more analysis is needed, drawing on the combined expertise of economists and epidemiologists.
croaking cassandra 
Hi Michael
Thank you for continuing with this train of thought. Reflecting on our choices and options regarding our response to Covid-19 is long overdue. I do note that there is a well qualified group in New Zealand who have been doing just that:
https://www.covidplanb.co.nz
Here is their most recent commentary asking ‘do we really need yet another lockdown’?
https://www.covidplanb.co.nz/our-posts/do-we-really-need-yet-another-lock-down/
It’s time for the public to realise just how unrealistic it is to pursue virus eradication by any means, and how unsustainable the Government’s present policy settings have become.
LikeLiked by 3 people
Cost Benefit Analysis? Surely it just needs to ‘feel right’?
LikeLike
Reblogged this on Utopia, you are standing in it!.
LikeLike
While cost benefit analyses may be useful if we are returning to the economy we had a year ago, is that really the situation?
According to some experts we will soon be a tenth of the way through the last ten years before temperatures reach 1.5% to 2% higher than ‘normal’.
Perhaps Covid-19 is giving us space to develop a very different economy to take account of flooded coastlines, ‘one in a hundred year’ droughts/floods/heatwaves every two or three years, restrictions on drinking water etc etc.
In this new era we may also be grappling with Covids 20 – 30.
While it may be cost-effective to let people die right now, what happens when those deaths climb from 570 per million to 10 or 100 times that?
As to the value of lockdowns has anyone costed the benefits of fresh air and the absence of toxic fumes and particulates around us?
LikeLike
The government and Jacinda Ardern has clearly identified Bungy jumping as a key industry to save. I think they also believe tying bungy knots is a highly skilled profession in our new post covid 19 future. Industries such as Tiwai point aluminium, smelter, Marsden point oil refinery and New Plymouth methanol with the loss of 7,500 jobs are considered not worth saving sadly.
LikeLike
Thomas Lumley, Professor of Statistics at the University of Auckland, said: “The Productivity Commission are arguing that the extra week in lockdown was unnecessary and very expensive. Their analysis is wrong; it does not seem to consider whether and how much the extra week reduced the risk of needing a second lockdown, which was part of the reason for doing in. I’m not saying the extra week was the right decision — you can’t tell, without modelling the extra risk, which they didn’t do. It’s like saying insurance is not cost-effective because your house didn’t burn done. Insurance may or may not be cost-effective, but that isn’t how you tell”.
Shaun Hendy, Professor of Physics at the same university, said the report was “based on a fundamental misunderstanding of the nature of disease transmission dynamics and, as a result, grossly underestimates the benefits of the level four extension”.
These are people whose views can’t be taken lightly. Michael, do you have any comment on their criticisms of the Productivity Commission report?
LikeLike
Not particularly: as I said in the post I was (much) less interested in the result than in the fact of the attempt, and that Tsy/MoH had not done a similar exercise.
Re Lumley’s point,as I recall I had a somewhat reaction when I read thru the PC report.
More generally, I think it is good the PC work was done, good that it was published (OIA and all that), but I would be interested in any response the PC or the author had to the specific criticisms – esp Hendy’s.
(Personally I don’t have a dog in the fight about the extension as I thought then, and think now, that the best indications were that we never needed anything quite as severe as level 4 in the first place.)
LikeLike
When you consider that we were in lockdown for 6 weeks to eliminate Covid19, a 2 week quarantine at the border seems rather light to prevent a 2nd wave. Therefore either 6 weeks is extreme overkill or 2 weeks border quarantine is doomed for failure. This means that it is not actually an extra week but more like 3 or 4 extra weeks in unnecessary lockdown.
LikeLike
A couple of points (which overlap with the PlanB team’s article referenced above:
1. treatment protocols for Covid have come a vast way since March, and the Case Fatality Rate (CFR) for the disease has dropped drastically. Early lockdowns which have have been justified by “excess of caution in absence of evidence” do not make sense now that we have that evidence to hand. Most people are unaware of the drop in CFR, as the MSM has switched from reporting death rates to reporting case rates
2. there is a strong correlation between diseases of poverty (up to and including suicide) and the employment rate. Consciously increasing the unemployment rate in the course of lockdowns costs lives. The common argument that it’s a trade-off between lockdowns (ie saving lives) and purely economic costs is a facetious one. Unfortunately I’ve not seen any good modelling of the QALY costs of the interventions, which needs to be added to the economic costs for a full accounting
LikeLike
1. treatment protocols for Covid have come a vast way since March, and the Case Fatality Rate (CFR) for the disease has dropped drastically.
—————-
But there’s still no agreed worldwide precise standard of reporting on what constitutes a Covid-19 death, or is there? Perhaps much better indication would be the ratio of hospitalization?
LikeLike
True there’s no standard, eg the UK just adjusted downward it’s Covid toll by 5k deaths (true!), as their previous metric has anyone who died having previously tested positive for Covid (regardless of the elapsed time between test and death) as a Covid death. True statistical nonsense..
I’m not sure the rate of hospitalisation would be a better metric though – hospitalisation rates have dropped a lot due to improvements in early treatment, but survival rates in hospital have also improved, so it’s a bit of a wash.
Best I think to acknowledge inter-country rates have reporting differences which make direct inference difficult, but studies across multiple juristictions (eg this Lancet study: https://www.thelancet.com/journals/eclinm/article/PIIS2589-5370(20)30208-X/fulltext) are likely to be accurate in overall trending.
LikeLike
Interesting post. I haven’t lived in New Zealand for a few years now and don’t think I am informed enough to comment on the what the Government should do for this, or potential future lockdowns, but a perspective from the UK where the situation is a bit different:
I think there are many reasons why reaching a ‘rational’ perspective on this issue is going to prove incredibly difficult and I genuinely fear that without a vaccine rapidly being developed we are collectively (perhaps with the example of the odd country like Sweden) sleepwalking into causing the most unnecessary and damaging self-inflicted economic and social harm that humanity has ever known (with the exception of acts of explicit evil, such as world wars).
Very simply I think the ‘plan B’ people were wrong at the time of the initial outbreak of the virus, when TV screens were full of scenes from Italian hospitals, but that six months later that perspective is worth re-visiting given what we know about how the virus affects, how to treat it and the costs and benefits of prioritising one group of lives over another, but that there is a critical mass of people who’s view of risk and the threat posed by this is stuck in Italy in March/April, and will prove very, very difficult to dislodge. The so called ‘V shaped recovery’ depends on this genie being put back in the bottle and people being comfortable enough to live relatively normal lives again. The alternative is ‘staying at home to save lives’ indefinitely and at the same time immediately going backwards ten years or so in economic development (not that there has actually been much in the way of that over the last decade or so in terms or per capita success but whatever).
The point on QALYs is an interesting one, there are all sorts of figures out there. I would more crudely suggest though that in the UK the average value for somebody old enough to die en masse from coronavirus is somewhere approaching £infinite and the average value for somebody who is not is relatively much closer to £0.
Why do I think this is? In the UK, according to the Office for National Statistics, 51,708 deaths have occurred where coronavirus is recorded on the death certificate. Of these, less than 1% are people under 45. 42% are over 85. 75% are over 75. 91% of them have a pre-existing condition (the mean number of pre-existing conditions is >2). A quarter of all lives lost had Dementia /Alzheimers. On the other hand the Government estimates that 200,000 people will lose their lives who should not otherwise have from the effects of cancellation of surgery, medical care, diagnosis of cancers etc. We are also keeping children out of school, revising their exam grades downwards in a cruel and arbitrary way and spending seemingly incalculable amounts on wage subsidies for jobs which cannot be undertaken under social distancing rules and will disappear if/when the Government is brave enough to cancel the scheme, leading to mass unemployment and the associated ills that brings.
To me it seems grossly in-just the damage being inflicted upon the current generation of young people (and the not so young who will also face going backwards in their lives as well) and they have been treated as an afterthought so far – it is inconvenient to discuss their plight and it seems largely ignored in the face of whatever the new virus obsession of the news hour is (taping off park benches, mask wars, the ‘R value’, ‘essential travel’, whether outdoor activity should be limited to an hour at a time, etc etc).
LikeLiked by 3 people
Interesting comments. As you say, hard to tell what is best. At a worldwide level perhaps a complete shutdown for two months might be enough to decisively break the outbreak, but there is no coordination mechanism (or probably will) to bring that about.
LikeLike
Well said – I very much agree, especially on the inter-generational consequences. Mind you, the Boomers have always been a most remarkably selfish generation – eg enjoying the infrastructural bequest of previous generations, then imposing user-pays on subsequent generations, privatising public assets etc. That subsequent generations should bear excessive burdens to protect them from their fear of death seems more a continuation of the trend.
It’s the mind-virus aspects of Covid that boggle me most. It is nigh-on impossible to have any constructive discussion around facts and options with the vast majority of people, who have thoroughly absorbed the official rhetoric. Yet the economic consequences of the current decisions will be perhaps the most profound of any public decisions in our lifetimes. And they’re all being essentially done by fiat, based on flimsy evidence, and without debate. If someone had told me a year ago this was to be lived reality in 2020 I’d have laughed in the face.
LikeLike
I agree. The vast majority of people think the value of a life = $infinity, in this case. They ignore the “normality” of vehicular deaths, deaths due to the cost of unaffordable drugs etc.They have been fooled (in my opinion) by the elimination strategy, which is really now we are told, the periodic elimination strategy, punctuated by periodic lock down strategy. It is all “costless” as we using the governments money and NZ is such a great place to be trapped etc.
To me it has never been a strategy, it was a tactic. The strategy is one of hope, hoping for a vaccine. Meanwhile the hidden costs mount, and will soon become very obvious.
LikeLike
At the moment our ICU beds are empty of patients, nurses and doctors are twiddling their thumbs waiting for covid19 patients. In the mean time, thousands of probable patients of cancer and other diseases are not getting diagnosed or dying on the wait list. The mind boggles that we are not counting the deaths from not covid19 as the cost of lockdowns.
LikeLike
(not disputing the point that there will be higher cancer etc deaths later) I saw a chart of weekly deaths data for NZ the other day showing that total deaths in NZ this year are running quite a bit below where one would normally expect, apparently reflecting the way in which rest home lockdowns and other distancing restrictions etc have reduced the incidence of the flu etc this winter.
LikeLike
The covidplanb symposium today was excellent, with speakers steering clear of party politics on favour of wider thought and evidence.
Even David Seymour was remarkable in restraint.. some excellent thought gone in.
I’ll post the link to the recorded YouTube once it’s up.
LikeLike
The PM keeps saying “the best economic response is a strong health response”, or words to that effect. No-one has challenged her when she’s said that (perhaps because she only says it in speeches), but the need is crying out.
I wish Bill English was in charge. He liked data and economic analysis (when it suited him).
LikeLike
Jacinda Ardern also says, A lie is never a lie when it is aspirational.
LikeLiked by 2 people
Reblogged this on The Inquiring Mind and commented:
Well worth reading
LikeLike
Michael
Im confused by a couple of aspects of the CBAs quoted:
1. You show QALYs saved = 7,070. Based on Sweden suffering 570 deaths per 1m people, implying that NZ would have had 2,850 deaths under a Sweden-style mitigation policy. You then calculate QALY as (2,850 – 22)*5*0.5 = 7,070. I presume the 5 is an average life expectancy of 5 years. But what is the 0.5?
2. I think you are indicating that Lally found a base cost of the C-19 policies per QALY of $9.5m. Is that 20% of GDP (circa $67B), which he noted could be reduced by 75% to give $2.1m per QALY, and would still be 47x the “official” QALY value. But where does the$67B come from? It is an estimate of the multi-year reduction in GDP? Does it take into account that Sweden’s GDP has also fallen?
I agree with the basic point of your articles (ie that there is a case for some CBA in deciding on policy initiatives), but Id like to understand the the estimated QALY saving and how the cost is calculated.
Thanks
Tim
LikeLike
Hi Tim
I’m pretty sure that the 0.5 in Martin’s “lives saved” is just our population (5m) relative to Sweden’s (10m).
Not sure about $67bn, but $87bn is the difference between the Treasury forecasts from Dec last year and those at the time of the Budget. Martin then uses an assumption – and it is only that – that a quarter of that loss (roughly $22bn) arises from our elimination approach rather than mitigating (ie the bulk of the GDP losses would have happened even if we’d followed a Swedish approach. As I note, it is possible that over a five year horizon, we might actually gain by following our approach – that is certainly the govt’s stated view – altho in the short-term (during the lockdown) it is likely that GDP losses will have been greater by choosing the stringency of a NZ lockdown vs the alternative of a mitigation approach.
LikeLike
No it isn’t – This is arbitrary discounting of the QALYs saved that he has applied to his equation and with no reference and no basis. What he is saying is that a typical person dying from COVID who he describes as ‘old’ and ‘suffering from at least one pre-existing condition’ is valued by that person as half (0.5) of what it would be living in perfect health. Being ‘old’ is defined as >65 years!!! This is brutal discounting – there are very few chronic medical conditions that lead to a quality of life that bad. You can translate this as ‘you can live for five years with your medical condition and then die or you can get rid of your medical condition but die in 2.5 years. This is a major flaw (amongst many – most notably the tortuous reasoning behind using Sweden as the reference point but then why not to and the complete omission of any healthcare costs from COVID other than years of life lost).
This is a non-peer reviewed paper written by a non-health economist. Frankly, it wouldn’t pass peer review…
LikeLike