The Limitations of QALY: A Literature Review

Introduction: The Quality Adjusted Life Year (QALY) is a recognised metric used to evaluate new and innovative healthcare treatments and optimise resource allocation via rational and explicit methodologies. This review examines present limitations of the QALY metric and foreseeable challenges linked to the advancement of regenerative medicine. Methods: The extant literature was reviewed through electronic searches of four key databases; namely Medline, EMBASE, Econlit and Cochrane. Manuscripts were selected according to pre-determined inclusion criteria. Results: Three common themes emerged concerning the limitations of QALYs. These were ethical considerations, methodological issues and theoretical assumptions and context or disease specific considerations. The Limitations of QALY: A Literature Review Pettitt DA1,2*, Raza S3, Naughton B3,8,9, Roscoe A3, Ramakrishnan A3, Ali A3, Davies B1,4, Dopson S3, Hollander G2,10, Smith JA1,4 and Brindley DA1-3,5-7* 1The Oxford – UCL Centre for the Advancement of Sustainable Medical Innovation (CASMI), The University of Oxford, New Richards Building, Old Road Campus, Oxford, OX3 7LG, UK 2Department of Paediatrics, University of Oxford, Children's Hospital, John Radcliffe, Oxford, OX3 9DU, UK 3Said Business School, University of Oxford, Park End Street, Oxford, UK 4Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Old Road, Oxford, OX3 7LD, UK 5Centre for Behavioural Medicine, UCL School of Pharmacy, University College London, BMA House, Tavistock Square, London WC1H 9JP, UK 6Harvard Stem Cell Institute, Cambridge, MA, 02138, USA 7USCF-Stanford Center of Excellence in Regulatory Science and Innovation (CERSI), USA 8Oxford University Hospitals NHS Foundation Trust, Headington, Oxford OX3 9DU, UK 9Institute for Science and Technology in Medicines (ISTM), Keele University, Staffordshire, ST5 5BG, UK 10Weatherall Institute of Molecular Medicine, University of Oxford, John Radcliffe Hospital, Headington, Oxford OX3 9DS, UK *Corresponding author: David A Pettitt, The Oxford – UCL Centre for the Advancement of Sustainable Medical Innovation (CASMI), The University of Oxford, New Richards Building, Old Road Campus, Oxford, OX3 7LG, UK, E-mail: david.pettitt@hertford.ox.ac.uk David Brindley, Department of Paediatrics, University of Oxford, Children's Hospital, John Radcliffe, Oxford, OX3 9DU, UK, E-mail: david.brindley@paediatrics.ox.ac.uk Received February 16, 2016; Accepted March 22, 2016; Published March 29, 2016 Citation: Pettitt DA, Raza S, Naughton B, Roscoe A, Ramakrishnan A, et al. (2016) The Limitations of QALY: A Literature Review. J Stem Cell Res Ther 6: 334. doi:10.4172/2157-7633.1000334 Copyright: © 2016 Pettitt DA, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.


Introduction
Although the QALY metric represents the most rigorous methodological tool available for evaluating new therapeutics or technologies, a number of limitations in its application currently exist. These are likely to become more apparent as regenerative medicines are increasingly adopted and the QALY calculation methodology may require refinement to realise the financial advantages and opportunity costs such interventions may convey.
The Quality Adjusted Life Year (QALY) is a recognised metric used by health economists, governments and healthcare specialists, amongst others, to evaluate new and innovative healthcare treatments. It enables optimisation of resource allocation via rational and explicit methodologies. The QALY, which was popularised in the 1970s in response to a need for improved decision-making around healthcare expenditure, has been effectively utilised in the United Kingdom (UK) and has secured the UK's position as a world leader in the application of QALYs to healthcare funding. This has largely been instrumented via the National Institute for Health and Care Excellence (NICE), whereby technology appraisals are conducted at the request of the Department of Health using the QALY measurement.
The QALY has established itself as the leading metric to perform cost effective analysis in evaluating new interventions. QALYs allow for the standardised measurement of health outcomes, thus allowing for comparisons across different disease and population subtypes. QALY is a composite metric that takes into account both morbidity (quality of life) and mortality (survival). The general calculation of a QALY is:

QALY = Time × Utility
Combining both quantity and quality of life into a single index value centres on the utility concept, which incorporates the idea that quality of life can be measured or quantified. When calculating QALYs, a graph is often constructed to visualize the expected quality of life improvements that a treatment will produce over time, as seen in Figure 1 [1].
Measuring quality of life (QOL) is a challenging process and numerous methods exist to measure quality of life or utility. The most common are listed in Table 1.
In direct methods, patients are asked how they value different health states, whereas with indirect methods, patients complete a questionnaire and an algorithm is then used to compute a utility score integrating both individual and community perspectives.
Since the popularisation of QALYs, clinicians, researchers, and ethicists have identified a number of limitations with the metricrelating to time factors, utility factors and algorithm variation. These limitations can be considered as ethical, methodological, and diseasespecific or contextual in nature. Furthermore, such limitations may be exacerbated as we transition into an era of precision (or "personalized") medicine, employing novel regenerative medicine technologies such as cellular based therapeutics and gene therapies.

Literature Review Process
Numerous articles discussing the limitations of QALYs exist in the extant literature; however, a succinct review of the topic has not been performed. Globally, governments are employing QALY calculations to rationalise multi-billion dollar healthcare investments, and thus, there is a requirement for an accurate, current and rigorous review of QALY limitations -the objective of this review. Ultimately, improved understandings of QALY limitations will enable policy makers to better-inform and improve their decision-making processes for existing and imminent medical therapies.
Broad inclusion criteria were considered most appropriate for a topic of this nature and therefore all types of article were considered eligible. Four key databases were electronically searched, Medline, EMBASE, Econlit and Cochrane. Truncation was used in all search strategies across all databases. Search terms and key words included "limitation, shortcoming, weakness, drawback, disadvantage, deficiency, fault, flaw, imperfect, detriment, or problem" combined with "QALY, quality adjusted life year".
Articles were selected based on pre-determined inclusion and exclusion criteria. Inclusion criteria included all article types discussing QALYs with identification or discussion of associated limitations. Articles were limited to those published in English. Articles were excluded if they did not discuss QALY limitations, focused entirely on another metric (e.g. DALY) or were duplicate manuscripts.
A total of 201 articles were identified as meeting the inclusion and exclusion criteria and were subsequently used as the basis for this review.

Discussion
Three common themes emerged when exploring the limitations of QALYs, namely:

Context or Disease Specific Considerations
Within each theme, specific limitations are discussed. Some overlap exists between themes and their corresponding specific limitations; however, this was expected given the nature of the topic. The above three themes were selected to facilitate the demarcation of issues. Additionally, the limitations were extrapolated to examine their potential impact on regenerative medicine and future medical technologies.

Ethical considerations
One of the most fraught elements of QALY implementation is ethical in nature. This review identified five key themes raised by critics in opposition to the QALY system ( Figure 2).
One of the most commonly cited objections concerned valuing one individual's life over another's. Critics argued that a perfect state of health did not necessarily make a life more or less valuable -for example, one cannot assume that someone who is wheel-chair bound cannot live as or more happily than someone who isn't and subsequently be considered less entitled to care [2,3].
Difficulty also surrounds the notion of determining personhood; whereby in order to measure quality of life, life must be present. This presents a difficulty in cases where there is debate about whether life actually exists, such as in foetuses or brain-dead patients [4].
Some critics consider QALYs to serve a limiting function, setting both false limits on healthcare and reducing freedom of choice. Regarding setting false limits on healthcare, some authors noted that QALY's may be used to justify overly restricted healthcare budgets, rather than encouraging improved efficiency or budget increases (10). Similarly, by being more prescriptive in what healthcare options are available, the QALY system reduces freedom of choice and ultimately, autonomous patient decisions [5,6].
Finally, critics highlighted that the quantitative nature of QALYs reflects an overly utilitarian approach that does not always produce optimal outcomes. The phrase "A QALY is a QALY is a QALY" is an often-cited statement that emphasises the fact that all QALYs are considered equal regardless of the patient or situational factors    [1,4,7,8].

Methodological issues and theoretical assumptions
Methodological issues and theoretical assumptions involve limitations relating to QALY derivation ( Figure 3). Measurement techniques, tools, assumptions, and mathematical operations are included in this analysis.
Several authors have mentioned that QALYs are based on poor measurement techniques. In particular, values attributed to 'wellbeing' were often developed with small, non-representative sample sizes [6,[9][10][11]. Multiple authors have also expressed concern regarding the validity and reliability of measurements focused on the utility value of health status [6,11]. Measuring utility values is a challenging process and different methodologies can produce different results -for example, one study of HIV patients found that the standard gamble scale resulted in higher utility scores compared to others. Furthermore, study subjects often misunderstand utility scales; in one study 3-25% of respondents could not complete or produced nonsensical results when given a standard gamble scale.
Dissimilar populations may also evaluate conditions differently -e.g. utility values derived from a physician vs. general population are likely to differ. This phenomenon has been highlighted in a study examining depression whereby patients and physicians assigned utility values of 0.31 vs. 0.42 respectively [12]. Moreover, utility scores do not account for contextual factors such as severity of initial health state, prevalence of disease, parent or caregiver status, or if a population is marginalized. Numerous articles have also highlighted issues with how adequate measurements are defined, their validity and the ability to reliably replicate them -such measurements can vastly under-estimate the value of interventions for acute conditions by placing an upperbound restriction [1,2,7,8,[12][13][14][15][16][17][18][19][20][21][22][23][24].
The comparison of QALYs is frequently presented through the means of league tables, with policy makers utilising them to compare QALYs for different interventions. Such practice is unsurprisingly fraught with issues, largely due to the fact that QALY calculations are principally dependent on both the population studied and period of study. Theoretically for example, a QALY calculated for angina treatment in 1997 in the UK may well have different results compared to a QALY calculation for the management of angina in Germany in 2005. Diverse populations have different preferences and even specific time periods have differing treatment paradigms. It is therefore extremely difficult to make direct comparisons when such factors are unmatched [13,20,22].
Discrimination has also been found to exist against new and innovative treatments, such as those falling within the remit of regenerative medicine, which are often expensive initially but significantly reduce in cost over time. QALY calculations are based on an assumption of constant cost effectiveness and therefore issue a fixed cost for an intervention that does not cogitate a potential reduction in costs as technology advances and interventions are refined [13,20].
The units of QALYs have also been the focus of some criticism. A QALY is the product of utility and time, and time is a non-zero variable. However, in QALY models, utility is assigned an arbitrary scale from 0 (death) to 1 (perfect health) and this use of 0 prevents arithmetic operations such as division and multiplication. A potential solution to this would be measuring time and utility using similar units of measurement, which would translate into more meaningful and accurate outcomes [25,26].
Looking beyond the remit of health, QALYs have been scrutinised for failing to recognise non-health benefits. Non-health benefits and in particular, societal benefits, such as a faster return to work, improved ability to act as caregiver or better school performance are not factored into QALY calculations despite being of potentially considerable importance [1,27].

Contextual or condition specific considerations
The QALY has limitations in producing reliable and valid measurements across disease categories and does not consider a variety of contextual factors including program-specificity, palliative care, mental health and indeed the future of the medical landscape. As it is currently defined, QALYs do not cover the nuances needed within and across disease categories and patients.
A number of manuscripts have highlighted issues with using defined measurements for decision-making and problems have arisen with attempted trade-offs between quality and quantity of life, especially when the acute condition(s) concerned are transient [12]. If quantity of life is used to value quality of life, the result is affected by an individual's perception of existing health state and value vs. years of life. For certain diseases, such as those on the mental health spectrum, such measurements are incredibly subjective [2,[12][13][14]24,28].
The QALY metric has also been critiqued for having insufficient sensitivity to measure small but clinically meaningful changes in health status -or utility. Such changes in health status are particularly applicable (and important) to certain patient sub-groups, for example cancer patients, where multiple studies have outlined a need for additional dimensions to be considered. Such patients may have short life expectancies or reduced endurance limits so standard 'trade-off'  [33]. What's unique is that these conditions were previously deemed incurable and instead ameliorated by existing pharmaceutical agents. Presently -and principally due to the relative infancy of the regenerative medicine field -few cellular based interventions have been subject to QALY-based analysis [34].
Of those that have, initial results appear encouraging [35] however the question remains as to the suitability of the current QALY calculation methodology for such therapies. Orthopaedic surgery, which has positioned itself at the forefront of cellular based interventions, has demonstrated positive results with autologous chondrocyte implantation (ACI) in acute knee injuries [36]. From its inception in 1994 as a two-stage process utilising a periosteal graft [37], numerous studies have demonstrated significant and sustained improvement in patient-reported symptoms [35]. However, owing to graft complications and the consequent need for arthroscopic revision [38,39] a modified ACI utilizing a type I/III collagen patch was developed and approved for European usage [40]. From a financial perspective, several studies have attempted to evaluate the cost effectiveness of ACI with a periosteal graft (ACI-P) and/or ACI with a collagen patch (ACI-C), as presented in Table 2.
All four studies conducted a cost-utility analysis (CUA) with outcomes reported as QALYs. Only Gerlier et al. [41] and Koerber et al. [42] disclosed their study perspective -from a global healthcare payer and the German statutory health insurance respectively. All four studies also employed a range of outcome assessment modalities spanning generic health-related quality of life measures (e.g. EQ-5D-3L) to disease specific measures including the Cincinnati Knee Rating Scale. Although the utilization of both generic and programme or diseasespecific outcome measures helps address some of the issues associated with QALY calculations (and facilitate inter-study comparative analyses), key aspects e.g. societal benefit, remain unaccounted for and underlying methodologies remain undisclosed -Samuelson et al. [35] for example, did not state how they obtained, used or estimated utility values to calculate QALYs. As evident in Table 2, inconsistencies exist in determining accurate QALY outputs, largely due to the lack of verified long-term data, non-standardized modelling and reliance upon speculative analysis. Such concerns have previously been raised in the literature -a study by Clar et al. [43] concluded that they were unable to reliably calculate costs per QALY due to the absence of data required.
QALY analysis for cell-based therapies currently serves as a key stratagem in cost effectiveness evaluations, forming a basis for interventional approval, treatment discontinuation and reimbursement mechanisms [44,45]. For example, bio-engineered skin substitutes utilised by plastic and reconstructive surgeons have been shown to successfully demonstrate gains in cost per QALY [46] whereas Sipuleucel-T (Provenge), a cell based therapy used by onco-urologists, has been withdrawn from the EU following a cost-effectiveness analysis by the NICE Appraisal Committee. The calculation of between £61204 and £108585 per QALY fell well outside the usual cost-effectiveness threshold and Sipuleucel-T was consequently not recommended for adoption [44].
The main limitation at present appears to be a paucity of available assessments examining QALYs in cell based therapies -largely as a consequence of insufficient follow-up data and accurate ascertainment of potentially enhanced quality of life. Evidence to date suggests that in terms of QALY calculations, even a high initial fiscal investment may equate to a moderate cost per QALY, and with the prospect of decisions may be invalid [1,2,12,22]. This must however be balanced against the use of QALY measurements as decision-making pools applicable to whole economies rather than at just single patient levels.
Several authors have mentioned the need for programme-specific outcome measures to be developed [12,17,28]. These would facilitate comparisons across programmes whilst maintaining the sensitivity of certain programme-specific scales and directly affect resource allocation. Existing indicator inputs for QALY calculations may be deemed inappropriate for certain populations -such as the elderly -because of their insensitivity to improvements in health status. If for example, a long-term care intervention for an elderly patient has minimal effect on life expectancy, by using an insensitive nonprogramme measure of quality of life, a subsequently 'low' score will be achieved and may result in poor decision-making [2,12,13,17,21,28].
In addition to mental health and cancer patient populations, QALYs also appear to have a limited function in several other subgroupsthese include palliative care, disabled and elderly populations. Low scores are often conferred to palliative care interventions due to the current methodological approach undervaluing the benefits acquired by patients receiving palliative care for a short amount of time [17,28]. This is similarly true of elderly patients, who by having a shorter lifespan may forego any improvements in QALYs that accrue over subsequent decades. Such improvements would therefore only be seen to benefit younger [and potentially healthier] individuals. With regards to disabled patient populations, several studies have cited QALY bias against disabled patients, who may not necessarily be able to achieve substantially higher quality of life when directly compared to healthier individuals [2,3,11,20,23,[29][30][31].
A number of authors have focused on paediatric populations and the limited ability of children to appropriately evaluate quality of life. Concerns have been raised as to whether it is best practice to rely on proxy report measures (such as parents) and authors have called for novel approaches to be established in order to overcome such limitations [24]. Some populations unfortunately remain under-represented in the QALY literature -particularly those falling into the rare disease domain. A paucity of quality research focusing on rare diseases in addition to small scale studies and treatment uncertainty have all been cited as reasons for QALYs failing to reflect the needs of those suffering from rare diseases [32].
Various articles have also highlighted the fact that the QALY system could lead to an innate preference for life saving over life enhancing treatments because preventative or basic long-term care measures generally score lower on QALY calculations than more dramatic treatments [2,3,5,15,32]. This places certain interventions at a disadvantage -for example those in mental healthcare, where treatment modalities largely fall into the remit of life enhancing measures [15].
One of the most important considerations highlighted in the literature concerns that of reduced provider input. Critics have remarked that a generic list of QALY's reduces the role and expertise of healthcare providers and ultimately undermines their ability to make judgements based on an individual's need [6].

Condition specific considerations: the case of regenerative medicine
In terms of future interventions, the expanding field of regenerative medicine offers unparalleled opportunities in both medical and surgical specialties -offering a potential cure for certain diseases such as diabetes or cost effectiveness thresholds is particularly complex and debate has previously been publicized on whether the value of a QALY should be dictated by first proposing the worth of a QALY and setting the healthcare budget at or below that value, or alternatively, proposing a healthcare budget and then allowing the cost of a QALY to declare itself following purchasing decisions [51,52]. With the advent of cellular based therapeutics and their comparably high upfront costs, the QALY calculation methodology may need refinement to realise the financial advantages and opportunity costs such interventions may convey [53][54][55][56][57][58][59][60] -particularly considering the degree of uncertainty associated with them. Furthermore, the benefit(s) of incorporating non-traditional outcome measures (e.g. societal benefits) into current or future tools should be further explored. Despite the aforementioned limitations, and, given the lack of robust alternatives, the QALY is still regarded as the most rigorous methodological tool available and provides a robust framework to guide healthcare providers. However, further research is required to at least enhance QALY harmonisation and cross-comparability amongst health populations and their respective management, or indeed focus on designing an entirely new metric that would incorporate the criticisms explored above. reinstating a patient's health without further intervention, the potential financial savings are likely to translate into long-term cost effectiveness [38][39][40][41][42]47]. Refining QALY calculation methodology will help reduce bias -especially where chronically ill patient subgroups are penalized with a high cost per QALY (secondary to limited life expectancy) despite being able to achieve significant financial savings when newer cellular based interventions are compared to standard treatment practices. It is therefore vital that contextual and social limitations of QALY evaluations are reduced to enable economically rational interpretation of results and appropriate adoption of therapeutics that serve a high unmet need [33,38].
As we embark on a paradigm shift away from conventional medicine and it's intuitive "one-size-fits-all" approach towards precision (or personalized) medicine, which utilizes an array of patient metrics including gene expression, metabolomics and predictive biomarkers, there will be an associated decrease in the level of variation in response to treatment [48]. This targeted approach will help mitigate the prescription of futile treatments -in the US for example; approximately ten of the highest grossing drugs are only effective in between 4% and 25% of the patients who take them [49]. Understandably, initial fiscal outgoings may be higher in comparison to conventional therapeutic modalities, however it should ultimately translate into improved longterm cost-effectiveness.

Conclusion
Debate continues to exist on whether QALYs should serve as the central means of health economics analysis. This review examines the potential shortfalls of QALYs, spanning current ethical, methodological, and contextual domains in addition to examining their suitability for regenerative medicine and future technologies. In the UK, NICE currently stipulates a threshold of £20 000 -£30 000 per QALY [50] when evaluating new therapeutics and/or technologies for NHS adoption, and has used this tool to apply a rational and transparent process to technological adoption for over ten years. Calculating QALY

Conflicts of Interest
This article represents the authors' individual opinions and may not necessarily represent the viewpoints of their employers. Brindley is a stockholder in Translation Ventures Ltd. (Charlbury, Oxfordshire, UK) and IP Asset Ventures Ltd. (Oxford, Oxfordshire, UK), companies that among other services provide cell therapy biomanufacturing, regulatory, and financial advice to pharmaceutical clients. Smith is a consultant with IP Asset Ventures Ltd. Brindley also is subject to the CFA Institute's codes, standards, and guidelines, so he must stress that this piece is provided for academic interest only and must not be construed in any way as an investment recommendation. Additionally, at time of publication, Brindley and the organizations with which he is affiliated may or may not have agreed and/or pending funding commitments from the organizations named herein.