Ambiguous advice, questionable quality: How a lack of development & evaluation standards is preventing the potential of health-apps

If you can think of an ailment, concern, or any aspect of your health and well-being which you wish to monitor or improve, you can be certain that there already exists an app for it. There may be ten, one hundred, or even upwards of a thousand. What is uncertain, however, is whether such apps will work as advertised, and whether first and foremost, they will ‘first do no harm?’

In the wake of unprecedented National Health Service (NHS) efficiency savings, increasing waiting lists, and a looming shortage of trained medical professionals, those concerned with proactively managing their health and wellbeing are increasingly turning to mobile or ‘m-health’, and the use of unregulated and largely un-validated ‘apps.’ This novel therapeutic medium is rapidly gaining momentum, and with an estimated 165,000 health apps available for download as of 2015 [1], the convenience and widespread availability of mobile health present an accessible, affordable, and alluring opportunity to those looking to actively manage their health and wellbeing.

As of 2015, it was estimated that 71% of Britons (45.5million in total) owned a smartphone [2], with 75% using smartphones or tablets to search for health information online [3], and over 90% revealing they would use mobile-health services to engage with healthcare professionals, were such services available to them [2]. By their very nature, apps provide the possibility of proactive health management from the comfort of your own home.

Furthermore, while not confined by the constraints of appointment times and waiting lists, the use of health apps may enable the treatment of thousands of individuals simultaneously [4], with the use of a health app by one individual having no impact on the time and resources available to treat others.
The widely accessible nature of health apps also presents a flexible and pragmatic opportunity to empower patients, improve access, and extend the effective reach of healthcare services, especially for those who are not currently able to engage and fully benefit from existing services. This may include the teacher who is too anxious or stigmatized to discuss alcohol dependency face-to-face, the depressed armed forces serviceman for whom a desire for anonymity is paramount [5], or the knowingly overweight and pre-diabetic single mother of three, who struggles to schedule a GP appointment around her childcare and work commitments.

Yet despite the significant potential for health apps to enhance the efficient and timely delivery of healthcare, there are currently numerous drawbacks when contemplating the use of these unregulated and largely un-validated technologies. Due to the rigorous processes involved in the evaluation of both safety and effectiveness, substandard pharmaceuticals and medical devices rarely make it to market, however the same cannot currently be said for health apps. The reality is that there exists a considerable gap between the potential benefits that apps could provide in theory, and what they are currently likely to deliver in practice. Recent reviews in the therapeutic areas of bulimia [6], asthma [7], PTSD [8], and even insulin dosing [9], and suicide prevention [10], have yielded disturbing conclusions regarding the quality, scientific basis, and often blatant disregard for safety [11], that characterize a great number of health-apps available to consumers.

Even for NHS-accredited apps, recent research from the University of Liverpool demonstrated that just 15% of apps endorsed and recommended to patients for the treatment of depression and anxiety, could provide evidence to corroborate claims of effectiveness made in the app store [4]. This naturally leads to the question of what, if anything, is the use of such apps likely to accomplish, and whether this novel therapeutic medium is viewed as a viable, effective, and ultimately safe option, for those in need of high-quality healthcare services.

Of course, this paucity of evidence is not a new phenomenon concerning electronic medical technologies, with the medical device industry historically suffering a similar shortage of evidence.[12] This is because, unlike pharmaceuticals, regulators are often evaluating medical devices and apps, at a very early stage of their market life cycle, and as such, the extent of product exposure, data collection, and research is typically very sparse, especially so if considering any longer-term outcomes and the sustainability of treatment effects.

Consequently, before blaming app developers for the prevailing lack of evidence within the ever-expanding mHealth arena, it is important to appreciate the barriers to evidence generation currently faced by developers. While in a first-best situation, the burden of proof concerning app safety, clinical, and cost-effectiveness ‘should’ ultimately lie with app developers themselves; the fact that ‘acceptable evidence’ itself is largely open to interpretation, means that it may be folly to expect this paucity of real-world effectiveness research to improve.[13]

This is because unlike established therapeutic options, including pharmaceuticals, there exists no clear guidance to app developers regarding what is expected, and what evidence should be collected, to demonstrate the efficacy and value of health apps. While some international ‘best-practice’ frameworks provide rudimentary recommendations, including the importance of both user feedback and user testing (PAS 277-2015 [6.2.3, 6.2.4, 6.5 and 6.7]) [14], formal guidance is practically non-existent. Considering that 30% of app-development companies have just a single employee, with 64% operating with less than nine employees [15] most app-development companies can be categorized as small and medium-sized enterprises. App developers, much like other businesses, are faced with trade-offs and decisions regarding how best to allocate their limited resources; and it is likely that under the current situation, characterized by a lack of evaluative guidance to ensure that any evidence collected carries any value, the risks of committing limited funding to evidence-generation efforts, currently likely outweigh the perceived benefits of doing so.

To ameliorate this expanding divide between the number of app users, and the number of evidence-based health apps, large-scale evaluative frameworks are likely required. An estimated 50% of health apps will receive fewer than 500 downloads across their entire product life cycle,[16] as such, if left to market forces, the rate of app uptake (and subsequent collection of data for evaluation) is likely to be prohibitively slow. This thereby limits the ability of developers, to detect meaningful treatment effects at conventional levels of statistical significance, further dis-incentivizing developers to attempt to collect evidence in the first place.

It is clear that setting a high standard for health apps from the outset is vital to achieving long-term benefits for both patients and the NHS. Through identifying and actively promoting the most clinically effective, safe, and beneficial apps earlier in an individual’s m-health journey, the many potential benefits of health apps, including reducing treatment costs, extending the effective reach of healthcare, and improving NHS efficiency, are far more likely to be realized. In providing a clear means of navigating the seemingly never-ending pages of results when searching for a specific app, the ‘opportunity costs’ of not gaining access to the best app at the earliest possible time can be avoided, and by informing the decisions of those who download such apps, the good can be sifted from the bad and even the dangerous. In doing so, the NHS and its patients can begin to take full advantage of the apps revolution, engaging with this 21st-century solution without the underlying threat of unknowingly inflicting harm on users, resulting in a more flexible, adaptive, and accessible healthcare system, and the maximizing the potential for large scale improvements in population health.

But only by negating the fear that any evidence collected may be of poor quality, and dispelling the ambiguity around what acceptable evidence can and should look like, can we begin to do so. By providing a much clearer path from development to reimbursement, and making it clear what is required, we can re-incentivize developers to engage in evidence-generation efforts. In doing so, we can maximize the likelihood of evidence-based decision-making within the arena of mHealth taking a firm hold, and achieve the many benefits that this novel therapeutic approach can deliver.

Authors

Liz Ashall-Payne – Founder (ORCHA)

Simon Leigh B.Sc M.Sc (HECON) – Principal Consultant & Senior Health Economist

References

[1] IMS institute for health informatics. Medicines Use and Spending in the U.S. – A Review of 2015 and Outlook to 2020. 2015. Available at:https://www.imshealth.com/en/thought-leadership/ims-institute.Accessed 2nd April 2016.

[2] Nuffield trust. Delivering the benefits of digital health care. 2016. 194

[3] Department of Health and UK Trade and Investment. The UK: your partner for digital health 195 solutions. 2015. Available at:https://www.gov.uk/government/publications/the-uk-your-partner-for-digital-health-solutionspartner-for-digital-health-solutions. Accessed 5th April 2016

[4] Leigh, S, Flatt, S. App-based psychological interventions: friend or foe? Evid Based Ment 205 Health; 2015, 18, 4:97-9.

[5] Iversen AC, van Staden L, Hughes JH, et al. The stigma of mental health problems and other barriers to care in the UK Armed Forces. BMC Health Serv Res 2011;11:31

[6] Nicholas J, Larsen ME, Proudfoot J, et al. Mobile apps for bipolar disorder: a systematic review of features and content quality. J Med Internet Res 2015;17:e198

[7] Huckvale, K, Car, M, Morrison, C, Car, J (2012). Apps for asthma self-management: a systematic assessment of content and tools. BMC Med, 10:144.

[8] Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol 2015;6:27882.

[9] Huckvale, K, Adomaviciute, S, Prieto, JT, Leow, MK, Car, J (2015). Smartphone apps for calculating insulin dose: a systematic assessment. BMC Med, 13:106.

[10] Larsen, ME, Nicholas, J, Christensen, H (2016). A Systematic Assessment of Smartphone Tools for Suicide Prevention. PLoS ONE, 11, 4:e0152285.

[11] The Daily Mail: Health warning over blood pressure monitoring apps as doctors warn they are ‘untested, inaccurate and potentially dangerous’. Available at: hhttps://www.dailymail.co.uk/sciencetech/article-2887791/Health-warning-blood-pressure-apps-doctors-warn-untested-inaccurate-potentially-dangerous.html

[12] Campbell B. NICE medical technologies guidance: aims for clinical practice. Perioper Med (Lond) 2013;2:15.

[13] Leigh, S (2016). Comparing applets and oranges: barriers to evidence-based practice for app-based psychological interventions. Evid Based Ment Health, 19, 3:90-2.

[14] PAS 277:2015 Health and wellness apps. Quality criteria across the life cycle. Code of practice

[15] IDC “Worldwide and U.S. Mobile Applications, Storefronts, Developer, and In-App Advertising 2011-

2015 Forecast: Emergence of Postdownload Business Models”.

[16] IMS Institute for Healthcare Informatics. Patient Apps for Improved Healthcare from Novelty to Mainstream. 2013. https://obroncology.com/imshealth/content/IIHI% 20Apps%20report%20231013F_interactive.pdf (accessed 3 Apr 2016).