Case Study
Digital Health Accreditation in the Netherlands
Improving the lives of those living with a mental health condition
Situation
Mental Health is a significant challenge for health services in the Netherlands. Depression has the highest burden of disease of all health conditions (8.2%), including cardiovascular diseases and cancer (1). To address this growing demand, professionals have identified that digital health can be useful in treatment, enhancing, and extending services. But they haven’t used it extensively, as there has been uncertainty around which products are good. Professionals wanted digital health to be assessed before recommending it to patients.
Yet at the same time, people experiencing mental health conditions have been increasingly using apps. This has typically been approached informally by trial and error or by asking peers. But this approach takes time, requires persistence, and isn’t informed by evidence.
So in 2019, the Ministry of Mental Health Services put in place the funding to establish a program to evaluate health apps, putting in place the tools to enable digital health to be safely part of mental health recovery services. With this funding at the end of 2019 ‘MIND’ – in co-creation with ‘de Nederlandse ggz’ – started the development of www.ggzappwijzer.nl, which was eventually launched at the beginning of 2021.
Approach
To be guided by evidence MIND reviewed scientific papers to understand if apps really do help. Once it was confirmed that there are good apps that do provide a positive impact, they researched how people find and choose apps.
As there is very little research into how people choose health apps, the team conducted experiments themselves. They discovered that searches result in hundreds of app suggestions, with no trusted information to indicate which are good.
Focus groups revealed the information people want when choosing an app. They don’t just want a score, they want the facts behind this. Basically, they want to know three things:
- Is it useful for me? There is no perfect one app that meets all needs of all people, so people want to understand if it will help their specific needs.
- Is it safe? Most people care if their personal information will be secure.
- Can I trust it? As health is important, people want to know if an app has been checked for clinical assurance.
Also, people are interested in the opinion of others about an app. Do peers and/or professionals give some kind of appreciation or depreciation of an app?
Whilst researching the sector, the team learned about the Organisation for the Review of Care and Health Apps (ORCHA), and the hundreds of assessments it had already conducted on mental health apps. The team felt that the ORCHA assessment was objective and thorough; they liked how the assessment publishes not only a score but also the additional information sought. Further research revealed that for every question the MIND team had developed to assess an app, ORCHA had already asked it. In fact, ORCHA asked more questions the team couldn’t possibly answer within a reasonable balance between costs and benefits.
ORCHA’s comprehensive base of reviews, together with its work with the NHS, led MIND to work with ORCHA. This collaboration was more appropriate for the program than trying to reinvent the wheel.
The program decided to utilize ORCHA’s objective reviews and layer on top of its own qualitative assessment information. For this additional layer, MIND arranged for 4 to 8 people to test an app for two weeks, after which each participant answers a structured questionnaire containing 20 (quantitative and qualitative) questions. Questions ranged from ‘How easy was this app for you to keep using?’, ‘To what extent would you recommend this app to other people?’ to ‘What improvements do you suggest to improve the app?’. Participants included both professionals and people with mental health conditions. The feedback has been consistent from both groups, each bringing a different but each other reinforcing perspective. The individual testing reports were transformed by the MIND team into one consolidated summary, which functions as a preview of what someone can expect from this app.
Results
As part of this program, almost 100 digital health products have been through the ORCHA evaluation and 20 of these have been through the additional MIND structured questionnaire layer. It is hoped this second figure will reach 50 by the end of this year. The evaluated products are hosted on a website that can be easily searched by everybody interested in mental health apps. The website is also useful for healthcare professionals to advise about using a mental health app.
Since its launch in January 2021, the site has been visited 35,000 times and feedback has been extremely positive. People welcome the trusted information on the apps that are in the library. This has been especially important during lockdown when an increasing number of apps from unreputable sources have been launched.
To reach more people, the library of reviews is now also integrated into QULI, one of the major personal health records in use in the Netherlands, as part of the Medmij program.
The combined ORCHA assessment and MIND reviews have also enabled the team to have informed discussions with developers of digital health. They have provided developers with clear and actionable advice on how their products can be improved.
Commenting on the program, Rimmert Brandsma, Project Leader, eHealth, MIND, said:
“The relationship between MIND, de Nederlandse ggz, and ORCHA has also been very successful. There is huge synergy and I feel the end product is better having all partners working actively together.
“We look forward to the next steps, of making multilingual versions of the website and increasing the number of apps assessed, possibly including AI, VR and wearables into the assessment structure. This will help us to support even more people to find the best digital health.”