Improving the lives of those living with a mental health condition

Situation

Mental Health is a significant challenge for health services in the Netherlands. Depression has the highest burden of disease of all health conditions (8.2%), including cardiovascular diseases and cancer (1). To address this growing demand, professionals have identified that digital health can be useful in treatment, enhancing and extending services.  But they haven’t used it extensively, as there has been uncertainty around which products are good. Professionals wanted digital health to be assessed before recommending it to patients. 

Yet at the same time, people experiencing mental health conditions have been increasingly using apps. This has typically been approached informally by trial and error or asking peers. But this approach takes time, requires persistence and isn’t informed by evidence. 

So in 2019, the Ministry of Mental Health Services put in place the funding to establish a programme to evaluate health apps, putting in place the tools to enable digital health to be safely part of mental health recovery services. With this funding at the end of 2019 ‘MIND’ – in co-creation with ‘de Nederlandse ggz’ – started the development of www.ggzappwijzer.nl, which was eventually launched in the beginning of 2021. 

Approach

To be guided by evidence MIND reviewed scientific papers to understand if apps really do help. Once it was confirmed that there are good apps that do provide a positive impact, they researched how people find and choose apps. 

As there is very little research into how people choose health apps, the team conducted experiments themselves. They discovered that searches result in hundreds of app suggestions, with no trusted information to indicate which are good.

Focus groups revealed the information people want when choosing an app. They don’t just want a score, they want the facts behind this. Basically they want to know three things:

1. Is it useful for me? There is no perfect one app that meets all needs for all people, so people want to understand if it will help their specific need.  
2. Is it safe? Most people care if their personal information will be secure. 
3. Can I trust it? As health is important, people want to know if an app has been checked for clinical assurance.

Also people are interested in the opinion of others about an app. Do peers and/or professionals give some kind of appreciation or depreciation of an app?

Whilst researching the sector, the team learned about the Organisation for the Review of Care and Health Apps (ORCHA), and the hundreds of assessments it had already conducted on mental health apps. The team felt that the ORCHA assessment was objective and thorough; they liked how the assessment publishes not only a score but also the additional information sought. Further research revealed that every question the MIND team had developed to assess an app, ORCHA had already asked it. In fact ORCHA asked more questions the team couldn’t possibly answer within a reasonable balance between costs and benefits.

ORCHA’s comprehensive base of reviews, together with its work with the NHS, led MIND to work with ORCHA. This collaboration was more appropriate for the programme than trying to reinvent the wheel. 

The programme decided to utilise ORCHA’s objective reviews and layer on top its own qualitative assessment information. For this additional layer, MIND arranged for 4 to 8 people to test an app for two weeks, after which each participant answers a structured questionnaire containing 20 (of quantitative and qualitative) questions. Questions ranged from ‘How easy was this app for you to keep using?’, ‘To what extent would you recommend this app to other people?’ to ‘What improvements do you suggest to improve the app?’. Participants included both professionals and people with mental health conditions. The feedback has been consistent from both groups, each bringing a different but each other reinforcing perspective. The individual testing reports were transformed by the MIND team into one consolidated summary, which functions as a preview of what someone can expect from this app.

Results 

As part of this programme, almost 100 digital health products have been through the ORCHA evaluation and 20 of these have been through the additional MIND structured questionnaire layer. It is hoped this second figure will reach 50 by the end of this year. The evaluated products are hosted on a website that can be easily searched by everybody who is interested in mental health apps. The website is also useful for health care professionals to give advice about using a mental health app.

Since its launch in January 2021, the site has been visited 35,000 times and feedback has been extremely positive. People welcome the trusted information on the apps that are in the library. This has been especially important during lock-down, when an increasing number of apps from un-reputable sources have been launched.

To reach more people, the library of reviews is now also integrated into QULI, one of the major personal health records in use in the Netherlands, as part of the Medmij programme.

The combined ORCHA assessment and MIND reviews have also enabled the team to have informed discussions with developers of digital health. They have provided developers with clear and actionable advice on how their product can be improved.  

Commenting on the programme, Rimmert Brandsma, Project Leader, eHealth, MIND, said: 

“The relationship between MIND, de Nederlandse ggz and ORCHA has also been very successful. There is huge synergy and I feel the end product is better having all partners working actively together. 

“We look forward to the next steps, of making multilingual versions of the website and increasing the number of apps assessed, possibly including AI, VR and wearables into the assessment structure. This will help us to support even more people to find the best digital health.” 

(1) https://www.oecd.org/els/health-systems/MMHC-Country-Press-Note-Netherlands.pdf

Situation

Good Thinking is London’s Digital Mental Wellbeing Service, backed by the London Mayor’s Office, Public Health England, London Councils, NHS and Thrive LDN. It was developed in response to repeated studies that demonstrated that mental health difficulties were common but accessing support was complicated. 

Intensive user testing with over 450 Londoners with mental health charity Mind revealed that people do not always fit into categories for, or want, traditional treatment options, and committing to a course of therapy is not always an option. However the research also revealed that people are willing to use online wellbeing resources about the four most common mental health concerns: anxiety, sleep, stress and low mood. There is also a desire for trusted interventions that are accessible 24/7, private, secure, online and where possible free. And people want to avoid stigma by being able to access services in private and at any time. 

These insights led to a model of service for Good Thinking that is agile, responsive and hugely flexible. The elements of the service include using digital marketing to help reach people, a comprehensive self-assessment, clear urgent support information, support in finding offline support, a quiz to find personalised support and relatable helpful content.  

Alongside these elements, it was also identified the programme should feature NHS-approved, class-leading Apps that offer choice and a range of ways of improving mental health and well-being, that are based on best-evidence, are safe and secure, and are free for users to use. But the team realised that trust in digital health may be an obstacle to progress and that trust and confidence must be earned. 

To address the trust concerns, the products would need to be completely free, without any in-app purchases and they would need to be fully assessed for compliance with the latest regulations for safety.

Solution

To help identify the best in class apps, the team developed a Standard Operating Procedure, to ensure the programme would meet the health priorities, reduce inequalities, and ensure people are offered the best options safely, with an expectation that use will improve health and well-being.

To ensure the programme was best informed, Good Thinking chose to work with the Organisation for the Review of Care and Health Apps (ORCHA). Experts in health app review and accreditation, ORCHA worked closely with Good Thinking to advise on each step of the process.  

It was decided that rather than offer a long list of options, the site would instead provide one best app for each health priority, accompanied by a clear usage explanation. This would enable people to quickly spot the most relevant apps for them.

Good Thinking identified a short list of the best apps for each health priority. ORCHA then identified which standards each app had to meet and conducted the relevant assessments of every app. This enabled Good Thinking to select the final class-leading apps.  

Examples of apps selected that provide support to a distinct user group include: 

• Apart of me – a game to help young people cope with bereavement. 
• Combined minds – a peer support app designed for those looking to help a friend who is struggling or in crisis. 

As standards evolve and apps regularly update, unlike traditional approaches, ORCHA’s unique platform conducts ongoing monitoring, automatically identifying when an app requires a new review and conducting this. This enabled the programme to put in place an important governance step of identifying when an app needs to be decommissioned and removed from the platform. 

Results

Alongside the other support services, the Good Thinking site includes a best in class app for a range of situations for adults and children, all of which when tested, exceed quality thresholds.   

Since its launch in November 2017, Good Thinking has attracted 500,000 users, or one in twelve of the region’s adult population. They’ve visited the service over 650,000 times, 24 hours a day, every day of the week, including national holidays.

Alongside all digital health services, it saw an increase in use during COVID, and especially during the first lockdown period. Since in-person services started to open-up, visitor numbers have dipped, but the team expects to see an increase in use of digital services over the next five years, much as we might expect in the retail sector.

Over the past year, there has been a great deal of work to ensure Good Thinking can complement other care pathways and services, and assist with demand management. For example, the apps that Good Thinking hosts can be prescribed directly by GPs.

Situation

Good Thinking is London’s Digital Mental Wellbeing Service, backed by the London Mayor’s Office, Public Health England, London Councils, NHS and Thrive LDN. It was developed in response to repeated studies that demonstrated that mental health difficulties were common but accessing support was complicated. 

Intensive user testing with over 450 Londoners with mental health charity Mind revealed that people do not always fit into categories for, or want, traditional treatment options, and committing to a course of therapy is not always an option. However the research also revealed that people are willing to use online wellbeing resources about the four most common mental health concerns: anxiety, sleep, stress and low mood. There is also a desire for trusted interventions that are accessible 24/7, private, secure, online and where possible free. And people want to avoid stigma by being able to access services in private and at any time. 

These insights led to a model of service for Good Thinking that is agile, responsive and hugely flexible. The elements of the service include using digital marketing to help reach people, a comprehensive self-assessment, clear urgent support information, support in finding offline support, a quiz to find personalised support and relatable helpful content.  

Alongside these elements, it was also identified the programme should feature NHS-approved, class-leading Apps that offer choice and a range of ways of improving mental health and well-being, that are based on best-evidence, are safe and secure, and are free for users to use. But the team realised that trust in digital health may be an obstacle to progress and that trust and confidence must be earned. 

To address the trust concerns, the products would need to be completely free, without any in-app purchases and they would need to be fully assessed for compliance with the latest regulations for safety.

Solution

To help identify the best in class apps, the team developed a Standard Operating Procedure, to ensure the programme would meet the health priorities, reduce inequalities, and ensure people are offered the best options safely, with an expectation that use will improve health and well-being.

Situation

Good Thinking is London’s Digital Mental Wellbeing Service, backed by the London Mayor’s Office, Public Health England, London Councils, NHS and Thrive LDN. It was developed in response to repeated studies that demonstrated that mental health difficulties were common but accessing support was complicated. 

Intensive user testing with over 450 Londoners with mental health charity Mind revealed that people do not always fit into categories for, or want, traditional treatment options, and committing to a course of therapy is not always an option. However the research also revealed that people are willing to use online wellbeing resources about the four most common mental health concerns: anxiety, sleep, stress and low mood. There is also a desire for trusted interventions that are accessible 24/7, private, secure, online and where possible free. And people want to avoid stigma by being able to access services in private and at any time. 

These insights led to a model of service for Good Thinking that is agile, responsive and hugely flexible. The elements of the service include using digital marketing to help reach people, a comprehensive self-assessment, clear urgent support information, support in finding offline support, a quiz to find personalised support and relatable helpful content.  

Alongside these elements, it was also identified the programme should feature NHS-approved, class-leading Apps that offer choice and a range of ways of improving mental health and well-being, that are based on best-evidence, are safe and secure, and are free for users to use. But the team realised that trust in digital health may be an obstacle to progress and that trust and confidence must be earned. 

To address the trust concerns, the products would need to be completely free, without any in-app purchases and they would need to be fully assessed for compliance with the latest regulations for safety.

Solution

To help identify the best in class apps, the team developed a Standard Operating Procedure, to ensure the programme would meet the health priorities, reduce inequalities, and ensure people are offered the best options safely, with an expectation that use will improve health and well-being.