What, Where, and How to Collect Real-World Data and Generate Real-World Evidence to Support Drug Reimbursement Decision-Making in Asia: A reflection Into the Past and A Way Forward

Background

Real-world data (RWD) in healthcare is data collected during routine delivery of healthcare from several sources such as electronic medical records (EMRs), claims and billing activities, registries, and patient-generated data using digital tools.^1,2 Evidence derived from extracting and analysing these data through, for example, observational studies or pragmatic trials, is known as real-world evidence (RWE).^2,3 RWE is already utilised globally in healthcare decision-making from market authorisation and coverage decisions of medical products to impact evaluation.^4,5 Research in the field continues to evolve and it is expected that new processes and global standards will emerge as more data and information become available on the use of RWE.⁵ RWD can play an instrumental role in Asia while making context-specific reimbursement decisions for several reasons. Asians are often under-represented in pivotal randomised controlled trials (RCTs).⁶ This problem is further compounded by the fact that although 60% of the global population lives in Asia, only 17% of global clinical trials are conducted in the region,^7,8 owing to financial and human capacity barriers, different ethical and regulatory systems, a lack of research environment in certain jurisdictions, and operational challenges.⁹ Thus, RCTs or observational studies conducted outside of Asia may not sufficiently report the expected health outcomes of Asian populations, due to differences in healthcare infrastructure and clinical practices, ethics and judicial systems compared to other regions, as well as differences in sociocultural environment and genetic makeup between Caucasians and Asians.¹⁰

Using RWE to estimate the benefits and risks of therapies in Asian populations can also help to contextualise health economic models to different local settings. In many Asian jurisdictions, such as China, India, Indonesia, Malaysia, the Philippines, Singapore, and Thailand, reimbursement decisions are currently made several years after regulatory approval and market entry, allowing sufficient time to collect RWD. For example, drugs prescribed by physicians before reimbursement decisions are made are paid for either out-of-pocket or through private insurance. In this case, RWD can provide more certainty regarding the effectiveness of technologies in the local setting and can inform the judicious use of new technologies recommended for reimbursement. Other Asian health systems, such as in Japan, Taiwan, and South Korea, make their reimbursement decisions concurrently with or closely after regulatory approval. In such jurisdictions, RWE is often used to re-assess initial funding decisions including the adjustment of price, and hence can help reduce uncertainty in decision-making and outcomes.^11,12

It is important to understand where Asia stands in using RWD to assess and create an environment where they can be effectively translated into RWE to inform decisions. The REAL World Data In ASia for HEalth Technology Assessment in Reimbursement (REALISE) initiative was established in 2019 as a collaboration between global experts, leaders from Asian health technology assessment (HTA) agencies, and academia from the Asia-Pacific region to understand how RWD and RWE currently inform reimbursement decisions, and to recommend ways to collect and use this information in a consistent and scientifically robust manner.The work was packaged into three thematic areas: (1) scenarios under which RWD is appropriate; (2) what types of RWD to collect, where and how to collect them; and (3) how to translate RWD into RWE. Subsequently, a landscape analysis of RWD for reimbursement decisions in Asia¹³ and a non-binding guidance document¹⁴ addressing the three themes were published in 2020. Based on these two documents, we learned that all 11 jurisdictions (Bhutan, China, India, Indonesia, Japan, Malaysia, the Philippines, Singapore, South Korea, Taiwan, and Thailand) collaborating under REALISE accepted using RWE in their HTA dossiers, usually as supplementary evidence. Only Bhutan, China, Malaysia, and the Philippines did not require justification for using RWE in their evaluations.¹³ Meanwhile, only South Korea had some guidance on the minimum standards for collecting and submitting RWD and RWE for HTAs.¹³ While there was consensus that country-specific guidance on the use of RWD and RWE may not be necessary as there are already several guidelines published for Europe and the United States which are generalisable to other settings, we considered that a consolidated document with insights from the Asian context would be beneficial to further the use of RWD and RWE in developing health policy in the region.

In this paper, we present the findings from our work on the second theme of the REALISE project which describe the types of RWD, their sources, and methods of collection in the Asian context. We then highlight issues related to the use of RWD and provide recommendations to address them.

Methods

REALISE was initiated at an in-person meeting during the 8th HTAsiaLink Conference (https://www.htasialink.org/) held in South Korea in April 2019, where the scope of the project was discussed. The meeting complemented a 16-question online needs assessment survey completed by project members representing different Asian jurisdictions. Findings from the first meeting and the results from the survey about the use of RWD/RWE to inform HTAs in Asia have been published, and led to the development of three thematic focus areas, namely: (1) when to use RWD and RWE; (2) what types of RWD to collect, where to collect them, and how to collect them; and (3) how to translate RWD into RWE.¹³ To address these themes systematically, the REALISE project team agreed that there was a need to produce a guidance document for Asia.

The second thematic area (what, where, and how to collect RWD), which is the subject of this article, was discussed at a second in-person meeting held in Singapore in October 2019. We conducted a facilitated, world-café¹⁵ style session comprising three topics (what type of RWD to collect, where, and how to collect them) with 10 (out of 11) REALISE members to consolidate experiences from their different jurisdictions. Data collected during this world-café session can be found in Supplementary file 1.

In addition, hour-long one-on-one interviews were conducted with 8 (out of 11) members to supplement the information collected during the world café session. Members were asked to answer 3 questions: (i) what examples of RWD are being collected in your country?; (ii) what are the sources of those RWD in your country?; and (iii) what methods are being used to translate those RWD into RWE?

Information from the world-café session and one-on-one interviews was summarised, then reviewed and endorsed by all 11 REALISE members before being incorporated into the guidance document. In addition, we conducted a pragmatic search of literature based on the knowledge of REALISE members to identify guidance and country-specific examples on RWD and RWE. Relevant information was incorporated into the REALISE guidance document. We conducted a public consultation on the draft guidance document (for all three themes) through the REALISE website (https://hiper.nus.edu.sg/realise-guidance) in October 2020 before publishing the final guidance (version 1.1) in November 2020.

Results

What Real-World Data to Collect?

The types of RWD used by REALISE members have been classified into four broad categories: (i) disease context – which includes incidence, prevalence, and transition probabilities; (ii) patient population – which includes socioeconomic, demographic, insurance and medical history; (iii) intervention and comparator – which includes dosage and regimens, continuation and discontinuation from treatment, and adherence; and (iv) outcomes – including safety, effectiveness, patient-reported outcomes (PROs), and costs. The RWD data sources identified in each jurisdiction included disease-specific or other registries, claims databases, health surveys, EMRs, and wearables and personal tracking devices. A summary of what type of RWD are used by each jurisdiction (proxied by responses from REALISE members) and which sources are typically used to collect RWD are summarised in Table. The information presented is not intended to be a comprehensive list and only highlights the types of RWD and their sources that have been used by REALISE members in their previous HTAs.

When considering members’ preferences for using certain types of local RWD in their HTAs, PROs emerged as the most common type (4 votes) followed by RWD about intervention and comparator, then outcomes – effectiveness, outcomes – safety, disease context, and adherence which received 3 votes each, followed by patient population and outcomes – cost which received 2 votes each (Figure 1).

Close

Figure 1. Types of Real-World Data That Should Be Context-Specific Ranked in Order. Abbreviation: RWD, real-world data.

Where and How to Collect Real-World Data?

Typical sources of RWD used by REALISE members are reported in Table. Members highlighted that they employed study designs such as observational studies and pragmatic clinical trials to extract and translate RWD into RWE for their local jurisdictions. A list of guidance on study design, optimal data collection, and quality assurance as well as some jurisdiction specific examples of how RWD collected from these sources have translated into policies are presented in the REALISE guidance. In response to public feedback,¹⁶ the draft REALISE guidance document was amended to also summarise the benefits and limitations of each data source identified, and incorporate good practices for collecting RWD and reporting results from observational studies and pragmatic clinical trials.

Despite extensive use of registries, members highlighted several common hurdles in their use. The National Institute of Health (NIH) Centers for Disease Control and Prevention (KCDC) in Korea accept registration for clinical research in their Clinical Research Information Service system (https://cris.nih.go.kr). Publicly funded research, including many disease registries funded by KCDC are available, but most of them are considered investigator-initiated rather than public as researchers view the registries as their own. Public access to disease registries is therefore limited and potentially requires the individual to know the researchers or have contacts in the NIH/KCDC. A similar situation occurs in Japan whose registries have been implemented by medical societies and parties and are hence researcher-owned. Japanese registries are not allowed for HTA use. National registries in Singapore produce standard public reports of aggregated outcomes periodically, but they often have limited use as evidence to inform reimbursement decisions, given the data is not disaggregated and may not be relevant to inform estimates for subgroups or disease types. Several disease and health relevant registries (eg, births and deaths) are hosted by different public health institutions or other ministries which exacerbates the issue of data access and linkages.

Members highlighted retrospective billing and claims databases often bring issues of data quality (missing data, unintentional miscoding, and intentional miscoding or ‘upcoding’), and may not contain all variables of interest (eg, symptoms, health status). Further, there is also often a lack of distinction between cost and charges, which can be an issue if claims data does not represent the economic value of resources used to provide services, and is influenced by monopolies or monopsony, then its utility for costing as part of a cost-effectiveness analysis is limited. Similar issues were reported with EMR data which are often incomplete, inaccurate, and requires cleaning before analysis. Health surveys were thought to be useful and additional source of RWD, especially disease or topic specific information is lacking either for a target sample or for the entire population. However, issues of subjectivity, and recall bias from respondents, and the lack of control over degree of data disaggregation were considered seen as common challenges by members while extracting RWD relevant to reimbursement from health surveys. Despite the promise of new technologies, extracting and using RWD from wearables and personal tracking devices remained a challenge for members due to lack of established accuracy, usability, and robustness. Therefore, use of RWD from such sources for regulatory or reimbursement decisions remains uncommon worldwide. Nonetheless, this was identified as a promising source of RWD and partnerships with the private sector (who are primary data custodians of such data) was considered necessary to future use.

Discussion and Recommendations

Based on our findings when developing the REALISE guidance document, it is evident that jurisdictions in Asia use RWD to produce RWE to inform reimbursement policies in different capacities. Focusing on the types and sources of RWD, depicted in Table, we noted key differences across jurisdictions. First, researchers in each country often accessed several data sources to collect the same type of RWD. For example, Malaysia used five sources to extract RWD on the patient population and we observed this for all other jurisdictions for different types of RWD. This highlights the current gaps and inefficiencies in data sharing across multiple agencies within each jurisdiction. For example, Thailand has three different health insurance schemes which are managed by different agencies. Therefore, certain RWD (even if they are the same type) need to be accessed from different sources. While this may be conducive to managing different schemes, it does point to fragmented data sources within health systems, incomplete data, or lack of standardising and integrating data.¹⁷ However, Thailand recognised this issues and launched the ‘Big Rock 1’ initiative, and is taking steps to link RWD within and outside the health sector.¹⁸ In Singapore, a white paper on responsible data sharing was jointly developed by the Precision Public Health Asia Society and the Saw Swee Hock School of Public Health.¹⁹ The authors advocated for establishing a data sharing framework, which will comprise: (i) a data sharing strategy, (ii) technical and technological capacity, (iii) regulatory and legal capacity, and (iv) an approach to operationalising data sharing. HTA agencies in Asia should contribute to developing data sharing framework as there are clear mutual benefits to be gained.

Second, not all sources of RWD were used uniformly by jurisdictions to inform HTAs. For example, only Taiwan consistently used RWD from wearables and personal tracking devices. Similarly, not all jurisdictions consistently used EMRs to collect RWD. These differences may reflect the varying levels of maturity of each health system, and differences in capacity and incentives to capture RWD, as well as lack of access to different data sources.²⁰ EMRs, if used properly, have the potential to collect most of the RWD we list in this paper, thereby reducing inefficiency in collecting data and eliminating the need to set up new disease specific registries, use claims databases to gather non-cost related data, or conduct ad-hoc surveys.^17,21

The format, completeness, and quality of RWD across different sources can vary significantly, and appropriate curation and validation are needed. In Asia, national standardisation of RWD variables between sources can and should be an important consideration among jurisdictions that are already making plans for country wide EMR systems (eg, Bhutan, China). Beneficiaries of standardising and harmonising data into EMRs include physicians, nurses, and researchers (RWD collectors and generators), health system managers (RWD custodians), researchers (RWE generators), and policy-makers (RWE consumers). Alternatively, stakeholders could assess the costs and benefits of accessing RWD from different sources in their settings and prioritise resources to those data sources. Incentives need to be developed for physicians and other providers, healthcare systems, payers, and patients to become invested stakeholders in the development and use of RWD and RWE.²² These include financial incentives from public and private insurance payers for quicker processing of reimbursement claims if accurate data are captured by the EMR; for hospitals if they submit laboratory and test results efficiently; for reporting on patient outcomes; providing RWD for studies; and for further adoption of payments, based on outcomes.²³ Non-financial incentives include ensuring that RWD meets research needs and has clear value for those who collect them.

Third, jurisdictions tended to vary in the way that they ranked the importance to different types of RWD with patient reported outcomes being the most preferred type of RWD, depicted in Figure 1. This may reflect the value attached to different types of RWD and their sources by researchers and policy-makers in those jurisdictions. These differences highlight that all RWD and their sources which could be used for HTA were not being used. This is not an issue if researchers and HTA agencies collectively decided to prioritise certain RWD depending on the context of the disease. However, it is an issue if they were not used due to structural or capacity barriers mentioned earlier. Furthermore, our initial findings from REALISE¹³ highlighted that (i) not all jurisdictions require justification for using RWD in HTA, and (ii) only South Korea (in 2019) had guidance or standards on using RWD for HTA. These observations are not conducive for using RWD for research.²⁴ Addressing these issues may require explicitly outlining and standardising processes where the use of each type of RWD (depending on disease context), their sources, and methods to translate them into evidence are clearly outlined within the existing HTA decision-making framework. This sentiment is shared by the HTA community²⁵ and the public in their feedback on the REALISE guidance document.¹⁶

Using HTA Process to Systematically Use RWD to Inform Decisions for Health Policy

There are ongoing efforts to create a framework to incorporate RWE into decisions such as the Canadian Real-world Evidence for Value of Cancer Drugs.²⁶ In the United States, the Food and Drug Administration published a preliminary framework for regulatory use of RWE in December 2018, with a focus on label expansion and changes, adding or modifying indications, as well as adding new populations and comparative effectiveness safety or cost information.²⁷ While this framework has been established it has not yet been applied, and efforts are currently focused on demonstration projects with a number of institutions.²⁸ Concurrently, the Institute for Clinical and Economic Review has published a white paper outlining the opportunities, challenges and limitations of RWE that need to be addressed while considering its use to inform insurer coverage decisions.⁴ European HTA organisations and payers also use RWE, however attitudes regarding its value differ by country and decision-making context. Although there are some collaborations in the field of RWE, such as the GetReal Initiative and European Health Data and Evidence Network, regulators have multi-faceted concerns about challenges from the heterogeneity of European healthcare systems and the different populations, resource levels, financing priorities, settings and delivery, and culture. These factors impact both how healthcare data are collected and managed and the content and quality of the data.²⁸ Other fields such as infectious diseases which also rely on RWD have created a framework for its use, for example, the ‘fitness-for-purpose’ models by the COVID-19 Multi-Model Comparison Collaboration.²⁹ In Asia, Singapore and Indonesia are the few jurisdictions that explicitly mention the use of RWD in their HTA decision-making process.³⁰ It will be useful for other jurisdictions to update their HTA process and methods guide to explicitly incorporate recommendations on the use of RWD/RWE in HTA to guide analysts and companies that are preparing evidence submissions about whether they should invest in the collection and analysis of RWD. Building on the existing framework,³¹ we highlight how such a process can be integrated into current HTA decision-making.

The process, methods, and principles of making evidence-informed decisions using HTA in healthcare have been well-established.²⁴ Generally, this process includes 6 steps: (i) topic nomination, (ii) topic prioritisation and selection, (iii) conducting economic evaluation, (iv) evidence appraisal, (v) decision-making and implementation, and (vi) monitoring and evaluating the outcomes of decisions. Different stakeholders participate at different stages of this process including decision-makers, HTA agencies, researchers, clinical experts, patients, and the private sector (Figure 2).

Close

Figure 2. Health Technology Assessment Process and Implications for Real-World Data. Abbreviations: RWD, real-world data; HTA, health technology assessment; RWE, real-world evidence.

The discussion of RWD can begin at steps 1 and 2 (topic nomination, and topic selection), where stakeholders nominate topics for evaluation and HTA agencies prioritise and select topics before conducting an HTA. Topics are usually prioritised and selected based on several criteria including burden and severity of disease, safety and effectiveness of health intervention, variation in practice, economic implications on household expenditure and payer’s budget, and equity, ethical and social considerations, feasibility, etc.³² Research on these criteria relies on the use of RWD (as RCTs may not be appropriate) and therefore, RWE can inform topic selection decisions.

In step 3 (conducting economic evaluation), researchers and HTA agencies can identify: (i) preferences or the need to include RWD, availability, and acceptance of RWD that are relevant to the study. Information related to the disease context, patient population, intervention and comparator, long-term safety, patient reported outcomes, and costs are usually where RWD may be used. While RCTs remain the gold standard, RWD can supplement or be used when RCTs are not available (or RCTs do not adequately represent target population) for key parameters such as long-term safety and effectiveness (real-world effectiveness). The ‘value-of-information analysis’ framework offers one approach to decision-making, for when and what types of data to collect³³; (ii) upon establishing the types of RWD to collect, researchers can engage with HTA agencies, clinical experts, and the private sector to identify different sources of those data. This engagement may allow researchers to access wider sources of RWD. Evidence costs money, therefore, a balance should be struck between the relevance of a new registry (or any other data) in relation to the burden of collection. The expected benefits and improvement from new evidence is weighed against the cost of the information.³⁴ Any consideration of patient medical records as a public good, calls into question the safety and security of individual data. Hence, we recommend that jurisdictions comply with their national guidelines but at the same time, promote active discussion of the tensions between access to RWD and adequate data protection, to arrive at a compromise between these two needs; (iii) HTA agencies can specify and guide researchers on the methods that are appropriate while extracting and translating RWD into RWE.^14,35 While many perceive observational approaches as lacking credibility, researchers can work to overcome these concerns and improve the overall rigor of such studies. The use of checklists for good reporting practices¹⁴ is strongly encouraged, and submission of completed checklists is now required by some journals to validate manuscripts. Another way to increase credibility is by publishing detailed protocols of real-world studies in a public and online repository. Study registration, particularly for RWD studies intended to formally test hypotheses around comparative effectiveness, has been proposed by the Professional Society for Health Economics and Outcomes Research and International Society for Pharmacoepidemiology joint task force to improve transparency and trust in RWE.¹ This may be communicated through consultation or by adding information on their national HTA methods guidelines. Such purposive data collection processes, where stakeholders enter binding agreements, has been shown to overcome challenges associated with using RWD.³⁶

In step 4 (evidence appraisal), HTA agencies may consider giving weight to RWE before making recommendations to decision-makers. This may be informed by the choices made when designing the study in step 3, ie, why was RWD sought in the first place (to supplement the RCT data, absence of data from RCTs, preference to use local and long-term data), what types of RWD were used, what were their sources (wearables versus EMRs, public versus private, etc), and methods used to translate RWD into RWE. Specific guidelines on asserting value to RWE produced using this approach are available.³¹

In step 5 (decision-making and implementation), HTA agencies should inform decision-makers about the uncertainty (exact parameters contributing to such uncertainty are usually communicated to HTA agencies by HTA researchers) attached to their recommendation. Together, researchers, HTA committees, and decision-makers, should make a commitment to monitor those parameters by identifying which RWD and data source they correspond to, for example, utilities or PROs which may be collected from EMRs. Decision-making approaches such as managed entry agreements are already practicing this process of monitoring key parameters and the benefits are evident.³⁷

Finally in step 6 (monitoring & evaluating decisions), HTA agencies may be able to track the impact of and amend their initial decision based on new RWE that emerges from analysis of RWD for parameters (with high uncertainty at the time of the initial decision) identified in step 5. This often-neglected step of monitoring and re-evaluation is becoming an increasingly important area of concern due to issues such as overdiagnosis, overtreatment, and use of low-value care.³⁸ They result in a range of downstream consequences such as increased cost to patients and health systems, potential toxicities to patients, diagnosis induced anxiety or depression, etc.³⁸ This issue was also raised in the public feedback¹⁶ on the REALISE guidance document. However, this presents an opportunity for health systems to ‘reset’ and further enhance efficiency. The case study of ‘observational RWE updated’ cost-effectiveness assessment of prophylactic treatment for hereditary angioedema,^39,40 where the initial decision was updated based on the use of RWD and RWE, presents practical lessons on how steps 5 and 6 may be implemented. Furthermore, the case study of surgery for appendicitis in Thailand during COVID-19 (what type of RWD – hospitalisation due to appendicitis surgery; where were they collected from – national hospital database for patients enrolled under the universal coverage scheme, how were they translated into evidence – by conducting time series analysis of hospitalisation data before and during COVID-19) highlights the overuse of this service for several contextual reasons.⁴¹ Collection and analysis of RWD enables us to identify such inefficiencies resulting from past decisions and offer opportunities to address them. However, implementing step 6 requires consideration around availability of resources and time for collecting RWD and conducting the re-evaluation, and the appropriate timeframe for re-evaluation.⁴⁰ Stakeholder consultation is advised to find consensus on these issues. This process of explicitly bringing RWD and RWE into reimbursement decisions is summarised in Figure 2. This sub-process (of overall HTA process) for using RWD should be governed by the same principles of the overall HTA process which include participation, transparency, reliability (quality), and accountability.²⁴ Similar approaches (albeit not explicitly linking with existing HTA processes) with details on considerations for different stakeholders have been recommended in the literature.³¹ We recommend HTA agencies to include the process of using RWD into HTA (depicted in Figure 2) in their next editions of their national HTA process guidelines.

Outside the realm of reimbursement decisions, broader health systems or service-related research are regular consumers of RWD and generators of RWE. These include pandemics (real-world effectiveness of vaccines,⁴² effectiveness of non-pharmaceutical interventions,⁴³ monitoring and projecting resource needs⁴³ such as intensive care unit beds, ventilators, etc, and adherence to policies⁴³), measuring health equity,⁴⁴ mental health,⁴⁵ access to care,⁴⁶ or even health systems efficiency.⁴⁷ The primary reason for using RWD in these areas is because conducting RCTs is not ethically or methodologically appropriate.¹³ Future studies could explore how RWD are defined in other research contexts (within and outside the health sector), their types, sources, and methods used to translate into RWE. For example, the term “big data” is also RWD but larger in volume and scope, expanding its use to multiple sectors.^48,49 Comparing the processes and methods used by different fields while using RWD to generate RWE may help draw out lessons. Furthermore, using a common vocabulary (RWD and RWE) in all research fields may help promote an understanding of their role and acceptance in decision-making. Strengthening the system, process, and methods of collecting and using RWD to inform policies presents benefits to wider health research communities.

Limitations

REALISE meetings were convened in 2019, therefore, some of the information we present in our results may be outdated, especially on the types and sources of RWD used by REALISE members. The need to resort to digital solutions such as telemedicine during the COVID-19 pandemic and accelerating momentum in the digitisation of healthcare systems may have increased the use of RWD sources such as EMRs and improved data linkages. While we have tried to include as many jurisdictions as possible when developing the REALISE guidance, given the varying level of maturity of the health systems and HTA processes across the region, our findings may not be completely applicable or generalisable to the whole of Asia. This is heightened by the lack of, and difficulties in establishing universally accepted methodological standards or principles for, the design, conduct, and reporting of RWE.⁵⁰ In addition, our search of literature was pragmatic informed by the knowledge of existing guidance, hence, our search may not be comprehensive. Nevertheless this paper and the REALISE guidance document should serve as helpful resources for HTA systems in Asia that are seeking to incorporate RWD and RWE in their evidence evaluations and healthcare decision-making.

Conclusion

RWD continues to play a significant role in informing decision-making for health globally and notably in Asia despite several barriers. The COVID-19 pandemic has accelerated the use of digital technologies and offers a unique opportunity to expand the use of RWD. The REALISE guidance document serves as one of the repositories of knowledge when considering the use of RWE to inform reimbursement decisions in Asia. RWD deserves its recognition in the current HTA decision-making process, and we call on all stakeholders to contribute to realising its potential going forward.

Acknowledgements

Juliet Eames, Gunjeet Kaur and Waranya Rattanavipapong from Health Intervention and Technology Assessment Program, Ministry of Health, Thailand who helped with note-taking during the meeting. National Evidence-based healthcare Collaborating Agency, Korea for supporting the logistics of organizing the first REALISE meeting and Secretariats of HTAsiaLink and National Evidence-based healthcare Collaborating Agency for facilitating communications with HTAsiaLink members.

As well as Yu Ting Chen, Brandon Chua, Hui Lan Chua, Lou Jing, Siobhan Li, and Wang Yi from the Saw Swee Hock School of Public Health, National University of Singapore (NUS), who helped with facilitating, note-taking and timekeeping during the second REALISE meeting.

Ethical issues

This research did not require an ethics approval since the study does not involve human research subjects. Ethics approval was not sought as data was collected from members who had consented to be part of the initiative. In addition, the survey responses were anonymised. The questions and discussions were not harmful to those involved.

Competing interests

All support for the present manuscript came from an unrestricted grant from The International Decision Support Initiative (iDSI, https://www.idsihealth.org) [grant number: OPP1134345]. In addition, the grant has covered the attendance of meetings and travel by the corresponding author. The corresponding author has disclosed all the information related to the current manuscript in the ICMJE Disclosure Form and has no relationships/activities/interests to declare related to the content of the manuscript.

Authors’ contributions

Preparation of first draft, review, and editing of subsequent drafts, approval of final submitted version: SKC and LWL. Conceptualized and designed the study, project management, supervised data collection, and data analysis, critically revised the first draft and approved the final version: WI and HLW. Review and editing of subsequent drafts and Approval of final submitted version: All authors.

Data availability statement

The underlying data is part of the article itself. In addition, this data has been published in the REALISE Guidance document, as well as in the International Journal of Technology Assessment in Health Care (IJTAHC).

Funding

This work was supported by an unrestricted grant from The International Decision Support Initiative (iDSI, https://www.idsihealth.org) [grant number: OPP1134345], a global network of health, policy and economic expertise, working to achieve universal health coverage and the health Sustainable Development Goal (SDG 3), and which supports countries to get the best value for money from health spending. iDSI receives funding support from the Bill & Melinda Gates Foundation, the UK Department for International Development, and the Rockefeller Foundation. Under the grant conditions of the Bill & Melinda Gates Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission. Ryota Nakamura was also supported by the Japan Society for the Promotion of Science KAKENHI (grant number: 18H00862). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The findings, interpretations and conclusions expressed in this article do not necessarily reflect the views of the aforementioned funding agencies. A representative from iDSI participated in the first REASLISE working group meeting as an observer.

Supplementary files

References

1. Orsini LS, Berger M, Crown W. Improving transparency to build trust in real-world secondary data studies for hypothesis testing-why, what, and how: recommendations and a road map from the real-world evidence transparency initiative. Value Health 2020; 23(9):1128-1136. doi: 10.1016/j.jval.2020.04.002 [Crossref] [ Google Scholar]
2. INAHTA, HTAi. HTA Glossary. http://htaglossary.net/HomePage. Accessed March 17, 2020.
3. Berger ML, Sox H, Willke RJ. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf 2017; 26(9):1033-1039. doi: 10.1002/pds.4297 [Crossref] [ Google Scholar]
4. Hampson G, Towse A, Dreitlein B, Henshall C, Pearson S. Real world evidence for coverage decisions: opportunities and challenges. A report from the 2017 ICER membership policy summit. 2018. https://icer-review.org/wp-content/uploads/2018/03/ICER-Real-World-Evidence-White-Paper-03282018.pdf. Accessed December 22, 2019.
5. Murphy G. Use of Real-World Evidence in Single Drug Technology Assessment Processes by Health Technology Assessment and Regulatory Organizations. 2018. https://www.cadth.ca/use-real-world-evidence-single-drug-technology-assessment-processes-health-technology-assessment-and. Accessed December 19, 2019.
6. Downing NS, Shah ND, Neiman JH, Aminawung JA, Krumholz HM, Ross JS. Participation of the elderly, women, and minorities in pivotal trials supporting 2011-2013 U.S. Food and Drug Administration approvals. Trials 2016; 17:199. doi: 10.1186/s13063-016-1322-4 [Crossref] [ Google Scholar]
7. Worldometer. Population by Region. 2021. https://www.worldometers.info/world-population/asia-population/. Accessed February 17, 2020.
8. Fashoyin-Aje LA, Fernandes LL, Lemery S. Asian representation in clinical trials of new drugs for the treatment of cancer. J Clin Oncol 2017; 35(15 Suppl):6564-6564. doi: 10.1200/JCO.2017.35.15_suppl.6564 [Crossref] [ Google Scholar]
9. Alemayehu C, Mitchell G, Nikles J. Barriers for conducting clinical trials in developing countries- a systematic review. Int J Equity Health 2018; 17(1):37. doi: 10.1186/s12939-018-0748-6 [Crossref] [ Google Scholar]
10. Lee E, Ryan S, Birmingham B. Rosuvastatin pharmacokinetics and pharmacogenetics in white and Asian subjects residing in the same environment. Clin PharmacolTher 2005; 78(4):330-341. doi: 10.1016/j.clpt.2005.06.013 [Crossref] [ Google Scholar]
11. Annemans L, Makady A. TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases. Orphanet J Rare Dis 2020; 15(1):127. doi: 10.1186/s13023-020-01370-3 [Crossref] [ Google Scholar]
12. Bloem LT, Vreman RA, Peeters NWL. Associations between uncertainties identified by the European Medicines Agency and national decision making on reimbursement by HTA agencies. Clin Transl Sci 2021; 14(4):1566-1577. doi: 10.1111/cts.13027 [Crossref] [ Google Scholar]
13. Lou J, Kc S, Toh KY. Real-world data for health technology assessment for reimbursement decisions in Asia: current landscape and a way forward. Int J Technol Assess Health Care 2020; 36(5):474-480. doi: 10.1017/s0266462320000628 [Crossref] [ Google Scholar]
14. REALISE. REALISE Guidance Document. 2021. https://hiper.nus.edu.sg/realise-guidance/. Accessed January 19, 2021.
15. World Cafe Method. https://theworldcafe.com/key-concepts-resources/world-cafe-method/. Accessed December 22, 2022.
16. REALISE. Use of Real-World Data and Real-World Evidence to Support Drug Reimbursement Decision-Making in Asia. Public Feedback. 2020. https://hiper.nus.edu.sg/realise-guidance/. Accessed August 8, 2022.
17. Scholte M, van Dulmen SA, Neeleman-Van der Steen CW, van der Wees PJ, Nijhuis-van der Sanden MW, Braspenning J. Data extraction from electronic health records (EHRs) for quality measurement of the physical therapy process: comparison between EHR data and survey data. BMC Med Inform DecisMak 2016; 16(1):141. doi: 10.1186/s12911-016-0382-4 [Crossref] [ Google Scholar]
18. Ministry of Public Health Ministry of Higher Education. 5 Ministry Joins Forces to Reform Big Rock 1 Public Health System, Driving Pandemic Ending Projects with Innovation. November 2021. https://www.mhesi.go.th/index.php/news/5006-5-big-rock-1.html. Accessed November 8, 2022.
19. Precision Public Health Asia Society. Responsible Data Sharing in Health and Healthcare. 2022. https://pphasia.com/publications/papers-reports/whitepaper-responsible-data-sharing-in-health-and-healthcare/?fbclid=IwAR3T41L9xAvm3cWnRAmRj3jkOfjNCpogHciZohHgIfQJoROMV5jnfxgod2Y.
20. Ngusie HS, Shiferaw AM, Bogale AD, Ahmed MH. Health data management practice and associated factors among health professionals working at public health facilities in resource limited settings. Adv Med Educ Pract 2021; 12:855-862. doi: 10.2147/amep.s320769 [Crossref] [ Google Scholar]
21. Nordo AH, Levaux HP, Becnel LB. Use of EHRs data for clinical research: historical progress and current applications. Learn Health Syst 2019; 3(1):e10076. doi: 10.1002/lrh2.10076 [Crossref] [ Google Scholar]
22. Berger M, Daniel G, Frank K, et al. A Framework for Regulatory Use of Real-World Evidence. Duke-Margolis Health Policy Center; 2017. https://healthpolicy.duke.edu/publications/real-world-evidence-white-paper. Accessed December 20, 2019.
23. Khozin S, Blumenthal GM, Pazdur R. Real-world Data for Clinical Evidence Generation in Oncology. J Natl Cancer Inst 2017; 109(11):djx187. doi: 10.1093/jnci/djx187 [Crossref] [ Google Scholar]
24. Wilkinson T, Sculpher MJ, Claxton K. The international decision support initiative reference case for economic evaluation: an aid to thought. Value Health 2016; 19(8):921-928. doi: 10.1016/j.jval.2016.04.015 [Crossref] [ Google Scholar]
25. Oortwijn W, Sampietro-Colom L, Trowman R. How to deal with the inevitable: generating real-world data and using real-world evidence for HTA purposes - from theory to action. Int J Technol Assess Health Care 2019; 35(4):346-350. doi: 10.1017/s0266462319000400 [Crossref] [ Google Scholar]
26. Chan K, Nam S, Evans B. Developing a framework to incorporate real-world evidence in cancer drug funding decisions: the Canadian Real-world Evidence for Value of Cancer Drugs (CanREValue) collaboration. BMJ Open 2020; 10(1):e032884. doi: 10.1136/bmjopen-2019-032884 [Crossref] [ Google Scholar]
27. FDA. Framework for FDA’s Real-World Evidence Program. 2018. https://www.fda.gov/media/120060/download. Accessed March 2, 2021.
28. McClellan M. Regional approaches to the development and use of real world data and evidence. Paper presented at: CoRE Symposium 2019; Duke-NUS Medical School, Amphitheatre Level 2.
29. Teerawattananon Y, Kc S, Chi YL. Recalibrating the notion of modelling for policymaking during pandemics. Epidemics 2022; 38:100552. doi: 10.1016/j.epidem.2022.100552 [Crossref] [ Google Scholar]
30. Regulation of the Minister of Health Indonesia: HTA in the National Health Insurance Program. Chapter 3 (HTA process). 2017. http://hukor.kemkes.go.id/uploads/produk_hukum/PMK_No._51_Th_2017_ttg_Pedoman_Penilaian_Teknologi_Kesehatan_Dalam_Program_JKN_.pdf.
31. Pearson SD, Dreitlein B, Towse A, Hampson G, Henshall C. Understanding the Context, Selecting the Standards: A Framework to Guide the Optimal Development and Use of Real-World Evidence for Coverage and Formulary Decisions. OHE, ICER; 2018.
32. Qiu Y, Thokala P, Dixon S, Marchand R, Xiao Y. Topic selection process in health technology assessment agencies around the world: a systematic review. Int J Technol Assess Health Care 2022; 38(1):e19. doi: 10.1017/s0266462321001690 [Crossref] [ Google Scholar]
33. Minelli C, Baio G. Value of information: a tool to improve research prioritization and reduce waste. PLoS Med 2015; 12(9):e1001882. doi: 10.1371/journal.pmed.1001882 [Crossref] [ Google Scholar]
34. Garrison LP Jr, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health 2007; 10(5):326-335. doi: 10.1111/j.1524-4733.2007.00186.x [Crossref] [ Google Scholar]
35. Jaksa A, Wu J, Jónsson P, Eichler HG, Vititoe S, Gatto NM. Organized structure of real-world evidence best practices: moving from fragmented recommendations to comprehensive guidance. J Comp Eff Res 2021; 10(9):711-731. doi: 10.2217/cer-2020-0228 [Crossref] [ Google Scholar]
36. Karen K. The use of real-world data (RWD) by health technology assessment (HTA) agencies. Eur J Public Health 2021;31(Suppl 3): ckab164.574. 10.1093/eurpub/ckab164.574.
37. Xoxi E, Rumi F, Kanavos P. A proposal for value-based managed entry agreements in an environment of technological change and economic challenge for publicly funded healthcare systems. Front Med Technol 2022; 4:888404. doi: 10.3389/fmedt.2022.888404 [Crossref] [ Google Scholar]
38. Kale MS, Korenstein D. Overdiagnosis in primary care: framing the problem and finding solutions. BMJ 2018; 362:k2820. doi: 10.1136/bmj.k2820 [Crossref] [ Google Scholar]
39. Bloudek L, Jaksa A, McKenna A, et al. Observational Real-World Evidence Update Prophylaxis of Hereditary Angioedema with Takhzyro and C1 Inhibitors: Effectiveness and Value. ICER; 2021. https://icer.org/wp-content/uploads/2020/10/ICER_Aetion_RWE_HAE_Report_082421.pdf.
40. Jaksa A, Bloudek L, Carlson JJ. Key learnings from Institute for Clinical and Economic Review’s real-world evidence reassessment pilot. Int J Technol Assess Health Care 2022; 38(1):e32. doi: 10.1017/s0266462322000162 [Crossref] [ Google Scholar]
41. Sukmanee J, Butchon R, Sarajan MH. Estimating the potential overdiagnosis and overtreatment of acute appendicitis in Thailand using a secondary data analysis of service utilization before, during and after the COVID-19 lockdown policy. PLoS One 2022; 17(11):e0270241. doi: 10.1371/journal.pone.0270241 [Crossref] [ Google Scholar]
42. Teerawattananon Y, Anothaisintawee T, Pheerapanyawaranun C. A systematic review of methodological approaches for evaluating real-world effectiveness of COVID-19 vaccines: advising resource-constrained settings. PLoS One 2022; 17(1):e0261930. doi: 10.1371/journal.pone.0261930 [Crossref] [ Google Scholar]
43. Janke AT, Mei H, Rothenberg C, Becher RD, Lin Z, Venkatesh AK. Analysis of hospital resource availability and COVID-19 mortality across the United States. J Hosp Med 2021; 16(4):211-214. doi: 10.12788/jhm.3539 [Crossref] [ Google Scholar]
44. Asaria M, Griffin S, Cookson R. Distributional cost-effectiveness analysis: a tutorial. Med Decis Making 2016; 36(1):8-19. [ Google Scholar]
45. Herrman H, Patel V, Kieling C. Time for united action on depression: a Lancet-World Psychiatric Association Commission. Lancet 2022; 399(10328):957-1022. doi: 10.1016/s0140-6736(21)02141-3 [Crossref] [ Google Scholar]
46. Ma L, Luo N, Wan T, Hu C, Peng M. An improved healthcare accessibility measure considering the temporal dimension and population demand of different ages. Int J Environ Res Public Health 2018; 15(11):2421. doi: 10.3390/ijerph15112421 [Crossref] [ Google Scholar]
47. Ahmed S, Hasan MZ, MacLennan M. Measuring the efficiency of health systems in Asia: a data envelopment analysis. BMJ Open 2019; 9(3):e022155. doi: 10.1136/bmjopen-2018-022155 [Crossref] [ Google Scholar]
48. Sagiroglu S, Sinanc D. Big data: a review. In: 2013 International Conference on Collaboration Technologies and Systems (CTS). IEEE; 2013.
49. Jin X, Wah BW, Cheng X, Wang Y. Significance and challenges of big data research. Big Data Res 2015; 2(2):59-64. doi: 10.1016/j.bdr.2015.01.006 [Crossref] [ Google Scholar]
50. Morton SC, Costlow MR, Graff JS, Dubois RW. Standards and guidelines for observational studies: quality is in the eye of the beholder. J Clin Epidemiol 2016; 71:3-10. doi: 10.1016/j.jclinepi.2015.10.014 [Crossref] [ Google Scholar]