Evidence-Informed Deliberative Processes for Legitimate Health Benefit Package Design − Part I: Conceptual Framework

Background

Countries around the world are increasingly rethinking the design of their health benefit packages to support the progressive realisation of universal health coverage (UHC), ie, to select the appropriate set of services at fair levels of coverage and financial protection.^1-3 Many countries have established health technology assessment (HTA) bodies to support governments in these choices.⁴ HTA assesses the value of a health technology and can inform decisions at different levels, eg, reimbursement decisions on a single health technology or regarding larger parts of the benefit package.

Decision-making on health technologies is an intrinsically complex and value-laden political process that takes place in an environment of diverging social values and interests.^5-9 However, value frameworks currently employed by HTA bodies do not sufficiently account for this complex reality. They are typically based on the use of ‘substantive’ criteria, which are believed to reflect the most important social values. This has led HTA bodies to use, for example, ‘clinical benefit,’ ‘safety’ and ‘cost-effectiveness’ as important decision criteria.⁴ These frameworks are ill fitted for considering the broad diversity of stakeholder values and may lead to insufficient sets of relevant information and sub-optimal decisions.^5-7,10,11

Several years ago, in response to these shortcomings, evidence-informed deliberative processes (EDPs) were developed.¹² An EDP is a practical and stepwise approach for HTA bodies to enhance legitimate health benefit package design based on deliberation between stakeholders to identify, reflect and learn about the meaning and importance of values, and to interpret available evidence on these values. Although the use of EDPs is relatively new, deliberative methods have been developed and used to some extent in the field of HTA since the 2000s.¹³ This paper reports on the further development of EDPs with an emphasis on the key concept of stakeholder participation with deliberation. This is based on academic knowledge exchange,^14-30 analysing practices of eight HTA bodies,³¹ surveying a total of 27 HTA bodies and 66 experts around the globe,^21,32 and experience with country level implementation of EDPs. With regard to the latter, EDPs are presently employed by national health authorities in Ghana, Iran, Moldova, Pakistan and Ukraine for revision of their health benefit packages,³³ and its principles were previously applied for similar use in Kazakhstan, Thailand,³⁴ the Netherlands³⁵ and Indonesia.²³

A companion paper reports on the development of a practical guide to support HTA bodies with the implementation of EDPs, including best practice examples from countries around the world – released at the HTAi meeting in June 2021.³¹ Both papers target researchers, analysts and policy-makers involved in benefit package design at Ministries of Health and/or related HTA bodies, and can together be read as a ‘How to’ guide for organising legitimate decision-making processes.

The paper starts with describing the conceptual framework of EDPs, including its elements and steps. It then focuses on the core element of EDPs, ie, stakeholder involvement ideally operationalised through stakeholder participation with deliberation. We conclude with several remarks on the added value of EDPs.

Conceptual Framework of Evidence-Informed Deliberative Processes

Authorities take decisions in health benefit package design on behalf of the population they serve. They should provide stakeholders – specific population groups who bear the consequences of decisions – with well-justified, reasonable reasons to appreciate the decision-making process as fair and confer legitimacy to the process despite favouring a different decision-making outcome.^36,37 If this is not taken seriously by decision-makers, they risk losing their moral authority for making decisions. Subsequently, they may undermine the legitimacy of their own decision-making process.

Stakeholders are often categorised as one of the 7Ps.³⁸ These are:

• Patients, the public and carer: patients are individuals with the lived experience of a disease or disorder who can provide information in the HTA process pertinent to that disease or disorder; public is an umbrella term which, in the context of the HTA process, incorporates all nonpatient, non-commercial, and nonprofessional stakeholders within the health sector; a carer or caregiver is usually recognized as holding the interests of the patient(s) in their care but may have additional needs and interests.³⁹

• Providers/health professionals: can bring in new perspectives, such as expert views on the effectiveness of technologies and the feasibility of alternative implementation options, including organisational aspects.

• Purchasers: can provide information on the feasibility of alternative implementation options, including organisational aspects.

• Payers: can provide generic non-disease specific perspectives and are often the public.

• Policy-makers: can provide information on the feasibility of alternative implementation options.

• Product makers (industry): can provide in-depth knowledge on the intervention under evaluation.

• Principal investigators (academia): can provide technical expertise on a variety of topics.

These stakeholders often have diverging social values and interests that result in different perceptions of what makes health technologies valuable.⁴⁰ In pluralistic societies, stakeholders can be expected to disagree about what values can be used to include or exclude technologies from the package.⁸ For example, in decisions on public funding of expensive cancer drugs, patients may argue that the best treatment should be made available, while other patients may argue that their treatment should not be displaced and taxpayers may reason that it is more important to use public resources efficiently. In comparison to all these other parties, health professionals may want to have access to the latest technological developments in their field. As a result, decisions are bound to be controversial because stakeholders likely disagree about what should be prioritised, who should benefit and who should not.

The accountability for reasonableness (A4R) framework recognises that stakeholders often justifiably disagree about the importance of specific social values in setting priorities and it argues that stakeholders are more likely to accept priorities that are the outcome of fair, legitimate processes.⁴¹ In other words, stakeholders may agree with outcomes of a fair process even though they may have preferred another outcome. The A4R framework identifies four key conditions for organising fair processes: (i) all relevant values should be considered; (ii) transparency must be ensured; (iii) appeal opportunities must be organised; and (iv) these conditions must be regulated. However, there is little practical guidance as to how these conditions should be implemented.

EDPs respond to this and provide a practical stepwise approach for HTA bodies to implement the A4R conditions in their decision-making processes (Figure 1).¹² As such, the use of EDPs, embodying fair processes, can improve the legitimacy of decision-making processes.

Close

Figure 1. Conceptual Framework of Evidence-Informed Deliberative Processes. Abbreviation: EDPs, evidence-informed deliberative processes.

Elements of Evidence-Informed Deliberative Processes

An EDP integrates four elements (Figure 1). First, the core element of EDPs is stakeholder involvement ideally operationalised through stakeholder participation with deliberation. Such stakeholder involvement ensures that all relevant values are considered, and is thereby a practical translation of the A4R relevance condition. By emphasizing the importance of stakeholder participation, EDPs go one step further than Daniels⁴¹ who recommends to consult those who are affected by a decision. We argue that participation is a much more powerful approach to stakeholder involvement than consultation – this is discussed in more detail below. Second, evidence-informed evaluation which allows for the use of scientific evidence and contributions from stakeholders in terms of their experiences and judgments when further evidence is unavailable. This ensures that relevant evidence is considered and is also a practical translation of the A4R relevance condition. Third, transparency whichensures that the deliberative processes, including their objectives, modes of stakeholder involvement, the decision reached and its related argumentation, is explicitly described and made publicly available. Fourth, appeal which ensures that a decision can be challenged and revised if new information or insights become available. These elements are direct operationalisations of the related A4R conditions. As such, EDPs provide the best way to combine evidence, information, perspectives and values, while also allowing these aspects to be identified and openly discussed.

Practical Steps

We distinguish six practical steps of a HTA process based on existing HTA methods and tools, (Figure 2) and provide recommendations on how elements of EDPs can be implemented in each of these steps. The latter is based on observed practices of HTA bodies around the world. We speak of ‘HTA’ when referring to the whole process, while ‘hta’ specifically refers to the evaluation of a single technology. In steps A–C, we offer advice on the installation of an advisory committee, including the organisation of stakeholder involvement; on how to define decision criteria, and on how to set up a process for identifying and selecting health technologies for hta. In steps D1–D3, we give advice on how to scope, assess and appraise a specific health technology. In steps E–F, we provide guidance on communication and appeal, and monitoring and evaluation respectively. It is well recognised that, while the steps are presented as separate activities and in a linear fashion, in practice there may be iteration of them.⁴² The development and organisation of the EDP practical steps is described in detail in the companion paper.³¹

Close

Figure 2. Six Steps of Implementing Evidence-Informed Deliberative Processes.

Overall, the practical recommendations on the implementation of EDPs in each step of the HTA process should not be considered as a blueprint, but rather as inspirational. Each country has a unique decision-making context determined by factors such as local values and traditions, level of HTA development, and available knowledge, skills and resources. From this follows that each country should introduce its own appropriate form of EDPs in its process, eg, mode of stakeholder involvement and transparency.

Organising Stakeholder Involvement

Stakeholder involvement is formally defined as “an iterative process of actively soliciting the knowledge, experience, judgment and values of individuals selected to represent a broad range of direct interest in a particular issue, for the dual purposes of creating a shared understanding; and making relevant, transparent, and effective decisions.”⁴³ It can improve the legitimacy of decision-making in three ways.¹³ First, stakeholder involvement can serve to identify the full range of interests that a society has in relation to a particular decision. Second, it can improve understanding among stakeholders by explicating the interests of all parties involved. Third, stakeholder involvement can contribute to improving the quality of decisions, as stakeholders contribute specific knowledge (eg, about barriers to implementation or meaningful patient outcomes).

Generally, three approaches to stakeholder involvement are distinguished: participation, consultation and communication,⁴⁴ and we discuss these below. Table provides examples on how HTA bodies can implement these approaches in the different steps of the HTA process.

Stakeholder Participation

Stakeholder participation means that stakeholders are actively engaged in deliberations during each EDP step. A deliberative process is defined as “a series of coordinated activities allowing a group of people (or relevant stakeholders) to receive and exchange information, to critically examine an issue, and to come to an overall group judgement.”⁴⁵ Through this interaction and practical reasoning, stakeholders may deepen their understanding of their own preferences and those of others affected by decisions. They may replace uninformed opinions by views that are more rational and better supported by arguments and evidence, improving the quality of the decisions. There is good evidence that participants learn from deliberative engagement, including considering information that is contrary to their opinions and can change their opinions in line with this new information.⁴⁶ For these reasons, stakeholder participation can best realise the full benefits of stakeholder involvement and contribute to the legitimacy of decision-making. Accordingly, we strongly advise HTA bodies to consider implementing stakeholder participation in their decision-making process.

Stakeholder participation on coverage decisions can be organised through an advisory committee. Advisory committees are typically related to an HTA body and have the mandate to formulate recommendations to decision-making bodies, such as the Ministry of Health. Stakeholder participation in the advisory committee can be organised in two complementary ways.

• HTA bodies may choose to include specific stakeholders as formal members of their advisory committee. Such stakeholders typically represent the general interest of patients and sometimes industry, and not a specific interest regarding certain health technologies. These members have voting power.

• HTA bodies can also organise stakeholder participation by inviting specific stakeholders to participate in their meetings. There stakeholders are not formal members of the advisory committee and are not granted voting power but can participate in deliberations. Such stakeholders typically represent interests or have specific expertise of the health technology being deliberated on. Nevertheless, this form of stakeholder participation does give some degree of influence and ownership of decisions to stakeholders.

There are several approaches to structuring the decision-making process to facilitate stakeholder participation and deliberation in committee meetings. We advise using a systematic approach that ensures all participants and members express their preferences and considerations, such as Nominal Group Technique. Practical guidance on Nominal Group Technique is available elsewhere.⁴⁷ In addition, HTA bodies should take into account several general principles for stakeholder participation and deliberation.⁴⁸ These include:

• Transparency: the deliberative processes – including the objectives, modes of stakeholder involvement, the decision reached and its related argumentation – should be explicitly described and made publicly available.

• Inclusivity: all relevant values pertaining to decisions on a health technology should be taken into account. This requires that relevant stakeholders are meaningfully involved in the decision-making process. This means that barriers to effective participation should be removed.

• Learning:stakeholders are ideally provided opportunities to participate and interact in deliberations as this likely improves the understanding of preferences, arguments and/or evidence, thereby improving the quality of the decision-making process.

• Impartiality: the deliberative process used for each decision and those involved in it should be free from undue influences, both internal (eg, from the agency supporting the HTA process) and external (eg, from stakeholders with vested interests).

The use of these principles in the organisation of stakeholder participation and deliberation is not without challenges. Ensuring transparency may be challenging in settings where there is little or no tradition of open decision-making. Ensuring inclusivity can be resource-intensive for HTA bodies who need to proactively identify affected stakeholders and involve them in decision-making processes and allocate adequately trained staff to the organisation of stakeholder participation. In addition, participating stakeholders may find it difficult to invest the time required to familiarise themselves with procedures of the HTA body. Ensuring learning takes place amongst stakeholders can be challenging for HTA bodies, as this requires multiple perspectives being effectively represented by participating stakeholders and that all perspectives are accounted for in the deliberations. Ensuring impartiality may also be challenging in contexts in institutions with less experience in managing conflicts of interest. Our guide provides practical guidance on how to deal with these challenges.³¹

Stakeholder Consultation

In many countries, meaningful stakeholder participation and actual deliberation between stakeholders is non-existent or in its infancy. Although not ideal, HTA bodies often rely on non-deliberative ways to involve stakeholder perspectives in their HTA processes such as through consultation.

Consultation refers to a structured process for collecting feedback from groups of stakeholders on specific decisions without providing opportunities for meaningful deliberation with the HTA body’s advisory committee. One example is the provision of oral or written patient testimonies to an advisory committee, as organised by the National Health Care Institute in the Netherlands, Canadian Agency for Drugs and Technologies in Health in Canada, the National Institute for Health and Care Excellence in the United Kingdom, and Pharmaceutical Benefits Advisory Committee in Australia.⁴⁹ Other examples include the use of surveys, stakeholder meetings and solicited feedback from stakeholders on HTA draft reports, including proposed recommendations. This latter approach is followed by the National Committee for Health Technology Incorporation in Brazil, and the Institute for Quality and Efficiency in Health Care in Germany among others.⁴⁹

The benefit of consultation is the large number of respondents that can be reached, however, there are some drawbacks. First, the timing and topic for input is predefined, which limits the scope of comments that stakeholders can provide. Second, consultation offers no opportunities for deliberation among stakeholders and the HTA body’s advisory committee, which limits transparency and the facilitation of mutual learning. Third, the quantity of feedback can be significant, so time should be reserved for providing feedback. If the HTA body in question has limited capacity to do so, this can restrict the scope of public consultation and affect its ability to contribute to the overall legitimacy of the process.

Stakeholder Communication

Stakeholder communication refers to efforts by HTA bodies to inform stakeholders of their activities and results by using communication platforms. Communication can be achieved through public meetings, by preparing plain-language versions of reports to increase accessibility, by dissemination to high priority groups using patients’ organisations or by using social media. Communication is characterised by a very low-level of stakeholder involvement. This one-way flow of information means there is little chance of stakeholders influencing the decision-making process. Note that stakeholder communication contributes to the requirement of transparency, the third element of legitimacy as defined in EDPs, and can thus be seen as a necessary activity of HTA bodies.

As an illustration, Box 1 describes the organisation of stakeholder involvement in the development of the UHC benefit package in Pakistan.

Conclusion

There is an increased attention for legitimacy, deliberative processes for HTA and health benefit package design, but the development of theories and methods for such processes remain behind.^10,26,50 This is in stark contrast with eg, the field of economic analysis of health technologies where the theory of welfare economics and methods for cost-effectiveness analysis are well-developed.²⁶ This indicates the importance of this paper and related initiatives such as the ‘Joint HTAi-ISPOR Task Force on Deliberative Processes for HTA.⁵¹

Cuyler observes in his commentary on EDPs that the complexity and multifaceted nature of the topic makes it challenging to develop a “single unifying theory of deliberative processes.”²⁶ He argues that the design of deliberative processes also requires imagination and descriptive evidence. This is indeed reflected in our work: the conceptual framework of EDPs stems from the general concept of legitimacy, the definition of four elements is a practical translation of the A4R framework,⁴¹ the definition of practical steps is based on existing HTA methods and tools, whereas related recommendations on best practices are inferred from observed practices of HTA bodies around the world.

We consider EDPs as complementary to other concepts and methods to support HTA bodies on health benefit package design such as reported in the publication What’s in, what’s out by the Center for Global Development,³ the World Health Organization (WHO) Making Fair Choices report⁵² and its handbook Strategizing national health in the 21st century.⁵³ Specific guidance on the use of HTA for benefit package design is provided in the HTA Toolkit by International Decision Support Initiative,⁵⁴ and the HTA roadmap by Management Sciences for Health.⁵⁵ All of these guides cover critical aspects of health benefit package design, including institutional set-up, required decision-making processes, the necessity of stakeholder involvement, collection of evidence and monitoring and evaluation aspects. The added value of EDPs lies in its explicit foundation in the concept of legitimacy and related elements (including stakeholder involvement ideally operationalised through stakeholder participation with deliberation), as well as in the provision of recommendations on how these elements can be implemented in each of the EDP steps.

The next step is to develop practical guidance for HTA bodies, on the basis of the conceptual framework as presented in this paper. We have made first efforts by analysing practices of eight HTA bodies around the world in terms of the six steps of EDPs and related elements of legitimacy, as described in the companion paper.³¹

Ethical issues

Ethical approval was not necessary as no primary data was used.

Competing interests

Authors declare that they have no competing interests.

Authors’ contributions

All authors have contributed equally.

References

1. Chalkidou K, Glassman A, Marten R. Priority-setting for achieving universal health coverage. Bull World Health Organ 2016; 94(6):462-467. doi: 10.2471/blt.15.155721 [Crossref] [ Google Scholar]
2. Verguet S, Hailu A, Eregata GT, Memirie ST, Johansson KA, Norheim OF. Toward universal health coverage in the post-COVID-19 era. Nat Med 2021; 27(3):380-387. doi: 10.1038/s41591-021-01268-y [Crossref] [ Google Scholar]
3. Glassman A, Giedion U, Smith PC. What’s In, What’s Out: Designing Benefits for Universal Health Coverage. Washington, DC, United States: Brookings Institution Press, Center for Global Development; 2017.
4. World Health Organization (WHO). Global Survey on Health Technology Assessment by National Authorities. WHO; 2015.
5. Holm S. The second phase of priority setting Goodbye to the simple solutions: the second phase of priority setting in health care. BMJ 1998; 317(7164):1000-1002. [ Google Scholar]
6. Mitton C, Donaldson C. Health care priority setting: principles, practice and challenges. Cost Eff Resour Alloc 2004; 2(1):3. doi: 10.1186/1478-7547-2-3 [Crossref] [ Google Scholar]
7. Kapiriri L, Martin DK. A strategy to improve priority setting in developing countries. Health Care Anal 2007; 15(3):159-167. doi: 10.1007/s10728-006-0037-1 [Crossref] [ Google Scholar]
8. Daniels N. Accountability for reasonableness. BMJ 2000; 321(7272):1300-1301. doi: 10.1136/bmj.321.7272.1300 [Crossref] [ Google Scholar]
9. Abelson J, Giacomini M, Lehoux P, Gauvin FP. Bringing ‘the public’ into health technology assessment and coverage policy decisions: from principles to practice. Health Policy 2007; 82(1):37-50. doi: 10.1016/j.healthpol.2006.07.009 [Crossref] [ Google Scholar]
10. Daniels N, Porteny T, Urritia J. Expanded HTA: enhancing fairness and legitimacy. Int J Health Policy Manag 2015; 5(1):1-3. doi: 10.15171/ijhpm.2015.187 [Crossref] [ Google Scholar]
11. Daniels N, van der Wilt GJ. Health technology assessment, deliberative process, and ethically contested issues. Int J Technol Assess Health Care 2016; 32(1-2):10-15. doi: 10.1017/s0266462316000155 [Crossref] [ Google Scholar]
12. Baltussen R, Jansen MP, Bijlmakers L. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health 2017; 20(2):256-260. doi: 10.1016/j.jval.2016.11.019 [Crossref] [ Google Scholar]
13. Baltussen R, Jansen M, Bijlmakers L. Stakeholder participation on the path to universal health coverage: the use of evidence-informed deliberative processes. Trop Med Int Health 2018; 23(10):1071-1074. doi: 10.1111/tmi.13138 [Crossref] [ Google Scholar]
14. Baltussen R, Jansen MP, Bijlmakers L, Tromp N, Yamin AE, Norheim OF. Progressive realisation of universal health coverage: what are the required processes and evidence?. BMJ Glob Health 2017; 2(3):e000342. doi: 10.1136/bmjgh-2017-000342 [Crossref] [ Google Scholar]
15. Baltussen R, Mitton C, Danis M, Williams I, Gold M. Global developments in priority setting in health. Int J Health Policy Manag 2017; 6(3):127-128. doi: 10.15171/ijhpm.2017.10 [Crossref] [ Google Scholar]
16. Jansen MP, Baltussen R, Mikkelsen E. Evidence-informed deliberative processes - early dialogue, broad focus and relevance: a response to recent commentaries. Int J Health Policy Manag 2018; 7(1):96-97. doi: 10.15171/ijhpm.2017.88 [Crossref] [ Google Scholar]
17. Jansen MP, Helderman JK, Boer B, Baltussen R. Fair processes for priority setting: putting theory into practice comment on “expanded HTA: enhancing fairness and legitimacy. ” Int J Health Policy Manag 2017; 6(1):43-47. doi: 10.15171/ijhpm.2016.85 [Crossref] [ Google Scholar]
18. Jansen MP, Baltussen R, Bærøe K. Stakeholder participation for legitimate priority setting: a checklist. Int J Health Policy Manag 2018; 7(11):973-976. doi: 10.15171/ijhpm.2018.57 [Crossref] [ Google Scholar]
19. Jansen MP, Bijlmakers L, Baltussen R, Rouwette EA, Broekhuizen H. A sustainable approach to universal health coverage. Lancet Glob Health 2019; 7(8):e1013. doi: 10.1016/s2214-109x(19)30252-9 [Crossref] [ Google Scholar]
20. Kapiriri L, Baltussen R, Oortwijn W. Implementing evidence-informed deliberative processes in health technology assessment: a low income country perspective. Int J Technol Assess Health Care 2020; 36(1):29-33. doi: 10.1017/s0266462319003398 [Crossref] [ Google Scholar]
21. Oortwijn W, Jansen M, Baltussen R. Use of evidence-informed deliberative processes by health technology assessment agencies around the globe. Int J Health Policy Manag 2020; 9(1):27-33. doi: 10.15171/ijhpm.2019.72 [Crossref] [ Google Scholar]
22. Seixas BV, Mitton C, Danis M, Williams I, Gold M, Baltussen R. Should priority setting also be concerned about profound socio-economic transformations? a response to recent commentary. Int J Health Policy Manag 2017; 6(12):733-734. doi: 10.15171/ijhpm.2017.85 [Crossref] [ Google Scholar]
23. Tromp N, Prawiranegara R, Siregar A. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health 2018; 23(3):279-294. doi: 10.1111/tmi.13031 [Crossref] [ Google Scholar]
24. Chalkidou K, Li R, Culyer AJ, Glassman A, Hofman KJ, Teerawattananon Y. Health technology assessment: global advocacy and local realities comment on “priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. ” Int J Health Policy Manag 2017; 6(4):233-236. doi: 10.15171/ijhpm.2016.118 [Crossref] [ Google Scholar]
25. Lauer JA, Rajan D, Bertram MY. Priority setting for universal health coverage: we need to focus both on substance and on process comment on “priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. ” Int J Health Policy Manag 2017; 6(10):601-603. doi: 10.15171/ijhpm.2017.06 [Crossref] [ Google Scholar]
26. Culyer AJ. Use of evidence-informed deliberative processes - learning by doing comment on “use of evidence-informed deliberative processes by health technology assessment agencies around the globe. ” Int J Health Policy Manag 2020; 9(6):263-265. doi: 10.15171/ijhpm.2019.116 [Crossref] [ Google Scholar]
27. Goetghebeur M, Cellier M. Deliberative processes by health technology assessment agencies: a reflection on legitimacy, values and patient and public involvement comment on “use of evidence-informed deliberative processes by health technology assessment agencies around the globe. ” Int J Health Policy Manag 2021; 10(4):228-231. doi: 10.34172/ijhpm.2020.46 [Crossref] [ Google Scholar]
28. Gopinathan U, Ottersen T. Evidence-informed deliberative processes for universal health coverage: broadening the scope comment on “priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. ” Int J Health Policy Manag 2017; 6(8):473-475. doi: 10.15171/ijhpm.2016.148 [Crossref] [ Google Scholar]
29. Hall W. Don’t discount societal value in cost-effectiveness comment on “priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. ” Int J Health Policy Manag 2017; 6(9):543-545. doi: 10.15171/ijhpm.2017.03 [Crossref] [ Google Scholar]
30. Schlander M. HTA agencies need evidence-informed deliberative processes comment on “use of evidence-informed deliberative processes by health technology assessment agencies around the globe. ” Int J Health Policy Manag 2021; 10(3):158-161. doi: 10.34172/ijhpm.2020.22 [Crossref] [ Google Scholar]
31. Oortwijn W, Jansen M, Baltussen R. Evidence-informed deliberative processes for health benefit package design – part II: A practical guide. Int J Health Policy Manag. 2021. 10.34172/ijhpm.2021.159.
32. Oortwijn W, van Oosterhout S, Kapiriri L. Application of evidence-informed deliberative processes in health technology assessment in low- and middle-income countries. Int J Technol Assess Health Care. 2020:1-5. 10.1017/s0266462320000549.
33. Radboud University Medical center: Global Health Priorities. Country applications of evidence-informed deliberative processes. https://www.radboudumc.nl/global-health-priorities. Accessed March 31, 2021.
34. Youngkong S, Baltussen R, Tantivess S, Mohara A, Teerawattananon Y. Multicriteria decision analysis for including health interventions in the universal health coverage benefit package in Thailand. Value Health 2012; 15(6):961-970. doi: 10.1016/j.jval.2012.06.006 [Crossref] [ Google Scholar]
35. Zorginstituut Nederland. Pakketadvies in de praktijk: wikken en wegen voor een rechtvaardig pakket. Zorginstituut Nederland; 2017.
36. Bærøe K, Baltussen R. Legitimate healthcare limit setting in a real-world setting: integrating accountability for reasonableness and multi-criteria decision analysis. Public Health Ethics 2014; 7(2):98-111. doi: 10.1093/phe/phu006 [Crossref] [ Google Scholar]
37. Peter F. Political Legitimacy. http://plato.stanford.edu/entries/legitimacy/. Accessed March 28, 2021. Published 2010.
38. Concannon TW, Meissner P, Grunbaum JA. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med 2012; 27(8):985-991. doi: 10.1007/s11606-012-2037-1 [Crossref] [ Google Scholar]
39. Street J, Stafinski T, Lopes E, Menon D. Defining the role of the public in Health Technology Assessment (HTA) and HTA-informed decision-making processes. Int J Technol Assess Health Care 2020; 36(2):87-95. doi: 10.1017/s0266462320000094 [Crossref] [ Google Scholar]
40. Grin J, van de Graaf H, Hoppe R. Technology Assessment Through Interaction. A Guide. The Hague, the Netherlands: Rathenau Institute; 1997.
41. Daniels N. Accountability for reasonableness. BMJ 2000; 321(7272):1300-1301. doi: 10.1136/bmj.321.7272.1300 [Crossref] [ Google Scholar]
42. Oortwijn W, Determann D, Schiffers K, Tan SS, van der Tuin J. Towards integrated health technology assessment for improving decision making in selected countries. Value Health 2017; 20(8):1121-1130. doi: 10.1016/j.jval.2017.03.011 [Crossref] [ Google Scholar]
43. Deverka PA, Lavallee DC, Desai PJ. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res 2012; 1(2):181-194. doi: 10.2217/cer.12.7 [Crossref] [ Google Scholar]
44. Abelson J, Wagner F, DeJean D. Public and patient involvement in health technology assessment: a framework for action. Int J Technol Assess Health Care 2016; 32(4):256-264. doi: 10.1017/s0266462316000362 [Crossref] [ Google Scholar]
45. Gauvin FP. Factsheet What is a deliberative process? Publication No. 1193. The National Collaborating Centre for Healthy Public Policy; 2008. http://www.ncchpp.ca/docs/DeliberativeDoc1_EN_pdf.pdf.
46. INVOLVE UK. The impact of deliberation on participants’ opinions. https://www.involve.org.uk/resources/knowledge-base/what-are-affects-deliberation/impact-deliberation-participants-opinions.
47. Dunham, Randall. Nominal Group Technique: A User’s Guide. University of Wisconsin. https://www.sswm.info/sites/default/files/reference_attachments/DUNHAM%201998%20Nominal%20Group%20Technique%20-%20A%20Users%27%20Guide.pdf.
48. Bond K, Stiffell R, Ollendorf DA. Principles for deliberative processes in health technology assessment. Int J Technol Assess Health Care. 2020:1-8. 10.1017/s0266462320000550.
49. Oortwijn W, Jansen M, Baltussen R. Evidence-Informed Deliberative Process: A Practical Guide for HTA Bodies for Legitimate Benefit Package Design. Nijmegen: Radboud University Medical Center; 2021. https://www.radboudumc.nl/global-health-priorities.
50. Baltussen R, Jansen MP, Mikkelsen E. Priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. Int J Health Policy Manag 2016; 5(11):615-618. doi: 10.15171/ijhpm.2016.83 [Crossref] [ Google Scholar]
51. Health Technology Assessment International (HTAi). Joint HTAi-ISPOR Task Force on Deliberative Processes for HTA. https://htai.org/deliberative-processes-for-hta-joint-task-force/.
52. World Health Organization (WHO). WHO Consultative Group on Equity and Universal Health Coverage. Making Fair Choices on the Path to UHC. Geneva: WHO; 2016.
53. Terwindt F, Rajan D, Soucat A. Priority-setting for national health policies, strategies and plans. In: Strategizing National Health in the 21st Century: A Handbook. Geneva: World Health Organization; 2016.
54. International Decision Support Initiative (iDSI). The HTA Toolkit, 2018. http://www.idsihealth.org/HTATOOLKIT/.
55. Castro H, Suharlim C, Kumar R. Moving LMICs Toward Self-Reliance: A Roadmap for Systematic Priority Setting for Resource Allocation. Management Sciences for Health; 2020.