Advancing the WHO-INTEGRATE Framework as a Tool for Evidence-Informed, Deliberative Decision-Making Processes: Exploring the Views of Developers and Users of WHO Guidelines

Background

Making evidence-informed decisions about public health and health system interventions and policies is complex.^1-3 On the one hand, producing and assessing evidence eg, on the effectiveness of public health and health policy interventions is challenging due to the complexity of the interventions themselves (eg, the number of components, or the pathway leading to multiple outcomes).⁴ Furthermore, due to interactions with the system in which these are implemented (eg, system changes due to emergent properties, adaptivity, or feedback mechanisms) as well as due to the high context-dependency of the effects of the intervention.^2,5 On the other hand, simply producing more and stronger evidence eg, on the efficacy or cost-effectiveness of an intervention is in itself not sufficient to make better choices, as evidence-informed decision-making is a deeply value-laden process.^6-8 Decision-makers must balance numerous and often conflicting normative and technical aspects for a decision-making process,^9-11 which represents an additional source of complexity (eg, which criteria should be considered and how should these be weighed against each other)? This holds true for all forms of structured decision-making processes in health, notably priority setting,^6,12,13 health technology assessments,¹⁴ and the development of guidelines.^15,16

To support decision-makers in making informed decisions on matters of public health and health policy, the World Health Organization (WHO) provides systematically developed guidelines.¹⁵ The recommendations set forth in these guidelines are particularly important for policy-makers and program managers in low- and middle-income countries who often have limited resources for conducting comprehensive processes of evidence gathering and analysis. Those responsible for developing WHO guidelines are challenged to balance the need for a comprehensive approach – which is indicated due to the complexity of the intervention, the challenges of evidence generation, and the multiplicity of values affected – with the necessity to provide recommendations in a timely manner and often under considerable resource constraints.

In guidelines and beyond, various approaches to integrate a range of specific considerations have been proposed. These approaches – the Accountability for Reasonableness (A4R) framework by Daniels and Sabin^17-21 among others^6,14,22,23 – emphasize the importance of transparency throughout the process, the inclusion of relevant stakeholders, the appropriate composition of the decision-making panel, and the identification and weighting of criteria to be considered, as well as the possibility of revisions and appeals. According to A4R,¹⁷ one key aspect is the condition of relevance: the decision or recommendation must rest on evidence, reasons, and principles that all fair-minded parties can agree are relevant to deciding how to meet the diverse needs of affected stakeholders under the imposed resource constraints.¹⁷ Such processes can increase the acceptability and perceived legitimacy of a decision^19,24,25 even if – given varying and sometimes contradictory interests – no consensus regarding the right selection and weighting of criteria can be achieved.¹⁹

Involving representatives of all relevant stakeholder groups, including community representatives (eg, citizens, patients) in the process of identifying these reasons and principles is considered ideal.^6,14,26,27 However, it is often difficult to meet this ideal due to time and resource constraints; this increases the risk of relevant criteria being overlooked. While not intended to nor able to replace stakeholder participation, Evidence-to-Decision (EtD) frameworks can support decision-makers and guideline developers in this balancing act.^28,29 EtD frameworks tend to comprise sets of criteria as well as procedural guidance. They are intended to ensure that all relevant criteria are considered, the best available evidence is assessed, and the underlying rationale is made explicit and transparent.³⁰ When applied well, these EtD frameworks can help identify and integrate the criteria of relevance for a given decision-making process, even if the voices of all relevant stakeholders were not heard. Their use should, however, not be misinterpreted as a justification for an unbalanced or incomplete composition of the committee preparing or making decisions. Furthermore, structured processes guided by EtD frameworks can lead to better, more rational decisions by counteracting inadequate (cognitive, emotional or social) heuristics, cognitive biases, or in-group dynamics.^31-35 Therefore, EtD frameworks should be as comprehensive as possible, which often is at odds with the constraints, and needs to be balanced against the resources and time available for developing an informed decision.

WHO uses EtD frameworks in their process to develop guidelines.^15,27 Given the reach and potential impact of the recommendations set forth in WHO guidelines, the nature of the guideline development process and the criteria used to inform recommendations set a benchmark for other uses. Both are described in the WHO guideline handbook for guideline development.²⁷ In formulating recommendations WHO Guideline Development Groups (GDG)²⁷ are asked to consider not only evidence of effectiveness but also a range of other criteria (eg, resource implications, acceptability, feasibility²⁷) derived from an early version of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) EtD framework.^27,30,35 Methods for guideline development were originally tailored to clinical interventions, and are still profoundly influenced by this field.^36,37 WHO recently commissioned a series of papers to make guideline development methods more applicable to complex public health and health system challenges.¹⁵ In this context, a new EtD framework, the WHO-INTEGRATE framework version 1.0,¹⁶ was developed with a strong conceptual and normative foundation¹⁶ primarily based on an evaluation of WHO norms and values derived from key WHO documents (eg, the WHO constitution) and widely used public health ethics frameworks. To ensure the relevance of the framework, this normative approach was combined with a literature review of decision criteria used in real world decision-making,³⁸ and an assessment of complexity features.¹⁶

The WHO-INTEGRATE framework comprises six criteria – balance of health benefits and harms; human rights and sociocultural acceptability; health equity, equality and non-discrimination; societal implications; financial and economic considerations; and feasibility and health system considerations – as well as the meta-criterion quality of evidence. Each criterion encompasses a detailed definition, a set of sub-criteria, example questions to assess these sub-criteria and a methodological toolbox with suggested methods for collecting, synthesizing and appraising evidence. Applicability and benefit of the WHO-INTEGRATE framework to real-world decision-making situations remain to be tested.

The objective of this study was to assess the WHO-INTEGRATE framework version 1.0 through a wider participatory process with experts involved in developing WHO guidelines on an international level as well as decision-makers developing national guidelines and/or adapting and implementing WHO guidelines on a national level. Specifically, it served to review the framework in terms of its overall structure and specific criteria and to shed light on the comprehensiveness, relevance, and usefulness in real-world decision-making contexts.

Methods

We conducted nine key informant interviews (KIIs) with experts involved in WHO guideline development on an international level as well as four focus group discussions (FGDs) with health decision-makers in Brazil, Germany, Nepal, and Uganda.

Participants and Data Collection

Key Informant Interviews

The KIIs were conducted with experts who had recently participated in a WHO guideline development process, either as the coordinating WHO staff, GDG chair, or the methodologist supporting the process. In consultation with the Secretariat of the WHO Guideline Review Committee, three WHO guidelines – on sexual and reproductive health and rights of women living with HIV,³⁹ on communicating risk in public health emergencies⁴⁰ and, on antenatal care for a positive pregnancy experience⁴¹ – were selected purposefully, with the aim to cover distinct types of interventions and to capture positive as well as challenging experiences with applying the GRADE EtD framework.²⁷

In the face-to-face or telephone/video interviews carried out between June and October 2017, we used a semi-structured, pre-tested interview guide, developed based on the guiding research questions (Supplementary file 1). The first interview part focused on the experience of using the GRADE EtD framework to formulate recommendations and decide on their strength. In the second part, the interviewees received an interim version of the WHO-INTEGRATE framework and were asked to comment on practical considerations (eg, understandability), the framework content (eg, missing criteria), and the implications of using the framework in WHO guideline development processes.

The audio files of the recorded interviews were transcribed, reviewed, pseudonymized, and then deleted.

Focus Group Discussions

The FGDs were conducted with decision-makers across four countries and continents, as detailed in Table 1 and Supplementary file 2. To capture the diversity of views, we sought to maximize heterogeneity among countries (ie, country income group, region) and topics of discussion (ie, type of intervention/approach).

Local researchers undertook the FGDs in close collaboration with the developers of the framework between August 2017 and October 2018. A topic of current importance was suggested by the local researchers. Decision-makers were identified by the local contact, who reached out to a purposive sample of experts involved with making national recommendations, and potential WHO guideline users, ie, those responsible for adapting and implementing recommendations locally, regionally, or nationally. The recruitment strategy and composition of decision-makers varied across the four FGDs regarding the nature of the committee (providing advice vs. making recommendations), the level of decision-making (national vs. local) and exact composition (representative sub-set of committee vs. ad hoc assembly of participants) (see Table 1 for details).

We developed a preliminary interview guide based on the KII interview guide and adapted it to the setting and topic of the FGDs (Supplementary file 1). The FGDs were set up as a thought experiment: First, the participants conducted a guided brainstorming session on criteria of relevance for the decision-making process on their chosen topic, at which point they were unaware about the content of the WHO-INTEGRATE framework. This was one in order to not have the discussion of criteria and considerations of relevance be “contaminated” or framed by the content of the framework. Second, they were presented with an interim version of the framework and asked to review whether the framework covered the previously discussed criteria, whether aspects were missing, and whether there might be specific suggestions for improvement.

The files of the audio-recorded FGDs were transcribed and the transcripts reviewed by the local researchers. Transcripts were subsequently pseudonymized, translated into English (Brazil and Nepal) and reviewed after translation in the light of the audio-records. Audio-records were then deleted.

Data Analysis

A two-person team (JMS and IS, ST or KK) analyzed the pseudonymized transcripts (KIIs and FGDs) through qualitative content analysis, following the approach by Mayring.⁴² Findings for the FGDs were provided to local contacts for feedback, clarification, and discussion. The analysis of the transcript of the FGD conducted in Germany was not translated but rather analyzed in German by two native speakers (JMS and KK). We followed mixed deductive and inductive approaches to develop the coding frames (Supplementary file 3) using the software MAXQDA12 (VERBI Software GmbH, Berlin). Furthermore, one researcher (JMS) assessed coded passages of the FGD transcripts with respect to whether aspects considered relevant for decision-making on the topic and theme in question were covered by the criteria and sub-criteria of the WHO-INTEGRATE framework.

Results

We conducted nine KIIs with WHO guideline staff (n = 2), GDG chairs (n = 4), and methodologists (n = 3) with a median duration of 62 minutes (range 57-69 minutes). Two additionally intended interviews did not take place, as one participant had retired (and therefore declined) and one interview could not be scheduled despite repeated attempts.

The duration of the four FGDs with between seven and seventeen participants ranged from 95-150 minutes (Table 1, Supplementary file 2). The topics were: tuberculosis guidelines and decentralized actions related to tuberculosis control (Brazil), the (health) effects of an increase of isoglucose in food and potential countermeasures (Germany), health services related to sexual reproductive health and rights of adolescents (Nepal), and the management of untreated wastewater, including sewage from septic tanks and fecal sludge from pit latrines (Uganda).

Overview of Focus Group Discussions

For conciseness, we aim to provide a synthesis rather than detailed account of all four FGDs here. Supplemental data is available by contacting the authors. The next section outlines the first phase of the discussions, as this varied, and summarizes selected main themes.

Focus Group Discussion in Brazil

The FGD in Brazil was concerned with the development of tuberculosis guidelines and manuals within the Brazilian national plan to fight tuberculosis.

During the first phase, participants discussed their experiences with the development of this plan, covering both general challenges in guideline development (eg, alignment with national and supranational strategies) and specific considerations in making recommendations; most often they referred to testing strategies as an example. Criteria of relevance brought up during this phase included affordability, availability, acceptability of healthcare access, adherence to treatments, economic and financial feasibility, cost-effectiveness, and political importance.

A central consensual theme was the need to accommodate the realities of a heterogeneous country (“multiple Brazils”) comprising municipalities with high- versus low-disease burdens, and the related needs to address subsidiarity and empowerment of municipalities to develop locally adapted approaches and to target social determinants within a health in all policies approach.

A second major theme was the criterion of acceptability: participants agreed on the importance of socio-cultural acceptability, especially among those intended to implement the intervention and the intended beneficiaries, stressing that acceptability could vary greatly even on a local level (eg, between healthcare institutions) and across population groups (eg, across different ethnicities or religious groups). Participants thought it unlikely to achieve socio-cultural acceptability overall. Several stated that they were unsure about how to handle the acceptability criterion based on a lack of or false knowledge among those rejecting an intervention (eg, the belief that a vaccination was developed to kill elderly).

An important point of controversy related to the question whether a separate sub-criterion regarding the right to health needed to be added. The argument in favour of adding a sub-criterion was the framework’s focus on health, the argument against doing so was that the right to health is already covered within the broader sub-criterion on human rights.

Focus Group Discussion in Germany

The FGD in Germany focused on food safety and food regulation, notably the expiration of an quota system of the European Union (EU) on the market share of isoglucose, which is expected to increase high-fructose corn syrup (HFCS) in foods and beverages. Due to concerns about adverse health effects of HFCS among the general public and parliamentarians, participants discussed whether the Bavarian Health and Food Safety Authority should issue a recommendation on countermeasures.

During a first phase, participants were presented with a rapid literature review, concluding that there is an ongoing controversy regarding adverse health effects of HFCS. The participants then engaged in lively discussions on whether countermeasures such as labelling, taxation and/or prohibition of products should be taken.

A central theme in the debate was the legal feasibility of such measures and lack of clarity regarding the responsible political level (ie, federal state, nation state, EU): participants discussed whether it would be in line with EU and national regulations and law if the Bavarian government would adopt and implement regulations.

A major point of controversy was the need to balance the expected avoidance of harm, the (lack of) certainty of evidence, and the intrusiveness of the intervention. Some participants argued that in view of inconclusive evidence of harm the government does not have a mandate to act. Others argued with the precautionary principle, which allows enacting regulations to protect public health despite unresolved uncertainties.

A further controversy was on the role of evidence in decision-making: methodological and ethical challenges can be prohibitive in proving harm beyond a reasonable doubt. Conflicts of interest can distort the evidence, both with respect to industry-funded research and evidence generated by researchers with ideological or personal interests (white hat bias⁴³).

Focus Group Discussion in Nepal

The FGD in Nepal focused on adolescent sexual and reproductive health and rights (ASRH). During the first phase, participants voiced their views on important criteria for developing and implementing guidelines focused on ASRH. These included: the capacity of healthcare providers, privacy, and user-friendliness. The discussion then shifted towards experience with adapting international guidelines (eg, from WHO) and implementing ASRH programs. While international guidelines were considered useful for procuring resources from the government, the group agreed that simply transferring global guideline recommendations to local realities can be challenging (eg, due to limited acceptability or resources).

A major topic of discussion was the issue of socio-cultural acceptability. Importantly, guideline recommendations and programs cannot achieve their intended goals if they do not meet the needs and expectations of adolescents. Participants suggested that this could be achieved by engaging adolescents in developing guidelines and thus creating ownership of the program. Acceptability also encompassed community norms and values, eg, regarding gender issues. If these are not considered, a program’s effectiveness and implementation would suffer.

Another central theme was the need to take local realities into account with respect to feasibility considerations. This covers locally available infrastructure as well as financial, technical and human resources. Local coordination within and beyond the health sector was regarded as essential.

Focus Group Discussion in Uganda

The FGD in Uganda focused on the management of untreated wastewater, including sewage from septic tanks and fecal sludge from pit latrines within the larger context of the WHO guidelines on sanitation and health.

During the first phase, participants discussed the importance of managing wastewater, septage and sludge, as well as the reasons for developing (international) guidelines on the topic and for potentially opposing such guidelines. Participants brought up considerations with respect to health implications of the measure, implementation and maintenance costs, and feasibility considerations.

A central theme was the interlinkage between financial costs, resource availability, feasibility, and acceptability of the intervention. Participants agreed that these aspects needed to be reflected both from the perspective of the national as well as the local government and end users. Guideline recommendations might be rejected if they were regarded as too imposing in terms of cost and resource claims or regarded as unachievable under local circumstances.

A related central topic was the need to consider resource requirements broadly in guideline recommendations: These need to reflect the required institutional infrastructure beyond the immediate needs for implementing and maintaining the intervention, such as infrastructure for planning, budgeting, or procuring resources as well as monitoring and evaluation.

Differences and Commonalities Across the FGDs

Health implications of interventions were discussed in all four FGDs, as was the need for multisectoral collaboration, implications for the health system, and consequences beyond the health sector.

While all FGDs emphasized the importance of evidence (regarding effectiveness), the debate in Germany focused on the trustworthiness of the evidence, while the other FGDs emphasized transferability and generalizability from an international level to local realities.

Socio-cultural acceptability of the intervention was discussed across all FGDs but the focus on different stakeholder groups varied: the FGD in Nepal concentrated on intended beneficiaries and the general population, the FGDs in Brazil and Uganda discussed the acceptability among those implementing and those intended to benefit from the intervention, and the German FGD was focused on political acceptability.

Furthermore, the need to consider local perspectives was raised in all FGDs. In Brazil, discussions primarily regarded the different needs and realities of municipalities, in Uganda and Nepal this focus lay on acceptability, resources availability, and feasibility, and in Germany with a concern was legal feasibility (of passing laws on the level of federal states). In this context, participants in the FGDs in Brazil, Nepal and Uganda emphasized procedural considerations in guideline development, highlighting the need to involve affected stakeholders in the process.

The importance of international guidelines and recommendations (eg, from WHO) were discussed in the FGDs in Brazil, Nepal, and Uganda, eg, regarding their usefulness in developing local guidelines or gaining political support. These were not addressed in the German FGD.

In Nepal and Uganda, there was limited controversy: FGD participants seemed to strive for consensus, eg, on regarding missing criteria. In contrast, within the FGD in Germany and, to a lesser extent, in Brazil, discussions were more controversial. To some degree this may be due to cultural norms (eg, regarding deference to authority or conflict tolerance).

General Reception of the WHO-INTEGRATE Framework

The majority of interviewees in the KIIs made positive remarks about the framework, notably its usefulness and comprehensiveness. One participant remarked that the new framework covers many important issues and expressed a clear preference for this framework compared to the one in the WHO guideline handbook.²⁷ Several interviewees explicitly stated that the criteria covered in the framework are important and none could or should be dropped in order to reduce the workload. Explicit positive statements were made regarding the new or expanded criteria Societal implications, Balance of health benefits and harms, Equity, equality and non-discrimination, Human rights and socio-cultural acceptability, and Feasibility and health system considerations and their sub-criteria, for example:

“I think that the framework in its new form with this additional guidance is really informative, and useful, and helpful to participants in these panels and hopefully leads to good recommendations” (KII_Methodologist).

One interviewee felt the WHO-INTEGRATE framework did not go far enough by following the same approach as the GRADE EtD framework, namely starting with a defined intervention, gathering evidence and deciding whether a recommendation should be made; accordingly, a more appropriate approach might be to focus on beneficiaries and ask what should be done to improve health and well-being. Another interviewee remarked that focusing on high quality, quantitative evidence of effectiveness for a clearly defined intervention and outcome may not be feasible for complex interventions.

It was also noted that many aspects in the framework were highly context-dependent and may therefore be less applicable in the development of global guidelines. Furthermore, two interviewees questioned the added value beyond current practice and implied that all newly added sub-criteria could also be addressed as part of the GRADE EtD framework. One interviewee, however, recognized that more explicit sub-criteria could function as “signposts” for less experienced methodologists:

“I like the idea of making it more explicit so that you do think of these things. But if you’re quite a high-level expert, you would automatically do that […]” (KII_Methodologist).

Participants in all four FGDs made positive remarks regarding the framework and its criteria, notably their comprehensiveness. They explicitly mentioned the importance of separating individual and population perspectives regarding health benefits and harms, the range of feasibility considerations, and the broad perspective beyond mere health implications of an intervention. No general critical remarks about the framework were made in the FGDs.

Suggestions Towards Modifying the WHO-INTEGRATE Framework

Table 2 provides an overview of suggestions for improvement derived from the KIIs or FGDs.

Wording and Definitions

Participants in the KIIs and FGDs made several specific suggestions to expand upon and offer more guidance on selected criteria and sub-criteria; notably the criterion Societal implications was described as “fuzzy and vague” along with the sub-criteria Accordance with human rights, Environmental implications and Intrusiveness of the intervention.

“These [criteria] [...] ‘impact on health system,’ ‘social impact,’ they are very vague” (KII_WHO-Staff).

Other suggestions included rewording “impact” to “implications,” distinguishing affordability more clearly from financial considerations, and clarifying the types of stakeholders that should be considered with respect to acceptability.

Missing Aspects

Several interviewees stated explicitly that no criterion seemed to be missing in the framework; others suggested that the framework might not be sufficiently conducive for reflecting on underserved populations and vulnerable groups. They further recommended that a legal expert should assess whether supportive legal environments were sufficiently covered.

FGD discussants noted several potentially missing aspects including intervention sustainability, reliability and quality of an intervention, and outcomes related to well-being, for instance:

“The benefits […] we define as […] professional[s] and [..] that adolescent[s] would define [..] is different: The pleasure of being together with a partner, physical contacts, enjoying beer and cigarette for them is special. I am not sure if [these] benefits [are] considered” (FGD_Nepal).

Furthermore, participants in the FGDs discussed whether political feasibility (eg, political and administrative facilitators and barriers) was sufficiently covered, in particular in regard to political feasibility on the local administrative and political level.

Order and Grouping of Criteria and Sub-criteria

Several interviewees commented on the classification of Patients’/beneficiaries’ values in relation to health outcomes as a sub-criterion. As this is a (main) criterion in the current EtD framework,²⁷ some interviewees were concerned that this aspect may not receive enough attention if only addressed as a sub-criterion.

“I feel like [patients’/beneficiaries’ values in relation to health outcomes] is not really balance of benefits and harms. [...] So I wonder if maybe this can be part of the acceptability and values. Or something like that” (KII_WHO-Staff).

Discussants recommended a separation of human rights and acceptability considerations into two distinct criteria. Also, non-discrimination could be framed as a human rights consideration, rather than an aspect under Equity and equality. Furthermore, they suggested combining societal impact and health impact into one broad impact-oriented criterion.

Overlap, Redundancies and Delineation of Criteria and Sub-criteria

Several interviewees commented on blurred boundaries between criteria and sub-criteria, eg, between the criterion Health equity, equality and non-discrimination and the sub-criterion Social impact, between the sub-criterion Interaction with and impact on the health system and the criterion Financial and economic considerations, as well as between acceptability considerations and the sub-criterion Patients’/beneficiaries’ values in relation to health outcomes.

“I think it is just equity and non-discrimination and societal impact, there are some things that are overlapping. […] How would you really delineate?” (KII_WHO-Staff).

Participants in the FGDs whether the financial and economic as well as the resource considerations were adequately delineated in light of multiple payers on several geographical levels.

Relevance of Criteria and Sub-criteria Based on Focus Group Discussions Only

Depending on the theme and topic of the FGD, different criteria dominated the discussions; nevertheless, references to all six criteria were identified in all four FGDs (Table 3, Supplementary file 6). For example, the FGD in Germany was dominated by the challenge to balance the intrusiveness of interventions and the resultant limitations inflicted on individual liberties with the potential health impacts and the available evidence. However, not all sub-criteria were discussed in every FGD. For example, implications for the (natural) environment were only explicitly discussed in the FGD in Uganda (focused on wastewater management). The most discussed themes included various (health) implications of the intervention, acceptability, accessibility, and autonomy and feasibility considerations. When asked, participants from all four FGD judged the framework to cover their reasoning well.

“I think everything here needs to be kept. You’d rather furnish the decision-makers with more information than they need than less. And as far as I’m concerned, whatever’s in here would be really relevant” (FGD_Uganda).

Implications for Using the WHO-INTEGRATE Framework in the WHO Guideline Development Process Based on Key Informant Interviews Only

Several interviewees were concerned that the complexity and the additional workload associated with the WHO-INTEGRATE framework might be overwhelming for the guideline development process. This may lead to the process merely paying lip-service to criteria, such as Societal implications, and skipping over important domains. Budget constraints and limited time had to be considered when applying the framework.

“I think that the guideline panels will find it [the expanded criteria and sub-criteria] more burdensome because to discuss all of these things will take longer. […] I think the panels get exhausted. They get tired and then they start skipping over, and they skip quite a lot” (KII_Staff).

In contrast, one interviewee stated that “cutting corners” to reduce the workload would diminish the value of the final product. This participant emphasized the need to raise the appropriate resources for a guideline to be “done right” and that using the framework as part of a well-coordinated process would not necessarily lead to a more expensive endeavor.

Several participants stressed the need for additional guidance, including general guidance on how to apply the WHO-INTEGRATE framework and more specific guidance on how to select and interpret criteria and sub-criteria.

Several interviewees further remarked that identifying appropriate evidence eg, for Health system and feasibility considerations, Financial and Economic considerations, or Societal impact might be challenging due to limited availability, low certainty and high context-dependency of evidence.

Discussion

Discussion of Key Findings

In this qualitative study, we received feedback on the WHO-INTEGRATE framework from WHO guideline developers as well as national public health and health policy decision-makers, and identified suggestions for modifications (Table 2, Supplementary file 5). Overall, the framework, its underlying conceptualization and its comprehensive nature, as well as the detailed criteria and sub-criteria were positively received. Some key informants voiced concerns regarding the implications of applying this framework with its very comprehensive set of criteria in everyday guideline development processes. A need for practical guidance was emphasized.

All criteria and sub-criteria of the WHO-INTEGRATE framework were discussed or mentioned as relevant in at least one of the four FGDs (Table 3). Moreover, interviewees and participants in FGDs commented positively on the framework’s comprehensiveness. The developers of the WHO-INTEGRATE framework considered the few criteria highlighted to be missing by KII or FGD participants to be covered, pointing to a need to revise wording and provide clarification (see below). Since the FGDs and KIIs did not identify relevant gaps, this outcome aligns with the A4R framework’s¹⁷ condition of relevance, which depends on decision-makers using the framework properly. Our findings therefore suggest that the criteria included in the WHO-INTEGRATE framework can be considered both comprehensive and relevant for real-world public health decision-making.

These findings are noteworthy in view of the heterogeneity of topics and settings in the FGDs and the diverse WHO guidelines selected for the KIIs (Table 3, Supplementary file 5). Likely, the overview of systematic reviews of real world decision criteria,^16,38 undertaken to develop the WHO-INTEGRATE EtD framework played an important role: within the included systematic reviews,^9,10,44,45 KIIs^9,46,47 and FGDs^9,47-49 were employed and the criteria used or suggested for use in these studies were similar to those discussed or mentioned in our KIIs and FGDs.^16,38

Beyond the aspects explicitly mentioned in KIIs and FGDs, the developers of the WHO-INTEGRATE framework (JMS, ER, IBS) noted some additional domains where a modification of the framework may be warranted. For example, while “individual well-being” was reported as missing in one FGD, the developers considered this aspect covered by the broad WHO concept of health, which was a key building block towards the WHO INTEGRATE framework. Similarly, the broad conceptualization of health systems⁵⁰ (beyond healthcare systems) may not have been clear to all participants and may warrant changes in wording and definitions. Furthermore, in a future revision process the sub-criteria legislation and leadership and governance may need to be more fully described and the criteria relating to availability, accessibility or lack of a suitable alternative may need to be refined with reference to complex public health interventions (eg, labeling interventions).

A concern voiced in the KIIs was the potential additional burden that the use of the WHO-INTEGRATE framework could impose on the guideline development process. This concern was not expressed in the FGDs. These different perceptions are also reflected in the literature: On the one hand, not adequately considering relevant criteria and the views of public health and health policy decision-makers as end-users was found to be a barrier to guideline adherence and implementation.^51,52 On the other hand, the balancing act between rigorous methods and finite resources, notably limited time, repeatedly emerged as an obstacle in structured decision-making processes such as guideline development,^53,54 and the necessity of pragmatic approaches is frequently emphasized.^53-55 As highlighted in the publication of the WHO-INTEGRATE framework version 1.0,¹⁶ one solution to resolving the conflict between the comprehensiveness and granularity of the framework (both well-received) and the implications for guideline development (viewed with some concern) is to insist on a broad approach by considering all six criteria as well as the meta-criterion quality of evidence while allowing for much flexibility in terms of sub-criteria to be considered and evidence to be collected towards these (or not).⁵⁶

While EtD frameworks can support decision-makers in identifying relevant criteria for public health and health policy decisions, this does not supersede the need to address the value-laden nature of these decisions in other parts of the process.¹⁶ The substantive criteria put forward in the WHO-INTEGRATE framework need to be integrated with procedural considerations that address issues of fairness, participation, transparency and the right to appeal,^6,14,17,22 eg, as suggested in what is referred to as evidence-informed deliberative processes.^6,14,57 Major efforts should be made to achieve a balanced composition of the committee preparing for or making decisions, and to ensure representation of affected stakeholders. A balanced committee using an appropriate framework is poised to produce fair and reasonable decisions that are perceived as acceptable and legitimate – a point emphasized by interviewees and participants in the FGDs alike.

Strengths and Limitations

This study followed a comprehensive and rigorous approach to capture the perspectives of those developing WHO guidelines and those potentially adapting and implementing WHO guidelines and/or developing national recommendations across four different countries and continents. Participants represented diverse roles and backgrounds, and two modes of obtaining insights (KIIs and FGDs) were pursued. While the development of other frameworks for structured decision-making^58-62 mostly employed confirmatory approaches (eg, surveys to rate the importance of criteria⁶³), our FGDs pursued a more open-ended approach stimulating participants to reflect deeply on issues of relevance for a given topic and to discuss the framework, its structure and criteria more freely.

While the analysis was led by a researcher of German origin, data collection and analysis were conducted with local researchers to allow for an inter-cultural and interdisciplinary perspective. A thematic area and topic of relevance were proposed by local experts as the basis for the FGD in each country. Despite these strengths, the discussion represented a theoretical decision-making scenario, and criteria other than those discussed might arise in real-world decision-making. Two FGDs (Nepal and Brazil) were translated and it cannot be ruled out that nuances of the discussions were lost in translation, although efforts have been made to preserve the meaning in the process (eg, through contextualization of statements by local researchers and involvement of the local researchers in the analysis). A further limitation is that we did not systematically assess whether participants had financial or non-financial conflicts of interest in relation to the topic in all four FGDs, and analyzed how this may have affected their positioning in the discussion.

Given the multiple dimensions of heterogeneity present in the FGDs (notably in terms of countries and topics), it is difficult to assess whether saturation was reached. Due to the differences in the nature of the topics, the discussions focused on different aspects and criteria (eg, the FGD in Nepal concerned with sexual and reproductive health did not address environmental implications, while this was an important consideration in the FGD in Uganda concerned with the management of untreated wastewater).⁶⁴ Additional FGDs (among others in the WHO Western Pacific region) on an even broader set of topics might provide further considerations for advancing the framework. While we captured the perspective of a relatively small sample of users (four FGDs) and developers (nine KIIs), as discussed above, the WHO-INTEGRATE framework itself ⁶⁵ builds on a much more comprehensive sample of similar real-world events.³⁸

Conclusion

Our study suggests that the WHO-INTEGRATE framework can be a valuable resource for better-informed public health and health systems decisions. Reacting to the suggestions for improvement made by potential end-users, the developers are in the process of developing practical guidance for applying the WHO-INTEGRATE framework. Moreover, the applicability and added value of the framework will need to be tested in real-world guideline development and other decision-making processes, as planned with several upcoming WHO guidelines. The WHO-INTEGRATE framework was explicitly published as a living document. The findings presented here provide a valuable starting point to advance the framework towards a version 2.0.

Acknowledgements

The authors would like to thank all focus group participants and all key informants for their time and engagement. We are very grateful to Susan L. Norris, Anayda Portela, and Rebekah Thomas Bosco as well as other WHO staff for helping with the identification of suitable guidelines and key informants. Moreover, we would like to thank John Ssempebwa for facilitating the FGD in Uganda, and local assistants Monica Lima and Pedro Bossonario involved with supporting the authors AAM and CEMR in conducting the FGDs in Brazil. We are also grateful to the student assistants Sarah Thumbeck and Katja Krug for contributing to data analysis.

Ethical issues

Ethical approval was obtained from the Research Ethics Review Committees (RERC) of the WHO and the LMU Munich, as well as from local RERCs in Brazil, Nepal, and Uganda. All participants provided written and verbal consent prior to participation.

Competing interests

JMS and IBS: Developer of the WHO-INTEGRATE EtD framework. EAR: Developer of the WHO-INTEGRATE EtD framework, member of the GRADE Working Group. JMS reports grants from WHO Department of Maternal, Newborn, Child and Adolescent Health, personal fees from Bavarian Health and Food Safety Authority, other from Norwegian Agency for Development Cooperation (NORAD), during the conduct of the study.

Authors’ contributions

JMS and EAR designed and directed the project with contribution from IBS. JMS developed with interview guide for KIIs with input from EAR and IBS. JMS and EAR recruited the participants with the help from Susan L. Norris, Anayda Portela, Rebekah Thomas Bosco, and other WHO staff. JMS conducted the KIIs. KII-transcripts were analyzed by JMS with contribution from IBS, ST, and KK. DP, JO; KES, CEMdR, AAM, MF suggested topic and recruited participants for the FGDs as well as prepared and facilitated focus group discussions. DP, JO, KES, CEMR, AAM, MF revised and adapted the discussion guide drafted by JMS. JS supported this process in Uganda and ML and PB supported the facilitation of the FGDs in Brazil. JMS, DP, SES, CEMR, and AAM conducted the FGDs. DP, JO, KES, CEMR, AAM transcribed, translated and contextualized the FGD-audio records. JMS analyzed the FGD transcripts with contribution with contribution from IBS, ST, and KK, as well as input from all authors. JMS took the lead in writing the manuscript with input from all authors. All authors provided critical feedback and helped shape the research, analysis and manuscript.

Funding

The WHO-INTEGRATE EtD Framework Version 1.0 was developed with funding provided by the WHO Department of Maternal, Newborn, Child and Adolescent Health through grants received from the United States Agency for International Development and the Norwegian Agency for Development Cooperation. We also gratefully acknowledge that JMS’s position was funded by the Bavarian Health and Food Safety Authority, and that IBS’s input was funded by the Norwegian Agency for Development Cooperation (NORAD). The United States Agency for International Development, the Bavarian Health and Food Safety Authority and NORAD did not have any influence on the research process or content of this manuscript.

Authors’ affiliations

¹Institute for Medical Information Processing, Biometry, and Epidemiology – IBE, LMU Munich, Munich, Germany. ²Pettenkofer School of Public Health, Munich, Germany. ³Save the Children, Kathmandu, Nepal. ⁴The Water Institute, Department of Environmental Sciences and Engineering, The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. ⁵National Supplementary Health Agency, Ministry of Health, Brasília, Brazil. ⁶Department of Maternal-Infant Nursing and Public Health, College of Nursing, University of São Paulo, Ribeirão Preto, Brazil. ⁷Department of Disease Control and Environmental Health, Makerere University School of Public Health, Kampala, Uganda. ⁸Department of Global Health, Norwegian Institute of Public Health, Oslo, Norway. ⁹Bavarian Health and Food Safety Authority, Munich, Germany.

Supplementary files

References

1. Sanderson I. Intelligent policy making for a complex world: pragmatism, evidence and learning. Polit Stud 2009; 57(4):699-719. doi: 10.1111/j.1467-9248.2009.00791.x [Crossref] [ Google Scholar]
2. Petticrew M, Knai C, Thomas J. Implications of a complexity perspective for systematic reviews and guideline development in health decision making. BMJ Glob Health 2019; 4(Suppl 1):e000899. doi: 10.1136/bmjgh-2018-000899 [Crossref] [ Google Scholar]
3. Rutter H, Savona N, Glonti K. The need for a complex systems model of evidence for public health. Lancet 2017; 390(10112):2602-2604. doi: 10.1016/s0140-6736(17)31267-9 [Crossref] [ Google Scholar]
4. Lewin S, Hendry M, Chandler J. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Med Res Methodol 2017; 17(1):76. doi: 10.1186/s12874-017-0349-x [Crossref] [ Google Scholar]
5. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Community Psychol 2009; 43(3-4):267-276. doi: 10.1007/s10464-009-9229-9 [Crossref] [ Google Scholar]
6. Baltussen R, Jansen MP, Mikkelsen E. Priority setting for universal health coverage: we need evidence-informed deliberative processes, not just more evidence on cost-effectiveness. Int J Health Policy Manag 2016; 5(11):615-618. doi: 10.15171/ijhpm.2016.83 [Crossref] [ Google Scholar]
7. Baltussen R, Mikkelsen E, Tromp N. Balancing efficiency, equity and feasibility of HIV treatment in South Africa - development of programmatic guidance. Cost Eff Resour Alloc 2013; 11(1):26. doi: 10.1186/1478-7547-11-26 [Crossref] [ Google Scholar]
8. Kapiriri L, Lee NM, Wallace LJ, Kwesiga B. Beyond cost-effectiveness, morbidity and mortality: a comprehensive evaluation of priority setting for HIV programming in Uganda. BMC Public Health 2019; 19(1):359. doi: 10.1186/s12889-019-6690-8 [Crossref] [ Google Scholar]
9. Guindo LA, Wagner M, Baltussen R. From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc 2012; 10(1):9. doi: 10.1186/1478-7547-10-9 [Crossref] [ Google Scholar]
10. Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health 2009; 14(8):930-939. doi: 10.1111/j.1365-3156.2009.02311.x [Crossref] [ Google Scholar]
11. United Nations. Transforming our World: The 2030 Agenda for Sustainable Development. New York: United Nations; 2015.
12. Holm S. The second phase of priority setting Goodbye to the simple solutions: the second phase of priority setting in health care. BMJ 1998; 317(7164):1000-1002. [ Google Scholar]
13. Kapiriri L, Martin DK. A strategy to improve priority setting in developing countries. Health Care Anal 2007; 15(3):159-167. doi: 10.1007/s10728-006-0037-1 [Crossref] [ Google Scholar]
14. Baltussen R, Jansen MPM, Bijlmakers L. Value assessment frameworks for HTA agencies: the organization of evidence-informed deliberative processes. Value Health 2017; 20(2):256-260. doi: 10.1016/j.jval.2016.11.019 [Crossref] [ Google Scholar]
15. Norris SL, Rehfuess EA, Smith H. Complex health interventions in complex systems: improving the process and methods for evidence-informed health decisions. BMJ Glob Health 2019; 4(Suppl 1):e000963. doi: 10.1136/bmjgh-2018-000963 [Crossref] [ Google Scholar]
16. Rehfuess EA, Stratil JM, Scheel IB, Portela A, Norris SL, Baltussen R. The WHO-INTEGRATE evidence to decision framework version 10: integrating WHO norms and values and a complexity perspective. BMJ Glob Health 2019; 4(Suppl 1):e000844. doi: 10.1136/bmjgh-2018-000844 [Crossref] [ Google Scholar]
17. Daniels N, Sabin J. The ethics of accountability in managed care reform. Health Aff (Millwood) 1998; 17(5):50-64. doi: 10.1377/hlthaff.17.5.50 [Crossref] [ Google Scholar]
18. Daniels N. Accountability for reasonableness. BMJ 2000; 321:1300-1301. doi: 10.1136/bmj.321.7272.1300 [Crossref] [ Google Scholar]
19. Daniels N, Sabin JE. Accountability for reasonableness: an update. BMJ 2008; 337:a1850. doi: 10.1136/bmj.a1850 [Crossref] [ Google Scholar]
20. Daniels N, Sabin J. Just Health: Meeting Health Needs Fairly. 1st ed. Cambridge: Cambridge University Press; 2008.
21. Daniels N, Sabin J. The ethics of accountability in managed care reform. Health Aff (Millwood) 1998; 17(5):50-64. doi: 10.1377/hlthaff.17.5.50 [Crossref] [ Google Scholar]
22. Friedman A. Beyond accountability for reasonableness. Bioethics 2008; 22(2):101-112. doi: 10.1111/j.1467-8519.2007.00605.x [Crossref] [ Google Scholar]
23. Singer PA, Martin DK, Giacomini M, Purdy L. Priority setting for new technologies in medicine: qualitative case study. BMJ 2000; 321(7272):1316-1318. doi: 10.1136/bmj.321.7272.1316 [Crossref] [ Google Scholar]
24. Tyler TR. Psychological perspectives on legitimacy and legitimation. Annu Rev Psychol 2006; 57:375-400. doi: 10.1146/annurev.psych.57.102904.190038 [Crossref] [ Google Scholar]
25. Greenberg J, Folger R. . Procedural justice, participation, and the fair process effect in groups and organizations 1983.
26. Saltelli A, Giampietro M. What is wrong with evidence based policy, and how can it be improved?. Futures 2017; 91:62-71. doi: 10.1016/j.futures.2016.11.012 [Crossref] [ Google Scholar]
27. World Health Organization (WHO). WHO Handbook for Guideline Development. 2nd ed. Geneva, Switzerland: WHO; 2014.
28. Ham C. Priority setting in health care: learning from international experience. Health Policy 1997; 42(1):49-66. doi: 10.1016/s0168-8510(97)00054-7 [Crossref] [ Google Scholar]
29. Niessen LW, Bridges J, Lau BD, et al. Assessing the Impact of Economic Evidence on Policymakers in Health Care-A Systematic Review. Rockville, MD: Agency for Healthcare Research and Quality (US); 2012.
30. Alonso-Coello P, Schünemann HJ, Moberg J. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices 1: Introduction. BMJ 2016; 353:i2016. doi: 10.1136/bmj.i2016 [Crossref] [ Google Scholar]
31. Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Eff Resour Alloc 2006; 4:14. doi: 10.1186/1478-7547-4-14 [Crossref] [ Google Scholar]
32. Nutt PC. Surprising but true: half the decisions in organizations fail. Acad Manag Perspect 1999; 13(4):75-90. doi: 10.5465/ame.1999.2570556 [Crossref] [ Google Scholar]
33. Robinson R. Limits to rationality: economics, economists and priority setting. Health Policy 1999; 49(1-2):13-26. doi: 10.1016/s0168-8510(99)00040-8 [Crossref] [ Google Scholar]
34. Gigerenzer G. . Decision making: nonrational theories 2001.
35. Moberg J, Oxman AD, Rosenbaum S. The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst 2018; 16(1):45. doi: 10.1186/s12961-018-0320-2 [Crossref] [ Google Scholar]
36. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet 2017; 390(10092):415-423. doi: 10.1016/s0140-6736(16)31592-6 [Crossref] [ Google Scholar]
37. Gopinathan U, Hoffman SJ. Institutionalising an evidence-informed approach to guideline development: progress and challenges at the World Health Organization. BMJ Glob Health 2018; 3(5):e000716. doi: 10.1136/bmjgh-2018-000716 [Crossref] [ Google Scholar]
38. Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc 2020; 18:8. doi: 10.1186/s12962-020-0203-6 [Crossref] [ Google Scholar]
39. World Health Organization (WHO). Consolidated Guideline on Sexual and Reproductive Health and Rights of Women Living with HIV. Geneva, Switzerland: WHO; 2017.
40. World Health Organization (WHO). Communicating Risk in Public Health Emergencies: A WHO Guideline for Emergency Risk Communication (ERC) Policy and Practice. Geneva, Switzerland: WHO; 2018.
41. World Health Organization (WHO). WHO Recommendations on Antenatal Care for a Positive Pregnancy Experience. Geneva, Switzerland: WHO; 2014.
42. Mayring P. Qualitative Content Analysis: Theoretical Foundation, Basic Procedures and Software Solution. Klagenfurt, Austria: Social Science Open Access Repository; 2014.
43. Cope MB, Allison DB. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. Int J Obes (Lond) 2010; 34(1):84-88. doi: 10.1038/ijo.2009.239 [Crossref] [ Google Scholar]
44. Cromwell I, Peacock SJ, Mitton C. ‘Real-world’ health care priority setting using explicit decision criteria: a systematic review of the literature. BMC Health Serv Res 2015; 15:164. doi: 10.1186/s12913-015-0814-3 [Crossref] [ Google Scholar]
45. Noorani HZ, Husereau DR, Boudreau R, Skidmore B. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care 2007; 23(3):310-315. doi: 10.1017/s026646230707050x [Crossref] [ Google Scholar]
46. Asante AD, Zwi AB. Factors influencing resource allocation decisions and equity in the health system of Ghana. Public Health 2009; 123(5):371-377. doi: 10.1016/j.puhe.2009.02.006 [Crossref] [ Google Scholar]
47. Lasry A, Carter MW, Zaric GS. Allocating funds for HIV/AIDS: a descriptive study of KwaDukuza, South Africa. Health Policy Plan 2011; 26(1):33-42. doi: 10.1093/heapol/czq022 [Crossref] [ Google Scholar]
48. Baltussen R, Stolk E, Chisholm D, Aikins M. Towards a multi-criteria approach for priority setting: an application to Ghana. Health Econ 2006; 15(7):689-696. doi: 10.1002/hec.1092 [Crossref] [ Google Scholar]
49. Baltussen R, ten Asbroek AH, Koolman X, Shrestha N, Bhattarai P, Niessen LW. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal?. Health Policy Plan 2007; 22(3):178-185. doi: 10.1093/heapol/czm010 [Crossref] [ Google Scholar]
50. World Health Organization (WHO). Everybody’s Business -- Strengthening Health Systems to Improve Health Outcomes: WHO’s Framework for Action. Geneva: WHO; 2007.
51. Ryan MA. Adherence to clinical practice guidelines. Otolaryngol Head Neck Surg 2017; 157(4):548-550. doi: 10.1177/0194599817718822 [Crossref] [ Google Scholar]
52. Fischer F, Lange K, Klose K, Greiner W, Kraemer A. Barriers and strategies in guideline implementation-a scoping review. Healthcare (Basel) 2016; 4(3). doi: 10.3390/healthcare4030036 [Crossref]
53. McDonald S, Elliott JH, Green S, Turner T. Towards a new model for producing evidence-based guidelines: a qualitative study of current approaches and opportunities for innovation among Australian guideline developers. F1000Res 2019; 8:956. doi: 10.12688/f1000research.19661.1 [Crossref] [ Google Scholar]
54. Turner TJ. Developing evidence-based clinical practice guidelines in hospitals in Australia, Indonesia, Malaysia, the Philippines and Thailand: values, requirements and barriers. BMC Health Serv Res 2009; 9:235. doi: 10.1186/1472-6963-9-235 [Crossref] [ Google Scholar]
55. Florez ID, Morgan RL, Falavigna M. Development of rapid guidelines: 2 A qualitative study with WHO guideline developers. Health Res Policy Syst 2018; 16(1):62. doi: 10.1186/s12961-018-0329-6 [Crossref] [ Google Scholar]
56. Setty K, Jiménez A, Willetts J, Leifels M, Bartram J. Global water, sanitation and hygiene research priorities and learning challenges under Sustainable Development Goal 6. Dev Policy Rev 2020; 38(1):64-84. doi: 10.1111/dpr.12475 [Crossref] [ Google Scholar]
57. Tromp N, Prawiranegara R, Siregar A. Translating international HIV treatment guidelines into local priorities in Indonesia. Trop Med Int Health 2018; 23(3):279-294. doi: 10.1111/tmi.13031 [Crossref] [ Google Scholar]
58. Heintz E, Lintamo L, Hultcrantz M. Framework for systematic identification of ethical aspects of healthcare technologies: the SBU approach. Int J Technol Assess Health Care 2015; 31(3):124-130. doi: 10.1017/s0266462315000264 [Crossref] [ Google Scholar]
59. Norheim OF, Baltussen R, Johri M. Guidance on priority setting in health care (GPS-Health): the inclusion of equity criteria not captured by cost-effectiveness analysis. Cost Eff Resour Alloc 2014; 12:18. doi: 10.1186/1478-7547-12-18 [Crossref] [ Google Scholar]
60. Johnson AP, Sikich NJ, Evans G. Health technology assessment: a comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care 2009; 25(2):141-150. doi: 10.1017/s0266462309090199 [Crossref] [ Google Scholar]
61. Treweek S, Oxman AD, Alderson P. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci 2013; 8:6. doi: 10.1186/1748-5908-8-6 [Crossref] [ Google Scholar]
62. Alonso-Coello P, Schünemann HJ, Moberg J. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices 1: Introduction. BMJ 2016; 353:i2016. doi: 10.1136/bmj.i2016 [Crossref] [ Google Scholar]
63. Vogel JP, Oxman AD, Glenton C. Policymakers’ and other stakeholders’ perceptions of key considerations for health system decisions and the presentation of evidence to inform those considerations: an international survey. Health Res Policy Syst 2013; 11:19. doi: 10.1186/1478-4505-11-19 [Crossref] [ Google Scholar]
64. Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are enough?. Qual Health Res 2017; 27(4):591-608. doi: 10.1177/1049732316665344 [Crossref] [ Google Scholar]
65. Stratil JM, Baltussen R, Scheel I, Nacken A, Rehfuess EA. Development of the WHO-INTEGRATE evidence-to-decision framework: an overview of systematic reviews of decision criteria for health decision-making. Cost Eff Resour Alloc 2020; 18:8. doi: 10.1186/s12962-020-0203-6 [Crossref] [ Google Scholar]