Examining and Contextualizing Approaches to Establish Policy Support Organizations – A Mixed Method Study

Background

Policy-making is a complex process where evidence is one factor among many that shape policy-makers’ decisions and actions.¹ Policies informed by evidence are likely to be more effective and less expensive than policies formulated without it.² Therefore, evidence-informed policies give policy-makers confidence in the decisions they make.² Evidence can play an important role in clarifying a problem, framing viable options to address a problem, identifying implementation considerations and developing monitoring and evaluation plans that enable rapid-cycle improvements over time to implemented polices.^3-5 Furthermore, evidence can also be used to understand the contextual factors that influence the selection of particular policy options.³

Despite the importance and benefits of evidence-informed policy-making (EIPM), several barriers constrain the use of research evidence in health-system policy-making processes. Specifically, barriers can include limited access to high-quality and relevant research evidence, poor communication between policy-makers and researchers, combined with the complex nature of policy-making where evidence needs to be considered alongside institutional factors, pressure from interest groups and often competing political and societal values.^6,7

Over the last decade EIPM has gained global attention and a large body of literature from both academic and policy communities has emerged around how to address the challenges of supporting EIPM. Accordingly, there has been an increase in the number of policy support organizations (PSOs) that have been created to link research to policy and to foster the systematic use of research evidence in health systems policy-making.^8-10 Efforts from the World Health Organization (WHO) to establish such organizations started in 2005 through the creation of the global Evidence-Informed Policy Network to promote the systematic and transparent use of health research evidence in policy-making.^11,12

We know from the literature that typically, a PSO is a national or state-level entity that may take several different forms. It might be a web-based entity, a network that forms around a particular issue or an event, or it might have conventional office premises.^13,14 A PSO can stand-alone, or be part of a government or academic institution.^9,12,15,16 Each of these forms has a set of advantages and disadvantages. Standing alone, a PSO might have the advantage of neutrality and independence to successfully broker among different stakeholders; yet being independent might affect the organization’s financial stability.^9,12,15,16 On the other hand, being part of government, is advantageous in understanding the policy-making process and in strengthening the capacity of policy-makers to access, assess, adapt and apply research evidence. However, this proximity to policy-makers may compromise the organization neutrality which is essential in EIPM.^9,12,15,16

The literature further identify that a PSO may offer various services. These services include facilitating meetings among multiple stakeholders (eg, citizen panels and stakeholder dialogues), identifying and documenting local research (eg, administering clearinghouses), synthesizing and packaging research evidence (eg, rapid responses and policy briefs), aligning research topics with policy needs (eg, priority-setting exercises), and strengthening the capacity of researchers (eg, to understand the policy process) and policy-makers (eg, to access, assess, adapt and apply research evidence).^9,12-20

Although the literature provides insight about the features of PSOs’ (eg, location and services offered) and some country-specific examples of establishing PSOs,^21-24 there has been less emphasis on deriving context-specific insights that can be used to generate themes that may be more broadly applicable across a range of settings. We recently developed a conceptual framework about the process of establishing PSOs based on a critical interpretive synthesis.²⁵ However, there is a need to build on this work to integrate these theoretical findings with real-world experiences to provide additional insight into approaches for establishing PSOs and identifying contextual factors that influence the organizations’ establishment and its features. To achieve this goal, we conducted a sequential mixed methods study to complement the conceptual framework we developed earlier. Specifically, our aim was to solicit real-world insights from multiple policy-support organizations that exist in different settings and have different features.

Methods

To develop a broad as well as in-depth understanding about the phenomenon of establishing PSO, we used a sequential mixed method approached by combining elements of quantitative (survey) and qualitative (key informant interviews) research approaches. We first conducted a survey to identify the views and experiences of those who were directly involved in the establishment of PSOs that have been developed and implemented across a variety of political-, health- and research-system contexts. The survey findings were then used to refine a sample and interview guide for the qualitative interview study that we conducted in the second phase of the study. The interviews focused on: (i) identifying insights about the process of establishing a PSO; (ii) identifying the critical junctures in the life of PSOs; and (iii) identifying the approaches that PSOs have used to ensure organizational sustainability in the long term. Taken together, the quantitative and qualitative components yield a better mapping of the current features and contexts of different PSOs, in-depth understanding of their story of the establishment, and provided us with an opportunity to identifying different approaches for establishing PSOs that were not country-specific, which wouldn’t have been possible through each method in isolation.

Phase One: Survey

A questionnaire was developed and piloted with two PSOs which operate in different political, health, and research system contexts. The goal of the pilot study was to test the clarity of the questions and estimate the time needed to complete the questionnaire. The questionnaire covered five domains: (i) the political-, health- and research-system contexts in which the PSO operates; (ii) the focus/scope of the organization’s work; (iii) the PSO’s activities and products; (iv) the PSO’s attributes; and (v) the establishment phase of the PSO. Twenty-eight of the 37 questions asked were close ended.

Sampling and Recruitment

Purposive and snowball sampling techniques were used to identify the study population. The PSOs were purposively sampled to represent different political-, health-, and research-system contexts and demographic variables. The principal investigator (PI) generated a list of 57 potentially relevant organizations identified through electronic searches of peer-reviewed articles and organizations’ websites. The list was then reviewed by the research team which applied inclusion criteria, which resulted in 23 organizations being included in the sample. The inclusion criteria specified that for any organization to be eligible to be invited for the survey, it had to perform at least one of the following functions (or a closely related function): (i) produce systematic reviews, or other types of syntheses of research evidence in response to requests from policy-makers; (ii) identify and contextualize research evidence in response to requests from policy-makers; and (iii) plan, commission, or carry out evaluations of health policies in response to requests from policy-makers. The organizations were excluded if they receive core funding from industry, they only produce or provide health or healthcare utilization data, or those that do not primarily focus on translating research evidence for supporting health system policy-making (eg, clinical practice guideline or public health practice). In addition, social media was used to recruit organizations that were interested and eligible. The PI created a video to invite eligible organizations to participate in the study. The research team tweeted the video with the link to the survey. Finally, participants were asked to name similar organizations that would be eligible for the study.

Data Collection and Analysis

LimeSurvey was used to administer the survey. The survey link was sent by email with an invitation letter to the directors or senior managers of each eligible organization. Three reminders were sent to non-respondents a week, two weeks, and three weeks after the initial contact.

Survey data were summarized using simple descriptive statistics. The open-ended questions were analysed using a qualitative descriptive approach. The analysis was oriented toward summarizing the informational contents of that data regarding PSO features. Initially, the written comments were grouped by the questions to offer a comprehensive summary of the findings. While staying close to the data and to the surface of words and events, the summarized findings under each question were then coded further, and further modified in the course of analysis to reflect emerging themes and the research objectives.

Phase Two: Interviews

Semi-structured interviews were conducted using an interview guide with prompts focused on the establishment, critical juncture(s), and sustainability of the participant’s organization. The survey findings were used to inform the interview guide and it was iteratively revised as needed to allow for exploration of emerging themes and to validate assumptions or statements made by other participants.

Sampling and Recruitment

We purposively sampled PSOs to ensure a mix of those with different features of interest (eg, knowledge production vs. translation, extent of engagement with policy-makers, stakeholders and researchers, etc), that were located in high income countries vs. low- and middle-income countries and that were new (established within the last five years) vs. old (established within more than five years ago). Based on these criteria, we selected 10 PSOs from among the survey respondents. Once the final list was generated, the PI conducted three meetings with two research members (JNL and KM) who were familiar with the history of these organizations to provide additional context to use as interview prompts during the interviews. A summary of the sampling and recruitment strategy of both phases, the survey, and the interview can be found in Supplementary file 1.

Data Collection and Analysis

All participants were contacted via email to schedule a time for a telephone, Skype, or face to face interview with invitation letter and consent form. The aim was to conduct one interview for each PSO. However, in instances where the initial interviewee for a PSO was not able to speak to all of the areas of interest or they recommended an additional contact to provide further details, one additional interview per organization was scheduled.

In an earlier study we developed a framework that outlines the process of establishing PSOs.The framework includes four main stages: awareness, development, assessment, and maturation. The awareness stage provides the foundation for establishing a PSO by identifying the motivation that would advance the idea of establishing a PSO toward the development stage where the focus is on the actual implementation of a PSO. The assessment stage consists of monitoring and evaluating either specific programs and services provided by the PSO or its overall performance. Finally, the maturation stage focuses is ensuring sustainability over the long-term (More details about the framework can be found at Al Sabahi et al).²⁵

Using this framework, the data were analyzed using the constant comparative approach.²⁶ The open coding was started by classifying information into five main sections, which included the four stages in the framework and the contextual factors that facilitate or hinder the establishment. For each section, a number of codes were assigned to the interview for each respondent. These codes were then organized into categories, and the categories organized into larger themes. The themes and categories were refined iteratively. The findings from the survey open-ended questions were incorporated in this analysis for developing the themes where relevant. After the analysis was completed, we conducted member checking by sending a draft of the results to participants who were asked to review the findings and provide feedback.

Results

Survey Findings

A total of 19 surveys were completed, which included 14 from the 23 organizational leaders who were directly invited to participate and five additional surveys from the promotion of the survey on Twitter. The sample of participating organizations, although not perfectly balanced, are based on different geographical regions in the world (based on WHO regions) and from countries of different economic level (see Table S1, Supplementary file 2).

The participating organizations work in different political, health and research system contexts (see Table S1, Supplementary file 2). Although most organizations provide service at more than one jurisdictional level, the main focus of all the organizations is at the national and sub-national levels. In addition, most of the organizations operate in health systems that are mainly centralized (7/17), mainly publicly funded (12/18), and predominantly publicly delivered (11/18). Although more than two thirds of the organizations reported having centralized funds for health-system research, only half of the funding organizations prioritize knowledge translation (KT) activities and require collaboration with researchers and policy-makers. Neither the health system nor the research system arrangements seem to influence the jurisdictional level at which the organizations can provide service.

Organizational Characteristics

PSOs have unique features (see Table S2, Supplementary file 2) which are a critical enabler for their activities. Most PSOs indicated they are located as an independent organization (7/15) or in a university (5/15) with fewer embedded within government (3/15). However, slightly less than one-third of the organizations classified themselves as a government department and/or as having linkages with government institutions, which could be the reason for reporting the government as the most common source of funding (11/15). Having an executive board is the most common governance approach (9/15) across all regions, and (12/15) of the organizations have linkage with different organizations (with the most frequent collaborations being with governments and academic institutions). Organizational budgets ranged from $10 000 to $1.25 billion, however we suspect that those who reported very high budgets misinterpreted the question and provided the budget of the larger organizations in which they are embedded. Most organizations indicated that the most common background training of staff is in health services research and in population and public health research (13/15). For accessing needed research, most PSOs have access to more than one database. About two thirds of the organizations have a strategic plan and only half of them monitor and evaluate their impact and update their work irregularly. Another misinterpretation of the question may have occurred in relation to organizational age which was reported as ranging from 3-71 years, with those PSOs in larger organizations likely having reported that age instead of when their unit was established. For the purpose of establishing these organizations, participants reported that lessons from other organizations were helpful (8/15), one-third conducted a situation analysis (5/15) and few of them used a readiness assessment tool (1/15).

Organizational Focus, Activities, Products, and Target Audience Engagement

Across the different WHO regions, all organizations provide services in multiple domains and engage in a wide array of activities (see Table S3, Supplementary file 2), and (14/17) of the organizations reported engaging in the four main areas of policy development (ie, clarifying problems, framing options, identifying implementation considerations and supporting monitoring and evaluation). The most common activities to pursue this work are supporting learning about how to make evidence-informed decisions (14/16) and synthesizing research evidence (15/16). The data reflected that slightly more than two-thirds of the organizations formulate recommendations and among those that do, the most popular approach is through formal and informal consensus (6/10). These activities coincide with the different products produced (see Table S4, Supplementary file 2) with the most commonly produced products being evidence briefs (14/16), rapid syntheses or rapid reviews (12/16), databases of research evidence (10/16) and information about the health system in which the organization works (10/16). Using these activities and products, PSOs usually target different audience using different strategies (see Table S5, Supplementary file 2). In particular, health system policy-makers were more often engaged in PSO activities such as reviewing reports’ drafts (15/16) and being a member of the organizational governance or participants in the organization’s activities (14/16). Despite the international call for engagement of citizens in producing research and in developing policies and programs, this is the least frequently targeted audience by PSOs. This is consistent with our survey which found that only (7/16) convene deliberation with the citizens.

How the Survey Was Used to Inform the Second Phase of the Study

The survey results clearly indicate that PSOs vary in their contextual features, organizational attributes, focus, activities, products, and target audience. No special trend or pattern was observed to dominate organizations from a particular region, economic level, or setting. However, the results emphasized the importance of selecting organizations that have a rich story to contribute to answering the main research question about the organization establishment approach. Accordingly, after the complete analysis of the survey and carefully reading the participants answers to the open ended questions, a list of the participating organizations was generated with their main distinctive characteristics (the list also had comments on the depth of the participants responses) to guide the purposeful sampling of the participants to be interviewed. The study did not aim to make a comparison between organizations with different features, setting, or context. The aim was to get an in-depth insight from the full spectrum of PSOs about their establishment approach.

Qualitative Interview Findings

Eleven interviews were completed with 15 senior managers from nine organizations. Interview length varied from 45 to 90 minutes with an average of 60 minutes. Ten of them were conducted via Skype and one was face-to-face. We interviewed participants from organizations in five of the six WHO regions with three from the Americas, two from Africa, two from the Eastern Mediterranean, one from Europe, and one from the Western Pacific region, but none from South-East Asia.

We organize our findings below according to the four stages of developing a PSO (awareness, development, assessment, and maturation) that we used to structure our interview guide.In each section below we outline the common approaches and strategies used and actions taken for each stage were identified. This is followed by an analysis of the contextual factors that participants identified as being important for establishing a PSO, which includes a SWOT (Strengths, Weaknesses, Opportunities, and Threats) analysis.

Awareness Stage – Fostering the Conditions Needed to Establish A PSO

Our interviews revealed three common approaches that respondents identified as helpful to raise awareness about the need for establishing a PSO. We provide an overview of each in Table 1 along with illustrative quotes from the interviews. The first approach related to establishing a supportive climate among policy-makers and researchers to advocate for the importance and the advantages of using evidence in policy-making. Participants reported that conducting workshops, distributing information about KT (eg, books, CDs, or pamphlets), and conducting one-on-one meetings with policy-makers, or large-group meetings with researchers and policy-makers to raise the awareness about the EIPM were good starting points to establish a supportive climate for EIPM. Participants highlighted that these initiatives focused mainly on clarifying the concept of EIPM in order to reach consensus about what has to be done to support the use of evidence in policy-making and to build buy-in from policy-makers.

Second, participants focused on activities for making adequate resources (ie, human resources, financial resources, and infrastructure) available to help researchers and policy-makers determine the feasibility of establishing such an organization. Through participants sharing their stories with establishing a PSO we found that the establishment was derived either by the availability of the financial resources from local, national or international organizations, or by the availability of champion(s) who believed in the importance of EIPM and have the interest, the skills, and the experience needed for establishing such entity, or both. Participants highlighted that attracting resources and highlighting this could increase policy-makers’ willingness to accept the idea for establishing a PSO.

Lastly, participants highlighted activities for addressing the gap between research and policy communities and supporting the use of research evidence in policy-making. Participants highlighted two types of gaps in the system. The first is the gap between the producers (researchers) and users (policy-makers) of the research evidence. Mainly due to the poor communication between the policy and research communities, many participants indicated that policy-makers were not aware of the existence of evidence that might be useful for formulating policies. At the same time, researchers produce evidence that was more of an academic interest instead of focused on policy-makers’ needs. The second gap was related to what was offered as support for informing policy-making using evidence. For example, several participants pointed out that there was enough work in using evidence for developing clinical practice guidelines, health technology assessment, and public health practice, but not for informing decisions about the health systems that get the programs, services and products to people who need them. Participants emphasized that highlighting these gaps and the benefits of addressing them were used as a justification to express the need for establishing a PSO and gain support from the national and international organizations.

Development Stage – Understanding the Situation or Proving the Concept and Defining the Organization’s Activities

Participants identified three different approaches as part of the development stage for a PSO, which we present in Table 2 along with illustrative quotes from participants. The first approach focuses on understanding the situation or proving the concept for establishing the PSO. Most participants reported that they conducted a situation analysis before establishing their PSO using different tools (eg, SWOT analysis, stakeholder power analysis, feasibility assessment, situation analysis using WHO guideline). A situation analysis was noted as having been conducted to identify: the target audience, key players who might have an interest in the organization’s work and can advocate for it, competitors, model organizations, individuals who have power and could influence the process of establishing a PSO (whether in terms of material resources or political power), capacity constraints, priorities for the PSO, and potential partners. Furthermore, participants also reported that situation analysis was conducted to understand policy-makers’ and researchers’ views and experiences with supporting the use of evidence in policy-making.

Some organizations complemented the situation analysis with a proof of concept to give a concrete example for the target audience about what the organization can do and how they can benefit from its services. Most participants reported that they used the findings from the situation analysis for setting the organization’s perspectives and plans.

Second, strategic development was used to guide the subsequent steps for establishing the PSO. Participants that conducted a situation analysis reported that the findings were useful for carefully framing the organization’s perspectives. In addition, participants also reported that they constructed a temporary steering committee that formulated the organization’s vision, mission, and objectives. For some organizations, the members of the steering committee were structured in a way that allowed it to draw on the pre-existing expertise at the hosting organization. For example, one of the participants from a PSO that is part of a university, constructed a temporary steering committee that involved expert from the business school to help with developing the business plan, budgeting, and identifying potential risk factors. Other organizations selected the members in a way that ensured representatives from the policy and research communities were engaged to create an avenue for their communication during the early stages of the organization’s establishment.

Lastly, participants identified approaches for implementing start-up activities for their PSO. The most common activities identified at the early stage of developing a PSO were for building the capacity of researchers to synthesize evidence in a user-friendly format and for building the capacity of policy-makers to access, appraise, synthesize and use evidence. Workshops, meetings, face-to-face or online courses were the most common methods used for building the capacities of these groups. Participants also emphasized the importance of applying for all available funding opportunities at the national and international levels and therefore, building the capacity of PSO staff on how to apply for grants was reported as being highly useful for attracting more funds. In addition, training for conducting or leading the different activities of the PSO (eg, writing policy briefs or rapid syntheses, facilitating stakeholder dialogues, administering a database/clearinghouse for research evidence) was noted as also being crucial for implementing start-up activities. Participants indicated that building capacity typically took the form of mentorship by an expert (eg, the PSO leader and/or an in-house expert or from other organization or country), workshops, courses, and by providing scholarships for graduate students engaged in the PSO.

Assessment Stage – Identifying Opportunities for Enhancing the PSO

Once a PSO is developed and starts to function on a regular basis, the next step needs to focus on assessing the quality of its work and performance to support continued enhancement of the PSO. As outlined in Table 3, we identified three common approaches from participants to guide the assessment stage. First, participants emphasized the need to create a plan to know what to assess when, how, and why. This was identified as being essential to track the organization’s performance (either by comparing the organization performance with its previous year’s achievements, or by comparing it with other similar organizations, or both) to identify areas and mechanisms for improvement. Participants consistently linked robust assessment to the organization’s reputation and trustworthiness in order to sustaining the demand for PSO services. Some participants noted that their organizations have a clear assessment plan as part of their strategic plan, and they set some key performance indicators for each goal they wanted to achieve each year. Indicators identify by participants to assess the organization’s performance included tracking the number of products (eg, policy briefs, systematic reviews and rapid syntheses completed, policy dialogues and citizen panels convened, capacity building workshops provided), number of grants received, number of collaborations built, and media coverage.

After creating an assessment plan, the next approach involves implementing the assessment activities to identify what is working and what needs to be improved. Participants indicated that the assessment of the organization took place either annually (by the end of each financial year), or after specific activities (eg, end of project, after the delivery of a training workshop, policy dialogue, or a citizen panel), or by the end of the funding term in the cases where the organization started as a pilot project funded by international organization. Participants frequently reported that the assessment was conducted by PSO staff. One of the participants indicated that their PSO invited an external team (from a neighbor country that has a similar organization) to assess their organization to avoid bias and to be more objective. However, they found that experience to be challenging because it was hard to explain exactly how the system works because the impact on decision-making may not be apparent to someone without intimate knowledge about how local policy-making processes work. For aspects related to target audience feedback, surveys and interviews were used by most of the participants.

Lastly, following phases of assessment, participants emphasized the need to use the results to identify changes that could be made to position the organization to enhance its impacts and to ensure sustainability in the long run. The most frequently reported change to a PSO by participants after an assessment was for moving towards institutionalizing the organization (eg, from being a pilot project to be institutionalized in a pre-existing organization), expanding the organization’s audience (eg, involving the media), and expanding the organization’s scope of work (eg, from health systems to health and social systems). These efforts were highlighted as critical junctures in the organization’s work that led to a significant shift in its success (success was subjectively defined based on each organizations’ leader’s view, experience, and objectives).

Maturation Stage – Building Sustainability in the Long-term

Participants from all organizations reported facing a challenge that threatened its sustainability. In discussing these challenges, we identified three common approaches which were used (or planned to be used) to position PSOs to be more sustainable in the long-term (see Table 4). The first approach is sustaining sufficient funding from the hosting organization or other national and international organizations. All participants highlighted the importance of ensuring financial stability for the PSO. Most participants obtained government support using different methods. Some identified having an agreement with their government to contribute to the PSO annual budget for a predetermined amount of work (eg, systematic review, policy briefs, policy dialogue). Others started offering courses related to KT through contracts with their government to train their staff who can then provide internal support to policy-makers, and/or by offering in person or online course for local and international participants to raise some funding through course fees. Still others sought funding through contracts with stakeholders to provide regular services (eg, rapid syntheses, stakeholder dialogue and citizen panels) or project-based contracts to offer specific services.

Next, attracting and retaining staff with the right skills was noted being a challenge that had to be addressed in many PSOs. Participants consistently noted the need to have incentives (eg, competitive salary to avoid been attracted by competitor organizations or offering tenure positions for faculty) to retain staff. Participants from PSOs embedded in an academic institution expressed the importance of linking the PSO with programs provided through the university (eg, Master’s and PhD) as an opportunity to engage and retain skilled individuals, as well as a mechanism to address staff shortage by hiring students as part-time employees. This is dependent, however, based on whether there are staff at the PSO that are able to supervise graduate students.

Lastly, institutionalizing the PSO by making its work embedded in the target organizations’ norms, cultures and processes was identified as essential to sustain demand from policy-makers. Although institutionalization was reported to play an important role in PSO sustainability, it was conceptualized differently by participants. Some participants approached institutionalization by attaching or embedding the PSO in a pre-existing institution. This approach was reported to be useful in ensuring access to the basic needs for the organization to function (eg, physical space, access to technology and staff salaries). The reputation of the hosting organization was also reported as adding value to the PSO. Other participants focused on institutionalizing the organization’s work by making it part of the regular work of the target audience to sustain the use of the organization’s services. For example, some organizations incorporated the types of products produced by PSOs (ie, those that are not peer-reviewed manuscripts, such as rapid syntheses, policy briefs and/or thematic analyses of stakeholder dialogues) as part of the annual performance review for faculty members. Participants also reported the importance of institutionalizing collaborations and networks by making them formal (eg, signed contract or memorandum of understanding) and less dependent on a particular individual to avoid the discontinuity that can result from staff turnover from any organization.

None of these approaches were found to dominate over the other. However, being mindful of how to hire and retain staff and having the flexibility to quickly readjust the organization’s priorities to accommodate the continual changes in the health system were the most common strategies recommended by participants to approach sustainability. Our findings also highlight that PSO sustainability is an issue regardless of the organization’s age, which emphasizes the importance of taking maturation into consideration as early as possible given that it heavily depends on the actions that take place in the earlier stages (eg, awareness about EIPM and the organization’s work, the organizational arrangements, and the quality of the organization’s works).

Contextual Factors

Our interviews revealed a group of contextual factors that facilitated or hindered the establishment of PSOs throughout the four stages. Some of the factors are internal at the individual or organization level, while others were external (eg, at the system level). Table 5 provides a full list of these factors in a SWOT analysis format to distinguish between the internal and the external facilitators and barriers.

We found that the strengths and opportunities of some organizations were the weaknesses and threats of the others. Therefore, the organizations were capitalizing on the facilitators to mitigate the barriers they were facing. For instance, among the organizations that had a strength in the form of the availability of start-up funds (from governments, international organizations, the hosting organization, or a combination of different sources), many also faced the barrier of not having clear objectives, scope, and approaches to enhance the use of research in policy-making. In contrast, PSOs without sufficient start-up funds, expressed that they had clear objectives and a strategic plan that were framed by skillful personnel who were ready to lead the PSO. Consequently, this helped in attracting funds from local and international organizations. Similarly, many PSOs operating in a context of limited awareness about the importance of EIPM among policy and research communities, indicated that they capitalized on the international climate and publication of major reports (eg, SUPPORT Tools for evidence-informed health Policy-making) that highlight how the health system could be better informed by evidence.

Because a PSO’s work is based on collaboration between different organizations and communities, it is important to take into consideration the facilitators and barriers that emerge at the individual, organization, and system level. A PSO should try to mobilize the weaknesses and threats to strengths and opportunities by considering the different strategies highlighted in the four stages. Although each factor listed in Table 5 has been reported by more than two organizations, there was no single facilitator nor barrier reported by all organizations simultaneously.

Discussion

Principal Findings 

We found that PSOs function at different political, health, and research system contexts and at more than one jurisdictional level. However, regardless of the contexts in which they operate, we found that the PSOs we surveyed provide services in multiple domains and engage in a wide array of activities.

In our interviews with leaders of PSOs we derived in-depth insights approaches and strategies used to establish a PSO, as well as potential barriers and facilitators that can affect the process across the four stages of developing a PSO. Common approaches to establishing a PSO related to:

• building awareness for the need for a PSO (eg, establishing a supportive climate for EIPM, making adequate resources available, and demonstrating the conceivable role a PSO can play in addressing the gap between research and policy communities);

• developing the PSO (eg, understanding the situation where the organization would be established, setting the perspectives strategically to guide the subsequent steps for establishing the organization, and conducting the start-up activities to be able to function in a regular way);

• assessing the PSO to identify potential areas for enhancement (eg, planning for assessment, running the assessment activities to know what has been done appropriately and what needs to be improved, and considering the appropriate changes need to ensure sustainability in the long-term); and

• supporting maturation to build sustainability in the long-term (eg, ensuring sufficient funding; sustaining appropriate capacities by finding the right personnel, retaining them, and enhancing their skills; and approaching the institutionalization of the organization by making its work embedded in the target organizations’ norms, cultures, and processes).

We found that key barriers to this process included lack of funding, lack of human resources, and division of policy authorities. Key facilitators included awareness and government interest in EIPM, availability of support from national and international organizations, as well as at the individual level, organization level and system level. It is important to acknowledge that there were difference in the approaches and strategies used across contexts, and those working on establishing PSOs should be mindful about how these factors might differ in their context.

By incorporating insight form PSOs’ leaders from around the world, our findings offer unique contributions to the literature. We provided lessons for establishing PSOs that are not country or region specific.^13,27-32 Our study was inspired by the existing literature that focuses on identifying approaches to support EIPM (eg, setting the priorities, building capacities, and providing rapid response services) and the factors that might influence these approaches.^13,27-32 Our study builds on this literature by providing insights about the process for establishing the organizations that support EIPM. While the literature highlighted the role of the government structure in the use of evidence in the policy-making process,^33,34 our study was unable to provide evidence that supports any particular conclusion regarding the role of government structures in establishing PSOs. This might have three possible explanations which will need further investigation. First, there is a possibility that participants were not aware of the exact role of the government structure in establishing PSOs, either because they do not consider it as an important factor, or because it was not obvious enough to be considered as an important factor. The second potential explanation is that the diffusion of ideas about the importance of EIPM and establishing a KTP was mainly pushed by international organizations (many of the participants mentioned the support from the international organization). As a result, the some may have viewed government structures as not relevant given the more important influence from international organizations that influenced the actions of those structures. The last potential explanation is that the government structure was not an important factor for establishing PSO.

The findings of this study help to enrich the framework we developed in an earlier study that illustrated the process of establishing a PSO by providing more detailed approaches and strategies that can be used in each stage.

Strengths and Limitations

A key strength of our study was the use of a sequential mixed method approach. Conducting the survey first allowed us to document general PSO characteristics and then use those characteristics to purposively select a sample of PSOs that would provide rich insights from different regions of the world, from PSOs operating in different settings and from those which have demonstrated sustainability over the long-term and that have been recently established. In addition, our approach allowed us to confirm, triangulate and derive additional insights from the survey findings, which provides for a much richer set of data. A potential limitation of our study is the small sample size, which did not allow us to reliably identify patterns in establishment approaches across different contexts, regions, or organizational settings. However, this limitation is unavoidable given the limited number of PSOs that exist.

Implications for Policy and Research

The main implication of our findings is that they provide rich insights about the process of establishing a PSO from leaders who have undertaken the process. As a result, the stages of establishment presented in this paper and the insights about them from leaders of PSOs can provide important guidance for those who are considering establishing a PSO or who are already in the process of doing so. For example, our findings revealed the importance of conducting a situation analysis, and those who are interested in establishing a PSO should consider undertaking a similar process to enable them to develop the most appropriate strategy according to the context in which they work. In addition, our finding can be informative for leaders of PSOs to expand or refine their scope of work, such as by selecting a new program or service to provide and refining monitoring and evaluation plans to include assessments of the impact of their work.

Future research could include conducting: (i) a larger-scale study to identify patterns in PSO establishment approaches, (ii) co-design studies that document and provide in-depth insights about the establishment of one or more PSOs; (iii) evaluations of the impact of PSOs on supporting EIPM; (iv) apply broader categories and themes to a specific context to determine whether they can be useful in understanding the process, and whether these themes can facilitate cross-context comparisons of setting up PSOs.

Ethical issues

This project was approved by Hamilton Integrated Research Ethics Board (HiREB) under the project ID 5377.

Competing interests

Authors declare that they have no competing interests.

Authors’ contributions

SAS was responsible for the conception and design, collection of data, analysis and interpretation of data, drafting and reviewing the manuscript. MGW contributed to the conception and design, analysis and interpretation, drafting and reviewing the manuscript. MGW was also responsible for supervising this work. JNL, FEJ and KM provided critical revisions of the manuscript for important intellectual content.

Disclaimer

The views expressed in this paper represent the opinions of the authors and not an official position of the institutions of their affiliation.

Supplementary files

References

1. Strydom WF, Funke N, Nienaber S, Nortje K, Steyn M. Evidence-based policymaking: a review. S Afr J Sci 2010; 106(5-6):17-24. [ Google Scholar]
2. Campbell S, Benita S, Coates E, Davies P, Penn G. Analysis for Policy: Evidence-Based Policy in Practice. Government Social Research Unit; 2007.
3. Fierlbeck K. Paradigm freeze: why it is so hard to reform health-care policy in Canada. Can Public Adm 2014; 57(3):478-481. [ Google Scholar]
4. Kingdon J. Agendas, Alternatives, and Public Policies, Update Edition, with an Epilogue on Health Care. Pearson New International Edition; 2013.
5. Fafard P. Evidence and Healthy Public Policy: Insights from Health and Political Sciences. Quebec, CAN: Canadian Policy Research Networks; 2008.
6. Jewell CJ, Bero LA. “Developing good taste in evidence”: facilitators of and hindrances to evidence-informed health policymaking in state government. Milbank Q 2008; 86(2):177-208. doi: 10.1111/j.1468-0009.2008.00519.x [Crossref] [ Google Scholar]
7. Innvaer S, Vist G, Trommald M, Oxman A. Health policy-makers’ perceptions of their use of evidence: a systematic review. J Health Serv Res Policy 2002; 7(4):239-244. doi: 10.1258/135581902320432778 [Crossref] [ Google Scholar]
8. Ham C, Hunter DJ, Robinson R. Evidence based policymaking. BMJ 1995; 310(6972):71-72. doi: 10.1136/bmj.310.6972.71 [Crossref] [ Google Scholar]
9. Bennett S, Corluka A, Doherty J. Influencing policy change: the experience of health think tanks in low- and middle-income countries. Health Policy Plan 2012; 27(3):194-203. doi: 10.1093/heapol/czr035 [Crossref] [ Google Scholar]
10. Ongolo-Zogo P, Lavis JN, Tomson G, Sewankambo NK. Climate for evidence informed health system policymaking in Cameroon and Uganda before and after the introduction of knowledge translation platforms: a structured review of governmental policy documents. Health Res Policy Syst 2015; 13:2. doi: 10.1186/1478-4505-13-2 [Crossref] [ Google Scholar]
11. World Health Organization (WHO). Report from the Ministerial Summit on Health Research: Identify Challenges, Inform Actions, Correct Inequities. WHO; 2005.
12. Scarlett J, Köhler K, Reinap M. Evidence-informed Policy Network (EVIPNet) Europe: success stories in knowledge translation. Public Health Panor 2018; 4(2):161-169. [ Google Scholar]
13. Kasonde JM, Campbell S. Creating a knowledge translation platform: nine lessons from the Zambia Forum for Health Research. Health Res Policy Syst 2012; 10:31. doi: 10.1186/1478-4505-10-31 [Crossref] [ Google Scholar]
14. Lavis JN, Oxman AD, Moynihan R, Paulsen EJ. Evidence-informed health policy 1 - synthesis of findings from a multi-method study of organizations that support the use of research evidence. Implement Sci 2008; 3:53. doi: 10.1186/1748-5908-3-53 [Crossref] [ Google Scholar]
15. El-Jardali F, Lavis J, Moat K, Pantoja T, Ataya N. Capturing lessons learned from evidence-to-policy initiatives through structured reflection. Health Res Policy Syst 2014; 12:2. doi: 10.1186/1478-4505-12-2 [Crossref] [ Google Scholar]
16. Langlois EV, Becerril Montekio V, Young T, Song K, Alcalde-Rabanal J, Tran N. Enhancing evidence informed policymaking in complex health systems: lessons from multi-site collaborative approaches. Health Res Policy Syst 2016; 14:20. doi: 10.1186/s12961-016-0089-0 [Crossref] [ Google Scholar]
17. Cheung A, Lavis JN, Hamandi A, El-Jardali F, Sachs J, Sewankambo N. Climate for evidence-informed health systems: a print media analysis in 44 low- and middle-income countries that host knowledge-translation platforms. Health Res Policy Syst 2011; 9:7. doi: 10.1186/1478-4505-9-7 [Crossref] [ Google Scholar]
18. Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ 2006; 84(8):620-628. doi: 10.2471/blt.06.030312 [Crossref] [ Google Scholar]
19. Lavis JN, Moynihan R, Oxman AD, Paulsen EJ. Evidence-informed health policy 4 - case descriptions of organizations that support the use of research evidence. Implement Sci 2008; 3:56. doi: 10.1186/1748-5908-3-56 [Crossref] [ Google Scholar]
20. Lavis JN, Paulsen EJ, Oxman AD, Moynihan R. Evidence-informed health policy 2 - survey of organizations that support the use of research evidence. Implement Sci 2008; 3:54. doi: 10.1186/1748-5908-3-54 [Crossref] [ Google Scholar]
21. Contandriopoulos D, Lemire M, Denis JL, Tremblay E. Knowledge exchange processes in organizations and policy arenas: a narrative systematic review of the literature. Milbank Q 2010; 88(4):444-483. doi: 10.1111/j.1468-0009.2010.00608.x [Crossref] [ Google Scholar]
22. Lavis JN, Lomas J, Hamid M, Sewankambo NK. Assessing country-level efforts to link research to action. Bull World Health Organ 2006; 84(8):620-628. doi: 10.2471/blt.06.030312 [Crossref] [ Google Scholar]
23. Ongolo-Zogo P, Lavis JN, Tomson G, Sewankambo NK. Initiatives supporting evidence informed health system policymaking in Cameroon and Uganda: a comparative historical case study. BMC Health Serv Res 2014; 14:612. doi: 10.1186/s12913-014-0612-3 [Crossref] [ Google Scholar]
24. Mijumbi RM, Oxman AD, Panisset U, Sewankambo NK. Feasibility of a rapid response mechanism to meet policymakers’ urgent needs for research evidence about health systems in a low income country: a case study. Implement Sci 2014; 9:114. doi: 10.1186/s13012-014-0114-z [Crossref] [ Google Scholar]
25. Al Sabahi S, Wilson MG, Lavis JN, El-Jardali F, Moat K, Vélez M. Examining and contextualizing approaches to establish policy support organizations – a critical interpretive synthesis. Int J Health Policy Manag 2020; In Press. 10.34172/ijhpm.2020.181.
26. Boeije H. A purposeful approach to the constant comparative method in the analysis of qualitative interviews. Qual Quant 2002; 36(4):391-409. doi: 10.1023/a:1020909529486 [Crossref] [ Google Scholar]
27. Brennan SE, Cumpston M, Misso ML, McDonald S, Murphy MJ, Green SE. Design and formative evaluation of the Policy Liaison Initiative: a long-term knowledge translation strategy to encourage and support the use of Cochrane systematic reviews for informing health policy. Evid Policy 2016; 12(1):25-52. [ Google Scholar]
28. Campbell D, Donald B, Moore G, Frew D. Evidence check: knowledge brokering to commission research reviews for policy. Evid Policy 2011; 7(1):97-107. doi: 10.1332/174426411x553034 [Crossref] [ Google Scholar]
29. Goering P, Butterill D, Jacobson N, Sturtevant D. Linkage and exchange at the organizational level: a model of collaboration between research and policy. J Health Serv Res Policy 2003; 8 Suppl 2:14-19. doi: 10.1258/135581903322405126 [Crossref] [ Google Scholar]
30. Mansilla C, Herrera CA, Basagoitia A, Pantoja T. The Evidence-Informed Policy Network (EVIPNet) in Chile: lessons learned from a year of coordinated efforts. Rev Panam Salud Publica 2017; 43:e36. doi: 10.26633/rpsp.2017.36 [Crossref] [ Google Scholar]
31. Moore GM, Redman S, Turner T, Haines M. Rapid reviews in health policy: a study of intended use in the New South Wales’ Evidence Check programme. Evid Policy 2016; 12(4):505-519. doi: 10.1332/174426415x14446635524057 [Crossref] [ Google Scholar]
32. Roby DH, Jacobs K, Kertzner AE, Kominski GF. The California Health Policy Research Program - supporting policy making through evidence and responsive research. J Health Polit Policy Law 2014; 39(4):887-900. doi: 10.1215/03616878-2743263 [Crossref] [ Google Scholar]
33. Liverani M, Hawkins B, Parkhurst JO. Political and institutional influences on the use of evidence in public health policy A systematic review. PLoS One 2013; 8(10):e77404. doi: 10.1371/journal.pone.0077404 [Crossref] [ Google Scholar]
34. Sumner A, Harpham T. The market for ‘evidence’ in policy processes: the case of child health policy in Andhra Pradesh, India and Viet Nam. Eur J Dev Res 2008; 20(4):712-732. doi: 10.1080/09578810802493358 [Crossref] [ Google Scholar]