A Process for Evaluating Quality Decision-Making Practices During the Development, Review and Reimbursement of Medicines

Background

There has been increasing interest in research on decision-making over the past 2 decades and indeed the science of decision-making is well-established. ^1-4 It would therefore seem of value to explore whether the various findings and recommendations from this research have been incorporated into operations when decisions are made under conditions of uncertainty. Nevertheless, there is still a limited body of research regarding the evaluation of both the quality and transparency of decision-making through which medicines become available as well as how well the practices are aligned within and across organisations. Indeed, it is not always clear what explicit processes pharmaceutical companies, regulatory authorities and health technology assessment (HTA) agencies are using to make key strategic decisions. It is nevertheless important that there is better transparency around this, in order to ensure trust by patients and public, as well as across stakeholders in knowing that their decision-making processes are in line with best practice. This is particularly important with the growing trend and efforts to align regulatory and HTA standards and processes both within companies as well as across agencies, in order to further increase effectiveness and efficiency during the development, review and reimbursement of medicines. ^5,6

This gap in research into decision-making during medicines development was initially addressed through the research of Donelan and colleagues, which resulted in defining the best practices in decision-making during the lifecycle of medicines, namely the 10 Quality Decision-Making Practices (QDMPs) (Figure 1). Donelan et al also developed an instrument, the Quality of Decision-Making Orientation Scheme (QoDoS), which can be used to measure the incorporation of these 10 QDMPs into the operational processes of companies and agencies. ^7,8 The development of QoDoS has been underpinned by the well-established science of decision-making and it provides an assessment regarding its implementation into decision-making processes undertaken by individuals and organisations. Indeed, one way for organisations to determine how to improve their decision-making is first to analyse the implementation status of these best practices into their frameworks, which can be achieved with QoDoS. It should be noted however that QoDoS does not aim to assess the outcome but the focus is on the process through which decisions are made. Although a good process does not always guarantee a favourable outcome, the incorporation of best practices and having structured processes should increase the probability of more predictable, transparent and improved outcomes. In addition, it should ensure better trust and efficiency among teams and committees that are making such decisions, as well as across organisations particularly in the case of agency work-sharing or parallel HTA-regulatory approaches that are becoming increasingly more important.

Close

Figure 1. The 10 Quality Decision-Making Practices.

A systematic review of the literature demonstrated QoDoS to be the most promising available technique for such assessments. ⁹ Moreover, a pilot study with participants from pharmaceutical companies and regulatory authorities demonstrated the initial practicality of the QoDoS in assessing the strengths and weaknesses as well as similarities and differences in decision-making practices amongst individuals as well as their perception of their organisation. ¹⁰

Furthermore, a recent study demonstrated the reliability and relevance of QoDoS in the decision-making of pharmaceutical companies, regulatory authorities and HTA agencies. ¹¹ These 2 properties are crucial to any future applications of the instrument, particularly in longitudinal studies, to ensure that potential changes in decision-making practices are a result of modified organisational processes rather than due to measurement error. ¹²

In order to determine the value of the methodology in assessing the level of implementation of the 10 QDMPs by the 3 types of organisations, case studies were initiated with participants from specific teams, committees, or departments across companies, regulatory and HTA agencies. This study was used to determine the factors that influence decision-making within organisations, including favourable practices and those that might require improvement, as well as to identify common themes in quality decision-making across different organisations to determine whether there is good alignment in the use of best practices in decision-making. The aim was therefore to evaluate the practicality of QoDoS in identifying the incorporation of the 10 QDMPs through illustrative case studies involving a pharmaceutical company, a regulatory authority and an HTA agency. The objectives were to:

• Evaluate the quality of the decision-making practices of the individuals and their perception of their respective organisation’s decision-making;

• Identify the favourable and unfavourable QDMPs;

• Assess the consistency of the QDMPs within each organisation including different decision-making groups as well as across organisations;

• Evaluate the feasibility and the perceived benefits of the study methodology based on initial feedback discussions as well as lessons learned.

Methods

Design of the Study

This study was designed to examine the value of the QoDoS and the implementation of the 10 QDMPs in 3 case studies, each reflecting the decision-making environment of the respective stakeholders; that is, a pharmaceutical company, regulatory authority and HTA agency. The participants completed the QoDoS instrument (see Supplementary file 1) and each study was planned as a cross-sectional design, where the data were collected at one point in time. The purpose of each case study was to examine the organisation as a whole and to assess the QDMPs within a number of logical sub-units (eg, departments) within each organisation.

Assessment Technique

The 47-item QoDoS instrument (Supplementary file 1) consists of 2 parts: Part I relates to the organisation, comprising 2 domains (“Approach” and “Culture”) and Part II relates to the individual, grouped into 2 domains (“Competence” and “Style”), with each item to be rated on a 5-point Likert scale, where the following distinctions were made: Not at all = 0% of time; Sometimes = 25% of time; Frequently = 50% of time; Often = 75% of time; Always = 100% of time.

In addition, 4 demographic questions were used to collect data on gender, job title, professional experience and organisation type. The QoDoS items were rated as either favourable or unfavourable and based on this, the Likert scale response options were quantified by assigning scores. For QoDoS items considered as favourable practice, the following scores were assigned: Not at all = 0; Sometimes = 1; Frequently = 2; Often = 3; and Always = 4. For QoDoS items considered as unfavourable practice, the reverse scores were assigned: Not at all = 4; Sometimes = 3; Frequently = 2; Often = 1; and Always = 0 (Table 1).

Study Participants

Selection Process and Inclusion Criteria

Using purposive sampling, 12 organisations were selected from across the participants in the Centre for Innovation in Regulatory Science research programmes. ¹³ These include both companies (multinational companies with a global footprint) as well as regulatory and HTA agencies (mature and emerging agencies). Three companies, 3 regulatory and 3 HTA agencies were selected with the aim to identify one from each of the 3 different organisations. Positive responses were received from 2 pharmaceutical companies, 2 regulatory authorities and one HTA agency but only one of each organisation types were ultimately selected. This sampling process was considered appropriate as the aim was to produce illustrative case studies as opposed to generating company, regulatory authority, or HTA agency aggregated trends. The criteria for selecting the initial 12 organisations (first stage) and subsequently the 3 organisations for the case studies (second stage) are described below.

First stage inclusion criteria – The selected companies had research and development expenditures in excess of 1 billion USD, ¹⁴ thereby reflecting their innovativeness. The 4 regulatory authorities were all classified as stringent regulatory authorities as defined by the World Health Organization (WHO). ¹⁵ Lastly, to ensure that the organisations involved in these studies have well-established decision-making systems in place the 4 HTA agencies were selected based on size and maturity from those organisations that are part of the International Network of Agencies for Health Technology Assessment ¹⁶ or European Network for Health Technology Assessment. ¹⁷

Second stage inclusion criteria – The 3 organisations were selected for the case studies based on availability of resources in terms of staff and time as well as on cohort size. This would enable calculation of the response variance as well as sub-group analysis on the dataset to identify differences and similarities and the consistency of implementation of the 10 QDMPs in an organisation.

Selected Study Participants

The selected pharmaceutical company cohort consisted of 31 individuals from the regulatory affairs leadership team (LT) across the company who were responsible for ensuring the regulatory, quality and safety aspects of medicines for the purpose of submission of a new medicine to a regulatory authority; and 3 sub-teams (STs) who were responsible for compiling the data for the leadership committee; 2 of whom focus on medicines’ regulatory aspects and one on their safety.

For the selected regulatory agency, the 47 participants were all assessors who were engaged in reviewing the regulatory dossier prior to the granting of the marketing authorization of a medicine (pre-market assessors) as well as assessors who re-evaluate the marketing authorization status based on new information (post-market assessors). The 28 HTA agency participants were members of the appraisal committee, which recommends whether or not a new medicine should be reimbursed under the national health system. The members were all external experts not directly employed by the agency. As many decisions are made on a daily basis, each organisation selected a key strategic decision-making process of interest (Table 2).

Study Procedure

Each study followed the same format to ensure consistency, including introduction to the study through a seminar, followed by completion of the QoDoS. Initial feedback discussions were organized with the cohort leaders in order to review the relevance and clarity of the results as well as their initial perception of the value and benefits of the study methodology. The 3 case studies were completed by February 2018.

Data Processing and Analysis

Information was processed into an Excel database for the completed questionnaires where descriptive statistics were used to analyse the dataset. The scores for the individual QoDoS items were codified based on the categorization in Table 1. This was then used to calculate the overall score for each QDMP by taking a median across the relevant QoDoS item scores. Due to confidentiality reasons, only aggregated results are shown here and no data that identify an individual or a specific organisation are reported. No statistical tests were planned or conducted, as this study was designed to be illustrative, with an aim of providing an assessment of the objectives as well as to generate hypotheses for further research.

The median scores were presented in the form of spider plots and the following traffic light colour coding was used according to the overall QDMP scores, where score <1 = “unfavourable practice” = red; score >1 and <3 = “needs improvement” = yellow; and score >3 = “favourable practice” = green.

Results

This evaluation focused on assessing the implementation of the 10 QDMPs using the QoDoS through 3 cross-sectional case studies with a pharmaceutical company, a regulatory authority and an HTA agency. For the purpose of clarity, the results are presented in 3 parts:

• Part I – Pharmaceutical company

• Part II – Regulatory authority

• Part III – HTA agency

Characteristics of the Study Participants

The 3 QoDoS case studies were completed by a total of 31 individuals from the 4 teams within pharmaceutical companies, 40 individuals from the regulatory authority and 25 from the HTA agency. The response rate was 100% for the company (31 individuals), 85% for the regulatory authority (25/32 pre-markets assessors; 15/15 post-markets assessors) and 89% for the HTA agency (25/28 individuals) (Table 3).

Organisational Feedback Discussions

Informal feedback discussions were organized with the group leaders from each study to discuss the results. These confirmed the feasibility of the study method as well as initial benefits of the approach, including raising awareness of biases and best practices in decision-making, gaining a basis for discussion of the issues in decision-making and for making recommendations for improving the lowest scoring or least consistent practices. The discussions also helped to uncover the rationale for some differences in responses across sub-groups or when comparing individual and organisational perception and these were incorporated into the relevant results section for the company (part I), regulatory authority (part II) and the HTA agency (part III).

Part I – Pharmaceutical Company

Assessment of Individual Practices

The individual-level QDMPs of the company participants were generally favourable, namely how the individuals within the 3 STs perceived their decision-making process for presenting an emerging risk to a regulatory authority as well as how the individuals within the LT perceived their process for submitting a new drug application to a regulatory authority (Figure 2A).

Close

Figure 2. Ten QDMPs for pharmaceutical company LT and the 3 STs (A) Individual’s decision-making (QoDoS Part 2); (B) Perception of Organisation’s Decision-Making (QoDoS Part 1). Abbreviations: QDMPs, Quality Decision-Making Practices; QoDoS, Quality of Decision-Making Orientation Scheme; LT, leadership team; ST, Sub-team.

An analysis of the QDMP median scores for the 3 STs revealed that the individual practices were identical and generally favourable for ST1 and ST2, thereby suggesting more consistency across these 2 regulatory STs compared with the safety ST (ST3). One practice that was not favourable for STs 1 and 2 was QDMP 3 (assign values and relative importance to decision criteria), which received a median score of 2. In contrast, the practices for ST3 were generally favourable and differed from STs 1 and 2 regarding QDMP 3 (criteria) and 6 (consider uncertainty) with higher scores for ST3 compared with STs 1 and 2, whereas QDMP 9 (ensure transparency and provide a record trail) received a lower score for the safety group compared with the 2 regulatory STs.

The main differences between the practices of the 3 STs and the LT were in QDMP 2 (assign clear roles and responsibilities), where the practices of the LT were slightly less favourable, although both generally were good. Similarly, QDMP 5 (examine alternative solutions), where the LT scored in the area of “needing improvement” and QDMP 9 (transparency) was also less favourable for the LT compared with the 3 STs. A potential explanation received from the company during a feedback study session was that this may be due to the fact that the STs have clear remits regarding their roles and responsibilities for assembling the information to the leadership committee, including the decisions made by the individual team members, whereas the LT makes decisions as a group and therefore the group dynamics may result in a perception that there is less transparency around the decision-making process of the LT members, including how alternatives are considered.

The variance in the scores was also explored (Supplementary file 2) and demonstrated that despite some differences in medians, the overlap between the scores was considerable, for example for QDMP 2 (roles) where the median score was between 3 and 4 for the 3 STs, whereas the 25th-75th median range was also between 3 and 4, demonstrating that generally the individual practices are relatively consistent and favourable. On the other hand, some differences in variance exist for practices with the same median, for example QDMP 1 (have a systematic, structured approach to aid decision-making), where the 25th-75th box was in the area of ‘favourable practice’ for ST3, but suggests a potential need for improvement for the LT and ST2. The practice with the most variance across the group was QDMP 3 (criteria), indicating that this practice is not consistently applied across the individuals within the teams.

Assessment of Organisational Practices

The individuals from the 3 STs also evaluated their perception of the decision-making of the LT for submitting a new drug application to a regulatory authority, which was compared to the results of the LT assessing their own decision-making for that same decision-making process (Figure 2B). The 3 STs perceived a number of practices of the LT as favourable, namely QDMP 6 (consider uncertainty), QDMP 7 (re-evaluate as new information becomes available), QDMP 8 (perform impact analysis of the decision) and QDMP 10 (effectively communicate the basis of the decision). On the other hand, there were some differences in the median scores between the 3 STs relating to how they each perceived the LT for QDMP 1 (have a systematic, structured approach to aid decision-making), QDMP 2 (assign clear roles and responsibilities), QDMP 3 (assign values and relative importance to decision criteria), QDMP 4 (evaluate both internal and external influences/biases), QDMP 5 (examine alternative solutions) and 9 (ensure transparency and provide a record trail).

The results from the 3 STs were generally similar to those obtained directly from the LT, where one of the main differences was regarding QDMP 10 (effectively communicate the basis of the decision). For QDMP 10, the 3 STs generally agreed that the LT had clear communication practices in place, whereas the LT perceived this practice as “needing improvement.” A potential explanation received during the initial feedback session was that this may be a result of efficient and clear communication of the LT’s decisions by the ST managers to their direct reports in STs 1, 2 and 3; this was highlighted as key to informing the day-to-day activities of the 3 STs. As with the assessment of the individual practices, the perceived incorporation of the 10 QDMPs into the LT’s decision-making was also characterised by considerable variance, for example for QDMP 1 (structure), 2 (roles), 3 (criteria) and 4 (bias) (Supplementary file 2).

Part II – Regulatory Authority

Assessment of Individual Practices

An analysis of the decision-making practices across the 40 assessors indicated that both pre-market and post-market assessors perceive their practices as generally “favourable” across all 10 QDMPs (Figure 3A). The only area needing improvement was QDMP 5 (examine alternative solutions) for the post-market assessors. Overall, the practices were consistent between pre- and post-marketing staff; however, some differences in median scores were observed for QDMP 1 (have a systematic, structured approach to aid decision-making) nevertheless the practice was favourable for both. Other differences were observed for QDMP 4 (evaluate both internal and external influences/biases), QDMP 5 (examine alternative solutions) and QDMP 6 (consider uncertainty). Initial discussion with the agency based on their feedback indicated that the rationale for this response was that the decision of the regulator can be binary (to approve or reject) although in other situations there may be variations according to the type of indication, dose, or patient population. In addition, the variance around the median for each of those QDMP (Supplementary file 2) suggests that there are considerable differences around certain practices. For example, despite QDMP 3 (assign values and relative importance to decision criteria) having a median corresponding to ‘favourable practice,’ the variance (25th-75th percentile) was in the area of ‘needing improvement,’ particularly for pre-market assessors, indicating that this practice should be further explored.

Close

Figure 3. Ten QDMPs for Regulatory Authority Pre- and Post-market Assessors (A) Individual’s Decision-Making (QoDoS Part 2); (B) Perception of Organisation’s Decision-Making (QoDoS Part 1). Abbreviations: QDMPs, Quality Decision-Making Practices; QoDoS, Quality of Decision-Making Orientation Scheme.

Assessment of Organisational Practices

The perception of organisational QDMPs by the assessors was also in the area of “favourable practice” and the QDMP scores were similar for pre- and post-market assessors (Figure 3B). However, there was more consistency between responses from pre- and post-market assessors in their perception of their agency (Figure 3B) compared with their own decision-making (Figure 3A). The differences between pre- and post-market assessors (Figure 3B) were for QDMP 2 (assign clear roles and responsibilities), where pre-market responses resulted in “favourable practice” compared with “needing improvement” for post-market staff. In addition, QDMP 8 (perform impact analysis of the decision), was rated by pre-market assessors as “unfavourable practice” and by post-market assessors as “needing improvement.” Based on feedback discussions with the agency, the rationale for this score was that the analysis of impact of decision-making during the regulatory review is not seen as one of the roles of a regulatory agency and therefore may not be applicable.

Finally, there was also considerable variance for a number of the QDMPs, such as QDMP 3 (criteria) and QDMP 5 (alternatives) for pre-market assessors and QDMP 4 (bias) for post-market assesses (Supplementary file 2).

Part III – HTA Agency

Assessment of Individual and Organisational Practices

The combined results for the 25 individuals from the HTA agency appraisal committee showed that the committee members perceived their own decision-making practices as generally “favourable” as well as that of the organisation (Figure 4). The practices that were “unfavourable” or “needing improvement” were QDMP 3 (assign values and relative importance to decision criteria), QDMP 5 (examine alternative solutions) and QDMP 8 (perform impact analysis of the decision), which is similar to the regulatory authority. Minor differences were identified between how the individuals make the decision and how they view the organisation; for example, in QDMP 3, where the assessment of the decision criteria was favourable for the organisation but less so for the individual. The score of the individual QDMP 8 regarding impact analysis suggests an unfavourable practice, which was also the case for the regulatory agency. A number of QDMPs were characterised by considerable variance in the ‘needing improvement’ area, for example QDMP 2 (assign clear roles and responsibilities) and QDMP 7 (re-evaluate as new information becomes available) regarding the individuals’ perception of the organisation’s decision-making (Supplementary file 2).

Close

Figure 4. Ten QDMPs for Individuals’ and Organisation’s Decision-Making (QoDoS Parts 2 and 1, Respectively) for HTA Appraisal Committee Members. Abbreviations: QDMPs, Quality Decision-Making Practices; QoDoS, Quality of Decision-Making Orientation Scheme; HTA, health technology assessment.

Discussion

Advancing the Field of Quality Decision-Making

Although there is an increasing use of framework-based quantitative modelling to aid decision-making within the lifecycle of medicines, the various decision-making processes are subjective and often qualitative in nature. Therefore, these should be further explored to identify areas of best practice as well as those that may need improvement to increase process quality, consistency and transparency. ¹⁸ Although, the well-established principles of good decision-making are common sense, they are nevertheless not always common practice. This applies to both how individuals decisions on a daily basis, as well as how key strategic decisions are made across a range of organisations including the pharmaceutical industry. ⁶ A recent review of the literature identified general paucity of research in this area, particularly regarding the development and systematic application of techniques for evaluating quality decision-making, such as through the studies undertaken as part of this research. ⁹ Indeed this is the first assessment to evaluate quality of decision-making within the different organisations involved in the development, review and reimbursement of medicines.

Overall, these 3 studies demonstrated the practicality of QoDoS to identify favourable and unfavourable practices as well as to assess their consistency and transparency within as well as across the organisations. Importantly, this study was the first implementation of QoDoS in an in-depth organisational setting and the results confirmed the initial feasibility of the proposed method. All 3 case studies demonstrated generally favourable results across the QDMPs, where overall, all 3 organisations have incorporated the majority of the 10 QDMPs. This may not be surprising, as the organisations that were selected for the 3 studies have established decision-making systems based on their size and multinational status (for the company) and maturity level (in case of agencies).

Application of QoDoS in the Development and Review of Medicines

The 3 case studies differed in terms of cohort type and decision point in order to demonstrate the different ways in which QoDoS can be implemented. The pharmaceutical company study was used to assess the decision-making across 4 groups (3 STs and one LT), where the QoDoS questions relating to the perception of the organisation were used to assess the LT based on the LT’s perception as well as that of the 3 STs. In this case, the decision-making of the 4 teams was generally very similar, demonstrating good incorporation of QDMPs across the organisation.

Second, the regulatory authority study was used to demonstrate differences in individual practices and perception of the organisation by 2 different reviewer groups (pre- and post-market approval). In this case, the QoDoS responses demonstrated that the perception of the organisation is relatively consistent for the 2 groups and certain differences were identified in how the individuals make decisions, which may be due to the different processes in place for the pre- and post-marketing activity.

Lastly, the HTA case study involved one group, the appraisal committee, where the individuals assessed themselves and their organisation for the same decision point, where the results demonstrated general consistency.

Despite the fact that the scope and the decision-making processes within each organisation are different, it could be argued that each organisation should be implementing the same practices to promote quality and consistency. Although, in general, the outcomes of the 3 cohorts were very positive in demonstrating alignment in terms of the incorporation of the QDMPs, the QoDoS demonstrated the need for improvement across a number of practices, where some similarities were identified, such as the need for better assignment of values and relative importance of decision criteria (QDMP 3) as well as the evaluation of alternatives (QDMP 5) for the 3 organisations and the need for impact analysis by the individuals within the regulatory and the HTA agency appraisal committee (QDMP 8).

Interestingly, both QDMP 3 and QDMP 8 were seen as generally not incorporated into company and agency frameworks during medicines regulatory submission, review, and HTA processes. ¹⁹ As these practices were nevertheless seen as relevant by the respondents, QDMP 3 should be addressed through the incorporation of more formal frameworks, such as a benefit-risk framework during regulatory decision-making, ²⁰ and through having standardized evidence criteria for HTA. ²¹

Although both the regulatory agency and the HTA agency felt that impact analysis may be out of scope for the agency review and appraisal, this may be due to the interpretation of what impact analysis entails and in fact, may already be an intrinsic part of the decision-making process. Indeed, impact analysis (QDMP 8); such as an analysis of the impact on patients, society and other agencies, could still be considered as the remit of agencies. The practice could be incorporated within frameworks in 2 ways; first, through an assessment of linked decisions such as relevant precedents, including decisions previously made by the organisation or other relevant stakeholders. ¹ Second, such an assessment should focus on the impact of this decision on present and future processes and should address relevant stakeholders including patients. ²² For example, a pre-marketing authorization regulatory impact analysis would enable assessment of how similar medicines were reviewed (including outcomes) within their own jurisdiction and by other regulators; in addition, the analysis would facilitate understanding of the impact of the decision on other processes, such as HTA, as well as the ultimate effect of the decision (including the approved label) on patients.

Another key finding was the general variance (25th-75th percentile) around the responses obtained from the 3 groups for the incorporation of the 10 QDMPs. It is important to note that variation is not perceived as a shortcoming, as QoDoS assesses a process that is subjective in nature and aims to capture differences in perceptions that can then be explored through feedback discussions. Differences in scores may be a result of a mixture of factors: inconsistencies in individual practices and differences in the perception of the organisation due to poor transparency and documentation of the practices or different experiences within the organisation.

Next Steps for Improving the Practices of Individuals and Organisations

The initial discussions with the cohort leaders gave an early indication of the benefits of the implementation of the QoDoS; for example, having a basis to identify issues in decision-making in order to improve practices. The 3 organisations should now determine the best way forward to change any decision-making practices that have been shown to be inconsistent or to need improvement.

It would also be of interest to initiate similar studies with additional teams and groups from each of the 3 organisations to assess consistency of QDMPs eg, discovery, pre-clinical and clinical drug development in companies; as well as other committees, reviewers and project managers within agencies. It would also be of value to involve other organisations, including smaller companies or agencies from emerging markets with less established systems, to determine how QDMPs are built into those organisations, compared with larger companies and agencies. ²³ This could in turn help improve the productivity of companies and ensure agencies are not only undertaking a good-quality review or appraisal but are also making sure their decisions are consistent and transparent.

Limitations

Only one company, regulatory authority and HTA agency were selected to demonstrate the practicality of QoDoS for identifying the incorporation of the 10 QDMPs and generally to demonstrate the feasibility of the method. This sampling was considered nevertheless appropriate as the aim was to produce illustrative case studies for testing the practical application of the methodology, as opposed to generating aggregated trends or extrapolating the results to other organisations. Furthermore, the feedback discussions with the study participants were so far limited and it would be of interest to determine what changes were implemented based on the results and what impact they had on the decision-making practices going forward if a QoDoS study could be repeated in the future. Finally, other appropriate tools may exist to undertake such assessments, particularly if they were not published.

Conclusion

QoDoS can be applied as a diagnostic instrument within organisations for the routine assessment of quality of decision-making. Importantly, the data obtained from QoDoS studies could provide a basis for internal dialogue (within an organisation) and to ultimately address the least favourable practices by incorporating them more effectively into decision-making frameworks. The publication of such results may also help minimise reputational risk of an organisation, by demonstrating that best practices are in place (or improvements are being made) and therefore to increase trust in the process. Such trust is key both from patients to understand the decision-making process, as well as from other stakeholders particularly in the case of agencies aligning their decision-making processes. ^5,6

In addition, the routine assessment of the QDMPs has the ability to measure change over time in order to determine the impact of any improvement initiatives. The ongoing use of quality frameworks for making decisions will also reduce uncertainty around decision-making and might result in more predictable and favourable outcomes. Finally, QoDoS can be used to externally benchmark an organisation’s decision-making practices with other organisations. This in turn could be used to promote best practices as well as to build alignment into key strategic decisions made by organisations during medicines development, regulatory review and reimbursement.

Acknowledgements

The authors wish to thank the individuals from the pharmaceutical company, the regulatory authority and the health technology assessment agency who took part in the case studies and facilitated timely completion of the work.

Ethical issues

The study protocol received approval from the University of Hertfordshire Institutional Ethics Committee. Since the study participants were neither National Health Service patients or staff, it did not require the Local Ethics Committee approval. The eligible participants received a copy of the “Participant Information Sheet” describing the purpose of the study, reasons for their participation and explaining that only aggregated results will be reported, before they decide to agree or decline to take part in the study. Therefore, their agreement to participate in the study constituted consent.

Competing interests

Authors declare that they have no competing interests.

Authors’ contributions

MB designed the study, analyzed the data, and wrote the manuscript. NM designed the study and wrote the manuscript. SW designed the study and wrote the manuscript. SS designed the study and wrote the manuscript.

Authors’ affiliations

¹Department of Clinical and Pharmaceutical Sciences, School of Life & Medical Sciences, University of Hertfordshire, Hatfield, UK. ²Centre for Innovation in Regulatory Science (CIRS), London, UK.

Supplementary files

References

1. Hammond JS, Keeney RL, Raiffa H. Smart Choices: A Practical Guide to Making Better Decisions. New York: Harvard Business School; 1999.
2. Thaler RH, Sunstein CR. Nudge: Improving Decisions About Health, Wealth and Happiness. New Haven: Yale University Press; 2009.
3. Kahneman D. Thinking Fast and Slow. New York: Farrar, Straus and Giroux; 2011.
4. Spetzler C, Winter H, Meyer J. Decision Quality: Value Creation from Better Business Decisions. New York: Wiley; 2016.
5. Wang T, McAuslane N, Liberti L, Leufkens H, Hövels A. Building synergy between regulatory and HTA agencies beyond processes and procedures-can we effectively align the evidentiary requirements? a survey of stakeholder perceptions. Value Health 2018; 21(6):707-714. doi: 10.1016/j.jval.2017.11.003 [Crossref] [ Google Scholar]
6. McAuslane N, Liberti L, Connelly P. The confluence of accelerated regulatory and health technology assessment access pathways. Clin Pharmacol Ther 2019; 105(4):935-942. doi: 10.1002/cpt.1315 [Crossref] [ Google Scholar]
7. Donelan R, Walker S, Salek S. The development and validation of a generic instrument, QoDoS, for assessing the quality of decision making. Front Pharmacol 2016; 7:180. doi: 10.3389/fphar.2016.00180 [Crossref] [ Google Scholar]
8. Donelan R, Walker S, Salek S. Factors influencing quality decision-making: regulatory and pharmaceutical industry perspectives. Pharmacoepidemiol Drug Saf 2015; 24(3):319-328. doi: 10.1002/pds.3752 [Crossref] [ Google Scholar]
9. Bujar M, McAuslane N, Walker SR, Salek S. Evaluating quality of decision-making processes in medicines’ development, regulatory review, and health technology assessment: a systematic review of the literature. Front Pharmacol 2017; 8:189. doi: 10.3389/fphar.2017.00189 [Crossref] [ Google Scholar]
10. Bujar M, Donelan R, McAuslane N, Walker S, Salek S. Assessing the quality of decision making in the development and regulatory review of medicines: identifying biases and best practices. Ther Innov Regul Sci 2017; 51(2):250-256. doi: 10.1177/2168479016662681 [Crossref] [ Google Scholar]
11. Bujar M, McAuslane N, Walker S, Salek S. The reliability and relevance of a quality of decision making instrument, Quality of Decision-Making Orientation Scheme (QoDoS), for use during the lifecycle of medicines. Front Pharmacol 2019; 10:17. doi: 10.3389/fphar.2019.00017 [Crossref] [ Google Scholar]
12. Streiner DL, Norman GR, Cairney J. Health Measurement Scales: A Practical Guide to Their Development and Use. 5th ed. Oxford: Oxford University Press; 2015.
13. Centre for Innovation in Regulatory Science (CIRS). 2019 Agenda. http://www.cirsci.org/wp-content/uploads/2019/07/CIRS-2019-8-Page-agenda_8July_high-res.pdf. Accessed September 2, 2019.
14. Pharmaceutical Executive (PharmExec). Pharm Excec’s Pharma 50 companies 2017. http://www.pharmexec.com/pharm-execs-top-50-companies-2017. Accessed May 7, 2018. Published 2017.
15. World Health Organisation (WHO). Clarification with Respect to a Stringent Regulatory Organisation as Applicable to the Stringent Regulatory Authority (SRA) Guideline. https://extranet.who.int/prequal/sites/default/files/documents/75%20SRA%20clarification_February2017_0.pdf. Accessed May 7, 2018. Published 2017.
16. International Network of Agencies for Health Technology Assessment (INAHTA). Members List; 2018. http://www.inahta.org/members/members_list/. Accessed May 7, 2018.
17. European Network for Health Technology Assessment (EUnetHTA). Our Organisation; 2018. https://www.eunethta.eu/about-eunethta/organisation/. Accessed May 7, 2018. Published 2018.
18. Walker S, McAuslane N, Liberti L, Connelly P. Centre for Innovation in Regulatory Science Workshop report: Building the benefit-risk toolbox: Are there enough common elements across the different methodologies to enable a consensus on a scientifically acceptable framework for making benefit-risk decisions?. http://www.cirsci.org/sites/default/files/CIRS%20June%202012%20Workshop%20Report%20low-res.pdf. Accessed August 27, 2019. Published 2012.
19. Walker S, McAuslane N, Bujar M, Connelly P, Liberti L. Quality decision-making practices: Their application and impact in the development, review and reimbursement of medicines. Netherlands: Off Page; 2018.
20. Leong J, McAuslane N, Walker S, Salek S. Is there a need for a universal benefit-risk assessment framework for medicines? Regulatory and industry perspectives. Pharmacoepidemiol Drug Saf 2013; 22(9):1004-1012. doi: 10.1002/pds.3464 [Crossref] [ Google Scholar]
21. European Network for Health Technology Assessment (EUnetHTA). HTA Core Model; 2016. https://meka.thl.fi/htacore/ViewHandbook.aspx. Accessed May 7, 2018.
22. McAuslane N, Patel P, Liberti L, Connelly P. Centre for Innovation in Regulatory Science Workshop report: Evolving the Regulatory Review Process: What are the features that enable a transparent, timely, predictable and good-quality review? http://cirsci.org/publications/CIRS_December_2011_WS_Report.pdf. Accessed May 7, 2019. Published 2011.
23. Smietana K, Ekstrom L, Jeffery B, Møller M. Improving R&D productivity. Nat Rev Drug Discov 2015; 14(7):455-456. doi: 10.1038/nrd4650 [Crossref] [ Google Scholar]