Establishment and Application of an Index System for the Risk of Drug Shortages in China: Based on Delphi Method and Analytic Hierarchy Process

Background

The World Health Organization (WHO) identified that drug shortages have become a complex global challenge and have a severe impact on the healthcare system.^1-3 Several factors are contributing to drug shortages, including manufacturing and quality problems, insufficient production capacity, manufacturer business decisions, active pharmaceutical ingredient or raw material shortages or monopolies, and fluctuating drug demand.^4,5 Moreover, emergency public health crises, such as African swine fever and the coronavirus disease 2019 (COVID-19)⁵ can also pose a challenge to the global drug supply and patients’ access to therapies.

Drug shortages can affect clinical outcomes, lead to medical errors, endanger the health of patients, and place a heavy burden on healthcare, seriously affecting the safety, efficacy, timeliness, and cost-effectiveness of medication.^6,7 According to the American Society of Health-System Pharmacists, there were 205 drugs affected by shortages in 2020.⁸ Among these, 129 were new shortages often linked to COVID-19-related drugs such as hydroxychloroquine sulfate.⁹ In China, shortages of 370 drugs were reported in 2020 in Hunan province, according to the Hunan Drug Shortage Surveillance and Early Warning Center.¹⁰ A study in the United States reported that $209 million in additional costs were associated with the use of more expensive alternatives.⁴ Indeed, our research team found that the shortage of pyridostigmine bromide in Hunan province forced patients to buy drugs out of town and significantly increased indirect costs such as transportation, accommodation, and missed work, all of which accounted for roughly 80% of the total cost, further increased economic burden, and reduced treatment effect and compliance among patients.

In recent years, China has attached great importance to addressing drug shortages and paying more heed to the early warnings of drug shortage risk than to post-emergency management.¹¹ A series of effective measures have been proposed to alleviate and solve drug shortages, such as establishing and improving the three-level (province, city, county) drug use surveillance and early warning network, issuing guidelines on therapeutic alternatives, implementing a management list of drugs in shortage, and providing policy support to the list of drugs such as (1) manufacturers can set their prices for drugs in shortage, (2) medical institutions can purchase drugs in shortage offline without using provincial procurement platforms, and (3) the distribution of drugs in shortage is exempt from the two invoice policy.^12-14 Several provinces, such as Yunnan, Ningxia, Shanxi, Guizhou, Guangxi, and Hainan have successively released a list of drugs in shortage since 2019.^15-20 Recently, a national list of drug shortages and a national list monitoring clinically essential drugs vulnerable to shortages were also released for the first time.²¹

There are hundreds of drugs experiencing varying degrees of shortage each year. This shortage varies greatly across regions and levels of healthcare. In the context of limited public health resources, how to scientifically and effectively identify and evaluate drugs experiencing genuine shortages is critical to solving drug shortages. However, as the government has not yet issued relevant methodological guidance, the evaluation of drug shortages in China mainly relies on experts’ subjective experience, calling into question whether the results are scientific, reliable, and representative. Therefore, it is necessary to establish a quantifiable evaluation index for assessing the risk of drug shortages.

Using a modified Delphi method and the analytic hierarchy process (AHP), this study established an index system for evaluating the risk of drug shortages in China and applied it to guide the graded management of drugs in shortage in medical institutions.

Methods

A modified Delphi method was chosen by consensus among experts and was developed through a two-round Delphi consultation. We use the AHP model to compare and rank indexes. The duration of the study was 10 months. Index selection was started in May 2020, two-round of Delphi consultation were completed in July 2020, indicator system construction was completed in October 2020, and comprehensive evaluation of shortage drugs in 2020 was completed in February 2021. A study flow diagram is shown in Figure 1.

Close

Figure 1. Flow Diagram of the Study Process. Abbreviation: AHP, analytic hierarchy process.

Construct the Initial Assessment Indexes

According to the modified Delphi method, we assembled a research group including three experts specializing in the fields of pharmacy and public health and responsible for developing the initial indexes of evaluation of drug shortage risk. The initial indexes were developed based on literature,^5,22 documents issued by the State Council,²³ the China National Health Commission,^13,14 and research group opinion.

Inclusion Criteria of Experts

To make the group decision more reliable, 21 experts from five provinces or municipalities in China were selected for the Delphi survey. No data related to the medical institution of the expert is involved in this study, the consent of the medical institution is not required. The inclusion criteria were as follows: (1) more than 10 years of working experience in a medical institution (clinical pharmacy, pharmacy administration, or clinical medicine) or distribution enterprise (drug management), (2) deputy senior or above the position of professional or business administrator, and (3) voluntary participation in this survey and sufficient time and energy to complete the entire process.

Delphi Process

We conducted a two-round Delphi survey following the guidelines of Conducting and Reporting Delphi Studies.²⁴ A group of experts arrived at a consensus after two rounds of the Delphi process. Each round of the questionnaire was delivered and received by email, and respondents were asked to complete questionnaires within two weeks. Experts who did not complete a round of the survey were not invited to subsequent rounds. Experts were required to rate the rationality and importance of the indicators in each round with ‘agree,’ ‘modified,’ or ‘disagree.’ Experts were allowed to provide revision comments for each index. Importance was rated between 1 and 5 using a 5-point Likert scale: a score of 5 means extremely important, 4 means very important, 3 means important, 2 means generally important, and 1 means unimportant. Expert’s judgment criteria (Ca) and degree of familiarity with each indicator (Cs) were subsequently rated. The judgment criteria were based on four aspects: theoretical analysis, practical experience, reference literature at home and abroad, and intuitive feeling (see Table S1, Supplementary file 1). Experts’ familiarity with the index was categorized as follows: extremely familiar (1.00 point), very familiar (0.80 points), generally familiar (0.60 points), less familiar (0.40 points), and unfamiliar (0.20 points).

Inclusion Criteria Of Indexes

The coefficient of variation (CV) and the average value of the importance score were used to determine the index. The inclusion criteria of indexes based on the critical value method²⁵ were as follows: average value of the importance score ≥3.34 and CV ≤0.31 in the first round, and average value of the importance score ≥3.47 and CV ≤0.20 in the second round. Other indicators whose mean value or CV did not meet the inclusion criteria were adjusted or deleted based on expert opinion. The first-round questionnaire results were analyzed and fed back to respondents before the second round of consultation. Indexes not gaining consensus in the first round were repeated in the subsequent survey until consensus was reached and the index system was constructed.

Weight Determination of Each Indicator

The weight of each final indicator was rated by experts using the AHP after index determination.Hierarchical models including three levels were built using AHP to construct complex problems.²⁶ Then, a judgment matrix for pairwise comparisons was created using Satty’s fundamental scales of 1-9²⁷ (see Table S2, Supplementary file 1) to evaluate the relative importance of each index at the same hierarchical level. We calculated the initial weight and combination weight of each indicator and conducted consistency testing using yaahp12.4 (yaahp software, Taiyuan, Shanxi, China).

Statistical Analysis

All data were entered in Microsoft Excel 2013 and checked twice. SPSS 24 software was used for statistical analysis. Descriptive statistics were expressed as frequency, percentage, rate, mean, and standard deviation. The formula used for authority coefficient (Cr) calculation is: Cr = (Ca+Cs)/2, with Ca representing experts’ judgment criteria and Cs representing the degree of experts’ familiarity with each indicator. The degree of coordination of experts’ opinions is expressed by Kendall’s coefficients of concordance (Kendall’s W) and chi-square test, and the average random consistency index was used to test the satisfactory consistency of different judgment matrices using yaahp 12.4 (yaahp software, Taiyuan, Shanxi, China). Consistency ratio (CR)≤0.10 indicated that the judgment matrix had satisfactory consistency.²⁸ When CR >0.10, the judgment matrix required adjustment until CR ≤0.10.²⁸

Comprehensive Evaluation

We designed a rating scale of the drug shortage risk by converting the weight value of each three-level index into a score on a 100-point scale (see Table S3, Supplementary file 1). Data on drug shortages reported by medical institutions through the ‘Drug Classification and Procurement System of Hunan Province’²⁹ from January 1 to December 31, 2020 were collected. A total of 203 medical institutions in the province participated in the reporting process as monitoring posts. Based on the above data, we carried out a risk assessment of drug shortages in Hunan province. The risk score for each drug was calculated as the sum of the scores of each indicator. Total scores ranged from 0 to 100, with higher values indicating a higher risk of drug shortage.

Results

Characteristics of Experts

We invited 21 experts to participate in the Delphi process by email. In round one, 21 questionnaires were distributed with a recovery rate of 90% (19/21), while 79% (15/19) completed and returned the questionnaires in round two. Participants in the consultation had an average age of 47 years and were from five provinces or municipalities, including Hunan, Guangdong, Yunnan, Jiangsu, and Chongqing in China. Details of the demographic characteristics of the experts are summarized in Table S4 in Supplementary file 1.

Authority Coefficient of Experts

We calculated the authority coefficient of the experts based on self-evaluation scores. In the first round, the average authority coefficient of 19 experts was 0.88, increasing to 0.90 in the second round. In general, an authority coefficient of ≥0.70 indicates ‘highly credible.’³⁰

Degree of Coordination Among Experts

We evaluated the degree of coordination among experts with Kendall’s W. Generally, Kendall’s W is between 0 and 1; the greater the value, the higher the coordination degree of experts. In the first round of the Delphi process, Kendall’s W was 0.44 (χ² = 441.16, P= .00). In the second round, Kendall’s W was 0.49 (χ² = 370.44, P= .00). This indicated that the degree of coordination among experts in the second round was improved compared with the first round, and the degree of coordination among experts in both rounds was good.

Establishment of the Index System

The initial index framework, including five first-level, 11 second-level, and 38 third-level indexes, was developed based on literature, government documents, and research group opinion (see Table S5, Supplementary file 1).^5,7,14,22,31 Only indicators that met the inclusion criteria were included in the final index system. After the two-round Delphi process, eight third-level indicators were deleted. Four first-level indexes, five second-level indexes, and seven third-level indexes were modified. Eight new indexes were added, namely ‘Special classification’ (B2), ‘Emergency drugs’ (C3), ‘Detoxification drugs’ (C4), ‘Drugs for rare diseases’ (C5), ‘Other drugs’ (C6), ‘Number of manufacturers in province’ (B7), ‘Manufactured solely’ (C17), and ‘Number of manufacturers ≥2’ (C18). The final index system included four first-level indexes, 11 second-level indexes, and 36 third-level indexes (see Table 1).

Weight Determination of Each Indicator

The normalized weight and combination weight of each final indicator were calculated using AHP (see Table 2). A higher weight indicated that the indicator was more important for risk assessment. The CR of each modified judgment matrix was lower than 0.10, indicating that all of the judgment matrices had good consistency (see Table S6, Supplementary file 2). For first-level indicators, ‘supply stability’ (A2), ‘causes of shortage’ (A4), ‘medicine availability’ (A3), and ‘pharmaceutical properties’ (A1), the weight values were 0.3253, 0.2489, 0.2398, and 0.1860, respectively.

Evaluation of the Risk of Drug Shortage

A total of 370 drugs (453 dosage strengths) reported shortages from January 1 to December 31, 2020 in Hunan province. We excluded 67 drugs (70 dosage strengths) with a short supply time of less than one month and calculated the total risk scores of each remaining drug (383 dosage strengths). Based on the risk score, all drugs in shortage were classified as high risk (70-100 points), medium risk (40-69 points), or low risk (0-39 points). Among them, six drugs (dosage strength) were at high risk: sodium thiosulfate (0.64 g), posterior pituitary lobe hormones (1 mL:6 IU), protamine sulfate (5 mL:50 mg), thrombin (500 U), urokinase (10 WU), and rotundine sulfate (2 mL:60 mg) (see Table 3).

The characteristics of the drugs at different risk levels are shown in Table S7 in Supplementary file 2. Most drugs are at low shortage risk, and more than 90% of drug shortages were local (cities with short supply ≤5) and had alternatives. Only 3.66% of drugs with more than 10 medical institutions experienced short supply. Two findings were worth noting. First, more than half of the drugs in shortage (54.31%) were essential drugs. In China, drugs are deemed essentially when they are clinically necessary, safe, effective, and reasonable in price. As such, medical institutions should prioritize the use of essential drugs and focus on ensuring their supply. Second, half of the shortage (50.43%) was reported by primary medical institutions. This indicates that the shortage in primary medical institutions is more serious than that in secondary and tertiary hospitals.

Based on the risk assessment score, we built a decision tree for risk management strategies according to the features of drugs at different risk levels (see Figure 2). The risk assessment and graded management model has been approved by the Health Commission of Hunan Province for the monitoring, early warning, and selection process of drugs in shortage.

Close

Figure 2. The Graded Risk-Management Strategies for Drugs at Different Risk Levels.

Discussion

Drug shortages have become a global phenomenon. How to scientifically and effectively identify drugs experiencing genuine shortages is critical for solving this problem. Using the modified Delphi method combined with AHP, we constructed an index system with universal applicability and practicability for the risk assessment of drug shortages and applied the system to guide the graded response to drug shortage and differential management strategies to address these shortages.

The Delphi method can avoid the interplay between of experts’ mindset and the bias of ‘following the opinion of leaders,’ ensuring that the outcome is more objective and accurate than that of traditional group discussion.^32,33 The AHP objectifies subjective judgment by proposing a quantitative scale and enlarging the weight gap between indicators to refine the identification of key risk indicators.²⁶ The AHP is especially suitable for complex problems such as drug shortage risk assessments which are difficult to analyze with quantitative methods.^34,35 The Delphi method conducted before the AHP screens out the key risk indicators from a wide range of potential indexes and reduces the difficulty of statistical calculations that might be needed due to an excessive number of variables.²⁶

In this study, a three-level index system with four first-level indicators, 11 second-level indicators, and 36 third-level indicators was developed using a two-round Delphi process. The weight value of an indicator reflects its importance.³⁶ Of all first-level indicators, ‘supply stability’ (A2) was the most important. The most influential second-level indexes were ‘duration of short supply’ (B5) and ‘scope of short supply’ (B6). This suggests that the stability of drug supply is one of the most important factors for identifying and measuring the risk of drug shortage. ‘Causes of shortage’ (A4) was almost of equal importance to ‘drug accessibility’ (A3). In the above two first-level indicators, the highest scores are associated with ‘supply related causes’ (B10) and ‘number of medical institutions or distribution enterprises experiencing drug shortages’ (B8), respectively. This indicated that (1) it is necessary to understand the situation of production and raw material purchasing of drug manufacturers in a timely manner, and (2) adequate attention should be paid to the simultaneous reporting of shortages by multiple medical institutions or distribution enterprises. For the first-level indicator ‘pharmaceutical properties’ (A1), the most important second-level indicator subordinate was ‘alternatives availability’ (A2). This means that the availability of alternative medicines is important to address shortages. For drugs that can be substituted, experts should guide the selection of alternatives and assess the health risk of therapeutic substitution to ensure safe and rational drug use.¹³ For drugs with no alternatives, several measures such as strengthening drug stockpiles and optimizing distribution systems can be taken to ensure supply.

Based on the results of the risk evaluation of drug shortages, half of the shortage was reported by primary medical institutions. This may be attributed to the unsatisfactory implementation of the “hierarchical medical system” and the imperfect of “separation of dispensing from prescription (SDP)” in China to a certain extent. Despite the government’s advocacy of a “hierarchical medical system” in recent years, the majority of patients in China still prefer tertiary medical institutions due to the gap in the size, equipment, and treatment level of medical institutions at all levels. In China, although primary medical institutions account for the largest proportion among all types of medical institutions (94.61%), the proportion of patients attending primary healthcare institutions is not high (53%).³⁷ China has not yet realized the SDP,³⁸ and the gap between treatment level and patient resources leads to the passive and weak position of primary medical institutions in ensuring drug supply. The shortage of drugs in primary medical institutions is more prominent due to the lack of patient resources, basic drug demand, and the fact that most primary medical institutions are located in less-developed regions, and distribution enterprises are reluctant to deliver due to relatively low-profit margin. Primary medical institutions play an essential role in the treatment of common diseases, especially for patients in rural or remote areas. More efforts should be made to ensure drug supply in primary medical institutions.

Up to now, no other countries are reporting the characteristics of drug shortages at different levels of medical institutions. There are great differences between China and most developed countries in the classification of medical institutions, the scale and functional positioning of hospital pharmacies and so on. In most developed countries, such as the United States, the United Kingdom, Germany, and France, have achieved the SDP, hospital pharmacies mainly for inpatients.³⁸ In the United States, developed retail pharmacy networks account for 70% of drug sales, while medical institutions account for less than a third.³⁸ Patients usually go to retail pharmacies to obtained medicines. In China, medical institutions account for 85% of drug sales.³⁸ Due to long-standing buying habits, differences in drug types and prices between hospitals and retail pharmacies, relatively good drug reputation in hospitals, and restrictions on prescription outflow in hospitals, patients generally choose to get drugs in hospital pharmacies by prescription. Once there is a drug shortage in medical institutions, the treatment of patients will be seriously affected. Based on the above, although the shortages in medical institutions among different countries may not be comparable, we expect that the findings of our study could provide a certain degree of reference for other countries to alleviate and solve the problem of drug shortage.

Based on the risk assessment score, we built a graded risk-management strategy decision tree (see Figure 2) and applied it to the monitoring, early warning, and selection process of drugs in shortage in Hunan province. Drugs with high risk are assigned to the provincial level through a series of measures. If the intervention of relevant provincial departments has little effect, the case is assigned to the state. For medium-risk drugs, shortages are managed by municipal monitoring and warning centers, and if not resolved at the city level, are reported to the province or further to the state. Low-risk drugs are handled at the county or city level, with continuous monitoring as the main measure. The three-level framework (province, city, county) management effectively filters invalid information layer by layer and increases management efficiency. Classification management by risk level further saves limited public health resources and is more beneficial for drugs in genuine short supply.

Several studies have assessed the risk of drug shortages.^5,22,39 A study in Italy⁵ constructed an algorithm to measure the impact of shortage or unavailability on public health in Europe by using the three-dimension indexes of ‘type of disease,’ ‘therapeutic alternatives,’ and ‘market shares of product.’ However, the score weight criteria of each indicator were derived from expert consensus. Shang et al²² established a risk evaluation index system for shortages of essential drugs in China. However, the index system was only applicable to essential drugs and could not completely meet the evaluation needs of non-essential drugs. A study by Huang et al³⁹ constructed a selection index system for a drug shortage list based on AHP. This index system is only applicable to the selection of a drug shortage list and requires data from all links in the pharmaceutical supply chain, including production, circulation, and use. However, data accessibility is not mentioned in the study and practical application was not conducted.

Our study has several limitations. First, in the Delphi process, considering the feasibility, objectivity, funding, and time limit of the study, we included only experts in medical institutions and distribution enterprises, which may have led to the exclusion of comments from experts in other stakeholder group representatives such as legislative/regulatory departments, pharmaceutical manufacturer and ingredients supplier and so on. Although personnel from legislative/regulatory departments are not included in this study, the initial index framework was developed mainly based on government documents formulated by the State Council and the National Health Commission, which can largely reflect the opinions of legislative/regulatory. Due to relevant policy support, it is profitable for the pharmaceutical manufacturer or ingredients supplier to make a drug in shortage. The inclusion of them in the expert group may affect the objectivity of the assessment indicators of drug shortage risk. Second, the weight value of each index depends on the subjective judgment of experts and may not reflect the true preferences of policy-makers and decision-makers. Whenever participants were involved in the AHP stage, the assessment process had to be re-performed and the results will change accordingly. All the experts in this study were strictly selected and required to complete Delphi questionnaires carefully, accurately, and timely to ensure the quality of the study. Third, it was challenging to develop a set of indicators that apply to all regions in China. The index system has certain requirements for data acquisition, and not every province has access to all information needed for risk assessment.

Conclusions

A hierarchical indexed system for the risk assessment of drug shortages in China was established for the first time, using a modified Delphi process with AHP. The evaluation scores could provide clues for guiding the graded management of drugs at different risk levels. Decision trees for risk-management strategies have been established for the features of drugs at different risk levels. This study provides an objective and practical tool to guide risk assessment of drug shortages and a scientific basis for the classification and management of drugs with different risk levels. However, this risk assessment index system is not set in stone, but will require constant improvement in its practical application. We expect that the index system will contribute to the alleviation and solution of drug shortages, and thereby the reduction of disease burden and the establishment of a healthy China.

Acknowledgments

The authors gratefully acknowledge the Health Commission of Hunan Province and the members of the Delphi panel who provided their expertise and experience: Zaixin Yu, Xiangmin Li, Jianglin Zhang, Yaoyun Tang, Yu Zhang, Jun Quan, Sheng Deng, Min Yan, Ling Tang (Xiangya Hospital Central South University), Ping Xu (The Second Xiangya Hospital of Central South University), Xiaocong Zuo (The Third Xiangya Hospital of Central South University), Xin Zhao (Hunan Children’s Hospital), Xiaoke Wen (Hunan Provincial Maternal and child Health Care Hospital), Shujia Kong (Yunnan Cancer Hospital), Can Luo (Jiangsu Province Hospital), Yan Qian (The Second Affiliated Hospital of Chongqing Medical University), Haiyan Lao (Guangdong Provincial People’s Hospital), Tao Liu (Sun Yat-Sen University Cancer Center), Qunli Yuan (China Resources Hunan Ruige Pharmaceutical Co., Ltd.) and Zehua Wang (Sinopharm Group Hunan Co., Ltd.).

Ethical issues

Ethical approval was not necessary for the study because this research does not involve human subjects, human material, human tissues, or human data.

Competing interests

Authors declare that they have no competing interests.

Authors’ contributions

YS wrote the draft of the manuscript and interpreted the results. YS and QP collected and analysed the data. BZ designed the study and provided administrative and material support. SS and JD was involved in editing and revising each draft and provided overall guidance. SL, TY, ZG, and ZC provided administrative support and comments and suggestions in revisions of the paper. All authors approved the final submitted version.

Disclaimer

The funders had no role in the study design, data collection, data analysis, data interpretation and in writing the manuscript.

Funding

This work was supported by the Clinical Pharmacy Research Foundation of Hunan Medical Association (grant number HMA202001001) and the Hospital Pharmacy Research Foundation of Hunan Pharmaceutical Association (grant number 2020YXH007).

Supplementary files

References

1. Iyengar S, Hedman L, Forte G, Hill S. Medicine shortages: a commentary on causes and mitigation strategies. BMC Med 2016; 14(1):124. doi: 10.1186/s12916-016-0674-7 [Crossref] [ Google Scholar]
2. Schwartzberg E, Ainbinder D, Vishkauzan A, Gamzu R. Drug shortages in Israel: regulatory perspectives, challenges and solutions. Isr J Health Policy Res 2017; 6:17. doi: 10.1186/s13584-017-0140-9 [Crossref] [ Google Scholar]
3. Gray A, Manasse HR Jr. Shortages of medicines: a complex global challenge. Bull World Health Organ 2012; 90(3):158-158a. doi: 10.2471/blt.11.101303 [Crossref] [ Google Scholar]
4. Fox ER, McLaughlin MM. ASHP guidelines on managing drug product shortages. Am J Health Syst Pharm 2018; 75(21):1742-1750. doi: 10.2146/ajhp180441 [Crossref] [ Google Scholar]
5. Musazzi UM, Di Giorgio D, Minghetti P. New regulatory strategies to manage medicines shortages in Europe. Int J Pharm 2020; 579:119171. doi: 10.1016/j.ijpharm.2020.119171 [Crossref] [ Google Scholar]
6. Fox ER, Birt A, James KB, Kokko H, Salverson S, Soflin DL. ASHP guidelines on managing drug product shortages in hospitals and health systems. Am J Health Syst Pharm 2009; 66(15):1399-1406. doi: 10.2146/ajhp090026 [Crossref] [ Google Scholar]
7. Acosta A, Vanegas EP, Rovira J, Godman B, Bochenek T. Medicine shortages: gaps between countries and global perspectives. Front Pharmacol 2019; 10:763. doi: 10.3389/fphar.2019.00763 [Crossref] [ Google Scholar]
8. The American Society of Health-System Pharmacists. Drug Shortages Statistics. https://www.ashp.org/Drug-Shortages/Shortage-Resources/Drug-Shortages-Statistics. Accessed July 17, 2020.
9. Food and Drug Administration. FDA Drug Shortages. https://www.accessdata.fda.gov/scripts/drugshortages/default.cfm. Accessed July 17, 2020.
10. Hunan Public Resources Trading Service Platform. Drug shortages Reports. http://jyjy.hnsggzy.com/. Accessed July 17, 2020.
11. Shi Y, Yang P, Li X. Combating drug shortages in China: surveillance warning and practice standardization. Int J Clin Pharm 2020; 42(2):309-314. doi: 10.1007/s11096-020-00987-5 [Crossref] [ Google Scholar]
12. National Health Commission of the People’s Republic of China. Notice of the National Health Commission on the monitoring of drug use and comprehensive clinical evaluation. http://www.nhc.gov.cn/yaozs/pqt/201904/31149bb1845e4c019a04f30c0d69c2c9.shtml. Accessed August 16, 2020.
13. National Health Commision of the People’s Republic of China. National Health Commission technical guide for classifcation and grading of drug shortages in medical institutions. http://www.nhc.gov.cn/yaozs/s7653/201907/495842935554402baf6470a7c6244343/files/cebbe102b9b54f2283cedc62aae04749.pdf. Accessed July 20, 2020.
14. National Health Commission of the People’s Republic of China. Regulations for national list of drugs in shortage (trial version). http://www.nhc.gov.cn/yaozs/s7655g/202004/f1cbe67ec7ef4c228ed2290946745d04/files/1ab369dd5c7544f792ae9e08f27056a4.pdf. Accessed July 16, 2020.
15. Health Commission of Yunnan province. Drug shortage list of Yunan province of 2020 (the first issue). http://ynswsjkw.yn.gov.cn/wjwWebsite/web/doc/UU152341528561864802. Accessed September 3, 2020.
16. Health Commission of Ningxia Hui Autonomous Region. Drug shortage list of Ningxia Hui autonomous region of 2020. http://wsjkw.nx.gov.cn/info/1040/18687.htm. Accessed September 3, 2020.
17. Health Commission of Shanxi province. The list of drugs in shortage and the key monitoring list of drugs that are clinically necessary and vulnerable to shortage of Shanxi province (issue 2). http://wjw.shanxi.gov.cn/gsggl08/26506.hrh. Accessed September 3, 2020.
18. Health Commission of Guizhou province. The list of drugs in shortage (issue 3) and the key monitoring list of drugs that are clinically necessary and vulnerable to shortage (issue 1) of Guizhou province. http://www.gzhfpc.gov.cn/xwzx_500663/tzgg/202006/t20200617_61146538.html. Accessed September 3, 2020.
19. Health Commission of Guangxi Zhuang Autonomous Region. Drug shortage list of Guangxi Zhuang Autonomous Region (issue 2). http://wsjkw.gxzf.gov.cn/zfxxgk_49572/wsjszh/yzgl_49605/t5237736.shtml. Accessed September 3, 2020.
20. Health Commission of Hainan province. Drug shortage list of Hainan province. http://wst.hainan.gov.cn/swjw/xxgk/0200/0202/202007/t20200703_2813602.html. Accessed September 3, 2020.
21. National Health Commission of the People’s Republic of China. Notice on the issurance of the national list of drugs in shortage. http://www.nhc.gov.cn/yaozs/s7653/202012/f30aad8ec4ba48a9afa2e559f4d20e7c.shtml. Accessed December 31, 2020.
22. Shang CC. Establishment of Index System for the Risk of Essential Drug Shortages. Beijing: Beijing University of Chinese Medicine; 2017.
23. The State Council. Suggestions on further stabilizing the supply and price of drugs in shortage. http://www.nhc.gov.cn/bgt/gwywj2/201910/7d2b39c6f68d46bd82f184731de80dcc.shtml. Accessed July 20, 2020.
24. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med 2017; 31(8):684-706. doi: 10.1177/0269216317690685 [Crossref] [ Google Scholar]
25. Clough S, Shehabi Z, Morgan C. Medical risk assessment in dentistry: use of the American Society of Anesthesiologists Physical Status Classification. Br Dent J 2016; 220(3):103-108. doi: 10.1038/sj.bdj.2016.87 [Crossref] [ Google Scholar]
26. Lei L, Richards JS, Li ZH, Gong YF, Zhang SZ, Xiao N. A framework for assessing local transmission risk of imported malaria cases. Infect Dis Poverty 2019; 8(1):43. doi: 10.1186/s40249-019-0552-6 [Crossref] [ Google Scholar]
27. Saaty TL. The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation. Pittsburgh: RWS Publications; 1996.
28. Miller GA. The magical number seven plus or minus two: some limits on our capacity for processing information. Psychol Rev 1956; 63(2):81-97. [ Google Scholar]
29. Drug Classification and Procurement System of Hunan Province. Report of drug shortage, 2020. https://jyjy.hnsggzy.com/. Accessed August 16, 2020.
30. Chen S, Wang M, Chen L. Development of an indicator system to evaluate nursing safety and quality in ICU. Chin J Nurs 2014; 49(3):270-274. [ Google Scholar]
31. Whitfield D, Bosson N, Habrat D. Critical medication shortages: effective strategies to maintain continuous medication availability for emergency medical services. PrehospEmerg Care 2020; 24(2):290-296. doi: 10.1080/10903127.2019.1619001 [Crossref] [ Google Scholar]
32. Serena T, Bates-Jensen B, Carter MJ. Consensus principles for wound care research obtained using a Delphi process. Wound Repair Regen 2012; 20(3):284-293. doi: 10.1111/j.1524-475X.2012.00790.x [Crossref] [ Google Scholar]
33. Riggs WE. The Delphi technique: an experimental evaluation. Technol Forecast Soc Change 1983; 23(1):89-94. doi: 10.1016/0040-1625(83)90073-2 [Crossref] [ Google Scholar]
34. Schmidt K, Aumann I, Hollander I, Damm K, von der Schulenburg JM. Applying the analytic hierarchy process in healthcare research: a systematic literature review and evaluation of reporting. BMC Med Inform DecisMak 2015; 15:112. doi: 10.1186/s12911-015-0234-7 [Crossref] [ Google Scholar]
35. Scala NM, Rajgopal J, Vargas LG, Needy KL. Group decision making with dispersion in the analytic hierarchy process. Group DecisNegot 2016; 25(2):355-372. doi: 10.1007/s10726-015-9445-7 [Crossref] [ Google Scholar]
36. Pi HY, Peng PP, Su QQ, Wang YL, Tang JP, Zhang J. A study on long-term care of quality evaluation index system for disabled elderly. Chin J Nurs 2018; 53(9):1110-1114. [ Google Scholar]
37. National Health Commission of the People’s Republic of China. China Health Statistics Yearbook 2019. Beijing: Peking Union Medical College Press; 2019:6-8.
38. Du C, Zhu H, Fang Y. Institutional complementarities and differences in pharmaceutical circulation systems between foreign countries and China. Finance & Trade Economic 2015(4):109-120.
39. Huang RQ, Li F. Research on the construction of selection index system of drug shortage list based on analytic hierarchy process. Chinese Journal of Health Policy 2020; 13(6):8-13. [ Google Scholar]