Did an Intervention Programme Aimed at Strengthening the Maternal and Child Health Services in Nigeria Improve the Completeness of Routine Health Data Within the Health Management Information System?

Background

Effective policy and management decisions require good-quality evidence.^1,2 Data from routine Health Management Information Systems (HMIS) forms is an important and routinely collected source of such evidence.³ HMIS are specially designed to help health planners and managers in the management and planning of health services and programmes as well as decision-making to improve the availability and quality of care.^4,5 HMIS are also important for health system strengthening (HSS) yet, assuring the quality of health information systems remains a challenge.

In many low- and middle-income countries, comprehensive data needed to inform rational and effective health policy management and planning decisions are not generated, but are not of sufficient quality to use for decision-making especially at the primary health center (PHC) level.^6-8 Health data produced in low-resource settings are rarely routinely available for every population and quality issues limit their use for policy directions.⁸ As a result, many programmes particularly those financially supported by donor funding largely ignore HMIS and spend substantial resources on establishing and maintaining vertical programme information systems.

In Nigeria, the National HMIS (NHMIS) is designed to capture 233 variables in one proforma and routinely collected. It comprises information flow from health facilities to the local government area, and then the State and Federal level. Thus data from health facilities are collated and aggregated at these different levels.^9,10 The goal of the Nigeria HMIS was to have an effective HMIS for informed decision-making at all levels of government.¹⁰

Over the years the NHMIS have been noted to be weak specifically in terms of incomplete and inaccurate data in facility paper summaries, reliability and use in supporting the health system.^11-13 An assessment on the data quality of the routine health management information in one of the Nigerian states found poor data quality at health facility and district levels to consist of missing values, inconsistent data and poor usability.¹⁴

Efforts have been made to improve the availability of high-quality data to support decision-making at all levels of the health system in Nigeria including support to Federal Ministry of Health to develop a master facility list to improve data quality which will ultimately lead to better coordination of health services.¹⁵ Also, the Government has made considerable investment in strengthening health information systems, including District Health Information System 2, to support performance management and service delivery to reduce preventable deaths for mothers and newborns.^16,17

As Nigeria embraces the Sustainable Development Goals, maternal and child health (MCH) remains a national and international priority.¹⁸ MCH was the HSS component of the Subsidy Reinvestment and Empowerment Programme (SURE-P) that aimed to plough back the subsidy removed from petroleum products into projects that will benefit its citizens living in rural and underserved areas.¹⁹ The programme started in October 2012 but the funding was suddenly withdrawn in 2015 by a newly elected government. The MCH component of SURE-P (SURE-P/MCH) involved supply and demand components. The supply component comprised infrastructural upgrade of facilities, the supply of medical and surgical consumables and increased human resources (midwives, community health extension workers, and village health workers). These health workers received training on data management including data collection, data entry and storage, analysis, and use of routine health data as part of the supply side component. This was necessary so as to track the programme inputs and health services utilization indicators. The demand component involved paying out cash to pregnant mothers to register at a health facility and complete the continuum of care (antenatal care, delivery by a skilled birth attendant, postnatal attendance for immunization and family planning). The outputs from this programme were to be captured by the health workers using the HMIS data reporting tools which were present in every PHC facility including the non-SURE-P/MCH facilities nationwide.

Despite a compelling need for robust evidence of HMIS function, the contribution of HSS programmes (such as SURE-P/MCH) to HMIS data quality has not been sufficiently evaluated in Nigeria. Yet, ensuring the completeness of data at the source is critical to the overall quality of data available at other higher levels of the reporting system. This paper, therefore, aims to (a) evaluate if SURE-P improved on the completeness of MCH data at the health facilities from the HMIS (b) highlight issues affecting the completeness of data, and (c) identify a broad set of strategies which can help further improve the completeness of HMIS data and improve its potential for informing policy, planning and management decisions. The paper should be of interest and relevance to policy-makers, and development partners who are engaged in studying and improving NHMIS as well as programme managers and planners who are interested in improving the quality of data to inform their policy and planning decisions.

Methods

Study Area and Setting

This study was undertaken in Anambra State, southeast Nigeria. The state has a population of about 4.1 million and has a mix of urban and rural areas. MCH services are primarily provided from the PHCs, each of which covers a given catchment population. There are some trained (maternity homes) and untrained (traditional birth attendants, patent medicine vendors) who also offer unmonitored MCH services. In the context of the SURE-P/MCH programme, 4 PHCs are linked to a named general hospital for referral of emergency obstetric complications, and this is referred to as a cluster (4 PHCs +1 general hospital).

Since June 2015, we evaluated the extent which and under what circumstances the SURE-P/MCH programme in Anambra state, southeast Nigeria achieved and sustained its outputs and outcomes.²⁰ The project objectives and its methodological approach were reported elsewhere.²¹ The secondary analysis of MCH data from the HMIS (collected from the facility registers) is an important component of the research project. Thus, this is part of a larger study that sought to determine the effectiveness of a novel HSS and community health workers programme in improving MCH in Nigeria.

Study Design

The overall study used a case study mixed-methods approach as described elsewhere.¹⁴ There were 12 sites studied, which comprised 3 clusters: 4 control sites, 4 SURE-P sites, and 4 SURE-P+CCT (conditional cash transfer) sites. A cluster is made up of 4 PHCs clustered around a general hospital which serves as a referral centre for emergency obstetric care.¹² The location of these intervention clusters was entirely decided by the SURE-P project implementation unit at the federal ministry of health.

Data Collection

We collected information from 2 main sources of data: quantitative and qualitative. For our quantitative data we used a HMIS-based dataset that we had previously collected for a separate study from facility registers in the PHC facilities only which covered key monthly MCH indicators. We then used this to analysed how the completeness of those indicators changed before, during, and after the SURE-P programme across the 3 cluster. This allowed us to test, in a controlled way, whether the SURE-P programme (and its termination) had any impacts on HMIS data completeness. Second, we used in-depth interviews with purposively identified key stakeholders to understand HMIS data quality in terms of completeness and its use in their policy, management and planning decisions to understand their views and experiences with the HMIS data. We did not conduct a sample size calculation given we used an existing and fixed dataset.

In addition to the quantitative data collection, in-depth interviews were conducted with health facility workers in 2017 after the SURE-P programme. From each of the 8 PHC facilities in the 2 intervention clusters, the facility manager and a SURE-P/MCH program midwife and another health worker who may not necessarily be a SURE-P staff but from the pre-existing staff were purposively selected for the interviews giving a total of 24 health workers. These health workers were interviewed by the researchers who were trained in interviewing. Information was collected on how they collect, summarize and transmit data in the health center and what their experiences have been in doing this, including the enablers and challenges of data management.

Data Analysis

To evaluate HMIS data completeness our trained researchers had previously used a standardised proforma²² to collect the facility-level, monthly HMIS data on 4 key MCH indicators, all measured as counts per facility (See Supplementary file 1): (1) total antenatal clinic visits (the total number of women that month who visited the PHC for any antenatal clinic meeting); (2) total postnatal clinic visits, (the total number of women that month who visited the PHC for any postnatal clinic meeting); (3) number of deliveries taken by a skill birth attendant; (4) number of pregnant women receiving 2 doses of tetanus toxoid. We collected these facility data from all 3 clusters across 3 distinct periods (dates include the entire of each start and end month listed): (1) for the 9 months before SURE-P interventions began (January 2012 - September 2012) – the “before SURE-P” period, (2) for the 32 months during which SURE-P interventions were funded and running (October 2012 - April 2015) – the “during SURE-P” period, and (3) for the 32 months after SURE-P intervention activities ended (May 2015 - December 2017) – the “after SURE-P” period. The cut-off dates used for these periods were based on expert knowledge of the programme’s running. For this study we then used these count data to create a cluster-level, monthly measure of data completeness for these indicators. We did this by calculating the monthly, cluster-level percentage of non-missing values across the 4 indicators and across the 4 PHC facilities within each cluster as:

n16×100

where n = the number of non-missing values across the 4 indicators for all 4 PHC facilities per cluster. We then used a controlled, 3-period interrupted time series (ITS) analysis approach to analyse how this monthly completeness outcome varied before, during and after the SURE-P programme in (1) the SURE-P only cluster compared to the control cluster, and in (2) the SURE-P+CCT cluster compared to the control cluster, with a separate model for each comparison. We used the itsa function in Stata statistical software. The ITS analyses use multiple linear regression models, but with the inferential estimates (confidence intervals and P values) based on Newey-West standard errors to account for temporal autocorrelation (for a given lag) and heteroscedasticity. The models were tested for generalised serial correlation (using the actest function in Stata) and adjustments were made to the lag structure if necessary.²³ We specified the 2 ITS models as follows:

M _t + β ₀ + β ₁ T _t + β ₂ Z + β ₃ ZT _t + β ₄ X1_t + β₅X1_tT_t + β₆ZX1_t + β₇ZX1_tT_t + β₈X2_t + β₉X2_tT_t + β₁₀ZX2_t + β₁₁ZX2_tT_t + ε_t

where M_t is the outcome variable measured at each month. T_t is the month since the start of the time series (1:72), Z is a dummy/indicator variable indicating the intervention or control cluster (control = 0, intervention = 1), X1_t is a dummy variable indicating the “during SURE-P” period (0 = before or after SURE-P period months, 1 = during SURE-P period months), and X2_t is a dummy variable indicating the “after SURE-P” period (0 = during or before SURE-P period months, 1 = after SURE-P period months). The remaining variables are interaction terms among the variables explained above.

Therefore, the parameter estimates (βs) of interest for our research questions are: (1) β₆, which estimates the mean change, in percentage points of completeness, for the given intervention cluster, that occurred immediately following the start of SURE-P, after controlling for the same change as estimated for the control cluster; (2) β₇, which estimates how the slope/trend (linear change in percentage points of completeness per month) changed, for the given intervention cluster, between the before SURE-P period to the during SURE-P period, after controlling for the same change as estimated for the control cluster; (3) β₁₀, which estimates the mean change, in percentage points of completeness, for the given intervention cluster, that occurred immediately following the termination of SURE-P, after controlling for the same change as estimated for the control cluster; and (4) β₁₁, which estimates how the slope/trend (the expected linear change in percentage points of completeness per month) changed, for the given intervention cluster, between the during SURE-P period to the after SURE-P period, after controlling for the same change as estimated for the control cluster. The remaining βs are structural parts of the ITS models and not of direct interest for our research questions. Therefore, for simplicity we only present the results of these 4 key parameters estimates in the results tables.

In addition to these 2 controlled ITS analyses comparing before-to-during and during-to-after SURE-P, we also did 3 uncontrolled “two-period” ITS analyses to model how the same completeness outcome changed for each of the 2 intervention clusters and the control cluster alone, and just between 2 periods: from the start of the data until the end of SURE-P (January 2012 - March 2015), and from the end of SURE-P until the end of the data (April 2015 - December 2017). These analyses allowed us to evaluate whether there were any long-term changes in key MCH HMIS indicator completeness that started before the SURE-P programme began, and if they were present in the control cluster as well as the intervention clusters. Therefore, these analyses allowed us to evaluate whether the data were consistent with any observed improvements in key MCH HMIS indicator completeness being due to secular/background changes rather than the effects of SURE-P. For simplicity we do not detail the model here. From these models the key parameter estimates we report indicate: (1) the estimated slope/trend (linear change in percentage points of completeness per month) for the before and during SURE-P period combined; (2) the estimated mean change, in percentage points of completeness, for the given cluster that occurred following the termination of SURE-P; and (3) the estimated change in slope/trend (linear change in percentage points of completeness per month), for the given cluster that occurred between the before and during SURE-P period combined compared to the after SURE-P period.

All the in-depth interviews (n = 24) were audio-recorded and transcribed after informed consent were collected from the respondents. Interviews were transcribed and analysed manually identifying emerging themes.

Results

We first present the results of the quantitative analysis of how the completeness of 4 key MCH HMIS indicators varied before, during and after SURE-P in each intervention cluster compared to the control clusters, and how the completeness varied from before and during SURE-P compared to after SURE-P in each cluster alone. We then present the results exploring health workers’ perspectives on the completeness of data as a proxy for data quality, the challenges to maintaining complete data and recommendations for reducing missing and incomplete data.

The results from the ITS analysis suggest that there is no statistically clear evidence of any additional changes in the percentage of data completeness for the key MCH HMIS indicators, either immediately after the introduction of SURE-P or immediately after the termination of SURE-P, in either the SURE-P only or SURE-P+CCT cluster, when compared to the same immediate changes observed in the control cluster (Table 1). Similarly, the results also show no statistically clear evidence of any additional changes in the trend with which key MCH HMIS indicator completeness changed over time, either after the introduction of SURE-P or after its termination, in either the SURE-P only or SURE-P+CCT cluster, when compared to the same trends observed in the control cluster.

However, when looking at changes in the level and trend of the completeness outcome across both the period before SURE-P and the period during SURE-P combined compared to the period after SURE-P there is: (1) a statistically clear and strongly positive trend in the percentage of the completeness outcome for all 3 clusters within the period combining before and during SURE-P (Table 2), and (2) a statistically clear “levelling out” in the percentage of the completeness outcome during the period after SURE-P in all 3 clusters, due to the outcome reaching its ceiling or near its ceiling (100% completeness).

This is also clearly apparent from Figures 1 and 2. Therefore, the results provide no evidence for any effect of SURE-P on key MCH HMIS indicator completeness within these intervention clusters, but clear evidence of a substantial positive trend in key MCH HMIS indicator completeness in all clusters that started from at least 9 months before the SURE-P period and continued for most, if not all, of the SURE-P period, and ultimately resulted in completeness improving by approximately 20%-30% points to reach typically 100% during after SURE-P period in all 3 clusters.

Close

Figure 1. Scatter Plot of Monthly Observed Percentage Completeness for Key MCH HMIS Indicators for SURE-P Only and Control Clusters and Predicted Values From the Controlled 3-Period ITS Model. Abbreviations: ITS, interrupted time series; HMIS, Health Management Information Systems; MCH, maternal and child health; SURE-P, Subsidy Reinvestment and Empowerment Programme.

Close

Figure 2. Scatter Plot of Monthly Observed Percentage Completeness for Key MCH HMIS Indicators for SURE-P Centers Giving CCT and Control Clusters and Predicted Values From the Controlled 3-Period ITS Model. Abbreviations: ITS, interrupted time series; HMIS, Health Management Information Systems; MCH, maternal and child health; SURE-P, Subsidy Reinvestment and Empowerment Programme; CCT, conditional cash transfer.

At the general hospitals which were part of the intervention clusters, it was found that the NHMIS forms were not used for reporting their data, both before during and after SURE-P programme. The data from different departments of each hospital was captured in notebooks and registers supplied by other programmes, eg, the malaria and HIV control programmes. In addition to poorly filled and missing registers, there was no formal data summary or harmonization. In addition, there were discrepancies in figures between the daily records and the monthly summary records, with monthly records having comparatively larger than expected numbers for certain variables. Consequently, as many of the HMIS indicators collected in the PHC facilities were not available in the general hospitals, the general hospital data did not form part of the ITS analyses of HMIS data completeness, but clearly this is an important qualitative finding regarding HMIS data completeness in Nigerian general hospitals.

Health Workers’ Perspective on Completeness of Data

During qualitative interviews, health workers felt that data completeness improved during SURE-P period and this was said to have resulted from the training given to them during the programme intervention period as well as the increased availability of staff. The participants also reflected that the quality in terms of completeness declined with the cessation of the programme. This was captured by some of the respondents thus:

“Yes, we received training on data collection, health education, delivery and many things and this has helped us in knowing how to manage our data” (P2C303).

“The presence of more staffs ensured that there are more people to handle the data collection …. and the staffs were trained in data management. But after SURE-P, the data collection started declining again because we still went back to the lack of staff… we are lacking staff to handle the data” (P2C304).

In general, interviews showed that existing records were poorly stored after SURE-P, and staff could not account for missing registers in some of the facilities due to alleged inadequate and improper handover by exiting staff. Interviews with the facility staff revealed that data management at the PHCs entails that each staff on duty or in charge of a particular unit collects data or records activities while on duty. This is collated by the officer in charge at the end of the month, and transmitted to the state, from where it is forwarded to the federal government. This was captured by respondents thus:

“In this facility we have staff that work in different areas in terms of data collection for antenatal clinic, Immunization, family planning etc. This is entered into different registers on a daily basis by the staff on shift and summarized at the end of the month. It is then forwarded to the local government at the end of the month by the facility in-charge” (P2C312).

Health workers were of the opinion that the completeness of HMIS data during the SURE-P period was as a result of more people available to handle the records and the training and re-training of staffs on data management. Other enablers of data management according to the respondents included availability of data management tools; staff motivation, which respondents pointed out that it keeps the staffs committed to the data management process; having a designated staff that is responsible for data management, mostly for the function of data collation and data summary. The enablers of data management were captured by a respondent thus:

“Some of the things that make it easier for us are availability of registers and staff because when you don’t have adequate staff to handle it, it will be delayed …another one is the motivation of staff by making the local government accessible to us through easy transportation when returning the data” (P2C301).

Challenges to Maintaining Complete Data

The respondents stated some of the challenges they face following the halt of the SURE-P/MCH programme, noting these challenges as the reason for the discrepancies in data management. These include lack of training and retraining of staffs which make the process tedious and compromises quality; the frequency of data summary which poses a challenge due to lack of adequate human resources brought about by halting of the SURE-P/MCH programme; the fact that there is no staff dedicated to data management, which is also due to the lack of human resources; the number of variables collected in the HMIS, as the form captures 233 variables. Also, the health workers are busy with other activities in the health center, thus not having enough time for data management because of increased workload. This was noted by several respondents thus:

“Imagine what happens when only one person is on duty and has to attend to all the patients and pregnant mothers, and still have to record data. They will record poorly and make mistakes” (P2C307).

“Generation of good and quality data is not easy; it is time-consuming. Sometimes the staffs don’t have enough time to see patients and they still have to document at the same time” (P2C308).

“There are too many things to collect information on. In our forms, we have 233 things to record and you do this on a daily and weekly basis” (P2C302).

Respondents were asked about the use for which data was being collected. According to most of the respondents, the information collected is used to make sure good services are delivered through informing health facility plans and staff performance management.

“The health facility committee at times use the information to plan how to deliver good services to the people like knowing how many deliveries in a month” (P2C317).

However, a few respondents were unable to state the uses of the data and distinguish which data is needed for service delivery and in fact felt it was just to monitor how well they are performing in the facilities and for their promotion.

“They just want to use the information to check how well we are performing and for our promotions” (P2C315).

“I don’t know which decisions they use the information for all I know is to collect the information” (P2C312).

Suggestions for Improving Missing and Incomplete Data

Some of the respondents made suggestions on how the improvement in availability and completeness of data can be solved and these included making sure that data is entered in the register from patients’ folders or files as patients arrive on a daily basis and then aggregated at the end of the month instead of waiting till the end of the month to enter the daily records from their folders to the register and then aggregate at the same time. As captured by a respondent:

“Just as I said the data should be filled daily, do not allow it to accumulate. Accumulation of data gives problem especially in a facility that a lot of patients come in … you can do that in a facility that does not have much client, but you don’t do it in a facility that have much client load. Sometimes, you might even start looking for their missing folders and files” (P2C305).

Training and retraining on data management and engaging more staff were also suggested by the respondents as means of improving the availability and completeness of data. According to a respondent: “Some of the staffs working now are untrained and needs fresh training to function well … a step-down training for the lay or volunteer staffs will help to manage the situation better” (P2C307).

Discussion

We explored if the SURE P/MCH programme contributed to improving the completeness of relevant HMIS data as a proxy for improved data quality. The facility data shows that there is evidence that key MCH HMIS indicator completeness improved substantially, but these changes cannot be attributed to SURE-P alone. Therefore, there appears to have been some background changes (possibly directly and indirectly related) to SURE-P affecting HMIS data completeness. For example, there were some interventions implemented in the State by some other programmes and donors to strengthen quality improvement in PHC facilities. One such intervention was improvement of quality of care in Nigeria’s PHC facilities in rural communities between October 2013 and March 2015.²⁴ The Malaria Consortium also implemented a capacity building project to improve the capacity of the national malaria elimination programme for evidence generation and use between 2008 and 2016.²⁵ In addition, there were extensive trainings of health staff on monitoring and evaluation which included data management in 2010, well before commencement of SURE-P.^26-28

Interestingly, although data completeness improved across all the clusters, there were differences in the SURE-P and the SURE-P+CCT clusters. It is not very clear what could have contributed to the seemingly greater improvement in the CCT cluster during the intervention period. However, it is important to note that data on monthly service uptake throughout the continuum of care was collected in these CCT during quarterly monitoring and supervisory visits as they needed to track beneficiary retention throughout the period as closely as possible, and avert loss to follow-up.²⁹ The supervisors also delivered spot training on record keeping during such visits. These quarterly monitoring and supervisory activities therefore may have resulted in greater improvement in data completeness in the CCT cluster during the intervention period. The non-use of NHMIS forms at the general hospitals and the preference for the use of notebooks and registers supplied by other vertical programmes is likely to lead to non-capture of the data in the NHMIS. The implication is that the data is cut off from the NHMIS and therefore cannot be used to inform planning and decision-making both at the state and country-level as the hospitals will be using different datasets. Evidence has shown that HMIS forms were less likely to be available in hospitals in Anambra State.³⁰

From health workers’ perspective, data completeness seems to have resulted from the training given to health workers during the SURE-P/MCH period as well as the increased availability of staff which was one of the intervention packages. Inadequate human resources as a result of halting of the SURE-P/MCH programme was also noted by the respondents as one of the factors affecting data completeness since there was no more staff dedicated to data management. This fact was also observed in a study where time constraints on recording tasks and the balance between recording tasks and clinic work was noted as a determinant of data completeness.³¹ However, there was no evidence to support this finding in the quantitative data which arguably provides a more accurate and objective measure of the data completeness.³² The in-depth staff perceptions are subjective and can be influenced by multiple biases such as their actual knowledge and expectations about data completeness and quality, and their experiences with the implementation of the SURE-P.

From a wider perspective about the data informing decision-making, we acknowledge the emphasis in the current literature that perceptions of robust evidence by key decision-makers form important determinants of whether data (or evidence) is used to inform policy, planning and management decisions.^2,33 Further to data completeness, such perceptions often include the source of evidence (ie, whether it comes from a reputable source such as rigorous study or established system), its comprehensiveness (ie, nationally-representative datasets often seen as better quality) and nature (‘hard’ quantitative data is often seen as better quality than ‘softer’ expert views and opinions) as well as local contents.^1,34

It is unclear from our study, the extent to which the design of the form to capture 233 variables in a single form contributes to the incompleteness of data. It is plausible that this burden could tire and demotivate health workers so that forms are not filled properly. Data completeness may also be reduced because most of the health workers filling the forms lack a good understanding of what the data is used for. In some settings, health workers lack an understanding of the use for which the data was being collected and are unable to distinguish which data is needed for service delivery.³⁵ Similarly, it has been reported in Nigeria how the cumbersomeness and complexity of the forms, brought about by the huge number of variables was a major factor hindering the completeness of data in the NHMIS forms.⁴ Also, a 2014 review of select low- and middle-income countries by the World Health Organization (WHO) found cumulative reporting requests requiring upwards of 600 indicators³⁶ and technical inconsistencies related to procedures and terms also lead to further fragmentation.³⁷

The presence of funds and human resources by SURE-P/MCH may have also contributed to a better result with the filling of the forms during the SURE-P/MCH period. However, all these were said to have decreased with cessation of the SURE-P/MCH. As noted in some African countries, human resources are a huge challenge for maintaining the quality of data within their health information systems.^38,39 And the effect of training of health workers and increasing staff and resources for data collection has been noted as a means of improving the quality of data collected from PHCs.⁴⁰

Although data completeness was said to have improved we identified from health workers’ perspective that availability and completeness of data are a challenge to them, and are dependent on many health systems variables including limited staff, a high burden on providers for data collection, many variables in the data collection tools, and health worker incentives. This raises a broader point about HMIS being also linked with other health systems components and appropriate staff support and development processes and mechanisms that can be important strategies to improve HMIS data completeness. It has been noted elsewhere that factors such as lack of training, appropriate data collection tools, overwhelming task of data collection are key challenges to data quality.⁴¹

Training and retraining on data management were suggested by the respondents as a means of improving the completeness of data. Evidence has shown that health HMIS training achieved an improvement in the data management practice of PHC workers.^42,43 Integrating capacity building in HMIS strengthening efforts is an essential component of a package of HMIS strengthening interventions and are also necessary for sustainability.⁴⁴ This has also been found to have significantly increased the completeness of the data used to monitor “prevention of mother-to-child transmission” services in South Africa.⁴⁵

Another strategy to improve data completeness especially at PHC level, will be to develop standard operating procedures for completing and accurately documenting in registers and monthly reporting forms. Progress towards establishing a strong, functional data collection and reporting system at PHC levels in Nigeria has been reported in a recent study that found improved data reporting and quality from the implementation of integrated community case management programs.⁴⁶ The use of electronic or mobile devices have also been reported to reduce the burden for healthcare workers, especially community health workers at the PHC level in Nigeria⁴⁶ and strengthening electronic health information systems, and harmonizing data collection systems have been suggested.⁴⁷

Study Limitations

Although we did not conduct a formal power calculation it is likely that our relatively small sample size meant that we lacked power to detect anything other than large changes. Also, ITS uses population-level data, so we cannot make inferences about each individual. In addition, our ITS analyses were based on only 2 SURE-P MCH clusters within one State of the Federation, and our statistical results therefore lack generalizability and are not intended to be statistically representative of all States in the Federation. There was also an imbalance in the length of the period for which HMIS data were available either side of the SURE-P period, with the pre-SURE-P period having less than a year’s worth of data available. Having access to a longer timeseries before SURE-P may have helped us understand when the improvements in HMIS completeness began, and therefore what their causes were likely to be. Also, while we attempted to assess completeness of data, we only used 4 key MCH indicators and did not conduct own observations of service provision against the recording of data. This resulted in a narrower objective measures of data completeness in our study and represents an important area for future research.

Conclusion

Completeness of data is an important element of data quality, which in turn will improve the chances of the resultant evidence to be used to inform policy, planning and management decisions. Although data completeness improved during SURE-P, the evidence suggests that there were no differences in improvement of data completeness between the control cluster and the SURE-P cluster; and between the control cluster and the SURE-P+CCT cluster. The observed increases in data completeness may therefore be due to other factors outside/beyond SURE-P. There were clearly factors causing improvements before and during the SURE-P period and whatever the factors, they were not harmed (or not significantly harmed) by the termination of SURE-P. However, health workers’ perceptions of how complete the HMIS data were at the relevant time periods is important and there are issues with the HMIS that need to be addressed. The functioning of national HMIS are inevitably linked with other health systems components. While health systems strengthening programmes have a great potential for improved overall systems performance, a more granular understanding of their implications on the specific components such as the resultant quality of HMIS data, is needed.

Acknowledgements

This work was funded by the Joint DFID/ESRC/MRC/Wellcome Trust Health Systems Research Initiative (Grant Ref: MR/M01472X/1). The funders had no role in the study design, data collection and analysis, decision to publish, or writing of the manuscript. The authors would like to thank The REVAMP Consortium project group members and the respondents for their contribution to the research.

Ethical issues

Ethical approvals for the study was obtained from the Health Research Ethics Committee at the University of Nigeria Teaching Hospital, Enugu (ref: NHREC/05/02/2008B-FWA00002458-1RB00002323), and the School of Medicine Research Ethics Committee at the Faculty of Medicine and Health at the University of Leeds (ref: SoMREC/14/097). Written informed consents were obtained from all the participants before data collection.

Competing interests

Authors declare that they have no competing interests.

Authors’ contributions

BU, TM, and OO designed the study. Data were collected by BU, OO, EE, CO, UO, and analysed by BU, OO, EE, CO, UO, and JH. The manuscript was drafted by BU and all authors reviewed and contributed content to the paper. BU critically revised the manuscript.

Funding

This work was funded by the Medical Research Council, Joint DFID/ESRC/ MRC/Wellcome Trust Health Systems Research Initiative (Grant Ref: MR/ M01472X/1 and 016530/Z/14/Z).

Supplementary files

References

1. Mirzoev T, Green A, Gerein N. Role of evidence in maternal health policy processes in Vietnam, India and China: findings from the HEPVIC project. Evid Policy 2013; 9(4):493-511. doi: 10.1332/174426413x669845 [Crossref] [ Google Scholar]
2. Onwujekwe O, Uguru N, Russo G. Role and use of evidence in policymaking: an analysis of case studies from the health sector in Nigeria. Health Res Policy Syst 2015; 13:46. doi: 10.1186/s12961-015-0049-0 [Crossref] [ Google Scholar]
3. Mirzoev T, Poudel AN, Gissing S. Is evidence-informed urban health planning a myth or reality? lessons from a qualitative assessment in three Asian cities. Health Policy Plan 2019; 34(10):773-783. doi: 10.1093/heapol/czz097 [Crossref] [ Google Scholar]
4. Abubakar A, Idris S, Nguku P, Sabitu K, Sambo M. Assessment of integrated disease surveillance and response strategy implementation in selected Local Government Areas of Kaduna state. Ann Niger Med 2013; 7(1):14-19. doi: 10.4103/0331-3131.119981 [Crossref] [ Google Scholar]
5. MEASURE Evaluation. Improving Continuity of Care by Strengthening Health Information Systems. MEASURE Evaluation; 2017.
6. Lippeveld T. Routine health information systems: the glue of a unified health system. Keynote Address at the Workshop on Issues and Innovation in Routine Health Information in Developing Countries, Potomac; March 14-16, 2001. Washington, DC: JSI; 2001.
7. AbouZahr C, Boerma T. Health Information Systems: The Foundations of Public Health. WHO; 2011.
8. Stevens GA, Hogan DR, Boerma T. Improving reporting of health estimates. Bull World Health Organ 2016; 94(7):483. doi: 10.2471/blt.16.179259 [Crossref] [ Google Scholar]
9. Federal Ministry of Health. National Standard Operating Procedure for the Collection and Management of Integrated Routine Health Data in Nigeria. Abuja, Nigeria: Department of Health Planning, Research & Statistics; 2015.
10. Federal Ministry of Health. Revised Policy-Programme and Strategic Plan of Action. 2006. http://cheld.org/wp-content/uploads/2012/04/NATIONAL-HEALTH-MANAGEMENT-INFORMATION-SYSTEM.pdf. Accessed February 28, 2020.
11. Anifalaje AA. Decentralisation and health systems performance in developing countries: impact of “decision space” on primary health care delivery in Nigeria. International Journal of Healthcare Delivery Reform Initiatives (IJHDRI) 2009; 1(1):25-47. doi: 10.4018/jhdri.2009010103 [Crossref] [ Google Scholar]
12. Health Reform Foundation of Nigeria. Nigerian Health Review. https://www.herfon.org.ng/. Accessed August 14, 2020. Published 2007.
13. Adejumo A. An Assessment of Data Quality in Routine Health Information Systems in Oyo State, Nigeria [Thesis]. Cape Town, South Africa: University of the Western Cape; 2017. https://etd.uwc.ac.za/handle/11394/5497. Accessed August 10, 2020.
14. Makinde OA, Onazi M, Adeleke O, Ohadi EM, Dieng AD, Osika JS, Abt Associates Inc. Assessment of the Routine Health Management Information System in Delta State, Federal Republic of Nigeria. Bethesda, MD: Health Systems 20/20 project, Abt Associates Inc; 2012.
15. MEASURE Evaluation. Strengthening Health Information Systems in Nigeria—Developing a Master Facility List. MEASURE Evaluation; 2019.
16. Federal Ministry of Health of Nigeria. Nigeria Every Newborn Action Plan: A Plan to End Preventable Newborn Deaths in Nigeria. Abuja, Nigeria: Federal Ministry of Health; 2016.
17. Federal Ministry of Health of Nigeria. National Health Information System Strategic Plan, 2014-2018. Abuja, Nigeria: Federal Ministry of Health; 2014.
18. United Nations. Zero Draft: Outcome Document to Adopt Post-2015 Development Agenda. Office of the Secretary-General’s Envoy on Youth; 2015. https://www.un.org/youthenvoy/2015/06/zero-draft-outcome-document-adopt-post-2015-development-agenda/. Accessed February 28, 2020. Published 2015.
19. Federal Ministry of Health. Nigerian Health Sector Performance Report: Subsidy Reinvestment and Empowerment Programme-Maternal and Child Health Services-Implementation Manual. Abuja, Nigeria: 2012. https://wbwaterdata.org/dataset/nigeria-subsidy-reinvestment-and-empowerment-programme-maternal-and-child-health-initiative.
20. Ebenso BE, Uzochukwu B, Manzano A, et al. De-constructing a complex programme using a logic map: realist evaluation of maternal and child health in Nigeria. In: The State of the Art of Realist Methodologies: A Two Day Conference; November; 2015; United Kingdom, Leeds.
21. Mirzoev T, Etiaba E, Ebenso B. Study protocol: realist evaluation of effectiveness and sustainability of a community health workers programme in improving maternal and child health in Nigeria. Implement Sci 2016; 11(1):83. doi: 10.1186/s13012-016-0443-1 [Crossref] [ Google Scholar]
22. Determinants of Effectiveness of a Novel Community Health Workers Programme in Improving Maternal and Child Health in Nigeria. https://gtr.ukri.org/projects?ref=MR%2FM01472X%2F1.
23. Baum CF, Schaffer ME. ACTEST: Stata Module to Perform Cumby-Huizinga General Test for Autocorrelation in Time Series. Statistical Software Components. https://ideas.repec.org/c/boc/bocode/s457668.html. Accessed February 27, 2020. Published January 2015.
24. Ugo O, Ezinne EA, Modupe O, Nicole S, Winifred E, Kelechi O. Improving quality of care in primary health-care facilities in rural Nigeria: successes and challenges. Health Serv Res Manag Epidemiol 2016; 3:2333392816662581. doi: 10.1177/2333392816662581 [Crossref] [ Google Scholar]
25. Malaria Consortium. FINAL REPORT Support to the National Malaria Programme. https://www.malariaconsortium.org/media-downloads/687. Accessed February 27, 2020. Published 2008.
26. Enemuoh JC, Onwujekwe OE, Uzochukwu BSC, Oranuba J, Ilika AL. Quality of monitoring and evaluation of malaria control activities: tracking stock-outs and replenishment of supplies in Anambra state, Nigeria. Sci J Public Health 2013; 1(5):201-208. doi: 10.11648/j.sjph.20130105.13 [Crossref] [ Google Scholar]
27. ANMCBP. A state malaria control Booster project. Report on Training of Local Government Team on Monitoring and Evaluation Tools. Awka 9th-15th August 2010, Anambra State Ministry of Health; 2010.
28. Mbachu CO, Uzochukwu BSC, Onwujekwe OE, Ilika AL, Oranuba J. How do health workers perceive and practice monitoring and evaluation of malaria control interventions in South-east Nigeria?. BMC Health Serv Res 2013; 13(1):81. doi: 10.1186/1472-6963-13-81 [Crossref] [ Google Scholar]
29. Okoli U, Morris L, Oshin A, Pate MA, Aigbe C, Muhammad A. Conditional cash transfer schemes in Nigeria: potential gains for maternal and child health service uptake in a national pilot programme. BMC Pregnancy Childbirth 2014; 14:408. doi: 10.1186/s12884-014-0408-9 [Crossref] [ Google Scholar]
30. Nnebue CC, Onwasigwe CN, Adogu PO, Onyeonoro UU. Awareness and knowledge of disease surveillance and notification by health-care workers and availability of facility records in Anambra state, Nigeria. Niger Med J 2012; 53(4):220-225. doi: 10.4103/0300-1652.107557 [Crossref] [ Google Scholar]
31. Taggart J, Liaw ST, Yu H. Structured data quality reports to improve EHR data quality. Int J Med Inform 2015; 84(12):1094-1098. doi: 10.1016/j.ijmedinf.2015.09.008 [Crossref] [ Google Scholar]
32. Quantitative and Qualitative Research - Objective or Subjective? Explorable website. https://explorable.com/quantitative-and-qualitative-research. Published 2009. Accessed February 27, 2020.
33. Mirzoev Mirzoev, Das M, Ebenso BE. Contextual influences on the role of evidence in health policy development: What can we learn from six policies in India and Nigeria?. Evid Policy 2015; 13(1):59-79. doi: 10.1332/174426415x14454407579925 [Crossref] [ Google Scholar]
34. Gabrysch S, Campbell OM. Still too far to walk: literature review of the determinants of delivery service use. BMC Pregnancy Childbirth 2009; 9:34. doi: 10.1186/1471-2393-9-34 [Crossref] [ Google Scholar]
35. World Health Organization. Regional Office for the Western Pacific. Developing Health Management Information Systems: A Practical Guide for Developing Countries. Manila: WHO Regional Office for the Western Pacific; 2004. http://www.wpro.who.int. Accessed February 28, 2020.
36. World Health Organization (WHO). A Rapid Assessment of the Burden of Indicators and Reporting Requirements for Health Monitoring. WHO; 2014.
37. Chilundo B, Aanestad M. Negotiating multiple rationalities in the process of integrating the information systems of disease-specific health programmes. Electron J Inf Syst Dev Ctries 2004; 20(1):1-28. doi: 10.1002/j.1681-4835.2004.tb00129.x [Crossref] [ Google Scholar]
38. Mphatswe W, Mate KS, Bennett B. Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa. Bull World Health Organ 2012; 90(3):176-182. doi: 10.2471/blt.11.092759 [Crossref] [ Google Scholar]
39. Makombe SD, Hochgesang M, Jahn A. Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bull World Health Organ 2008; 86(4):310-314. doi: 10.2471/blt.07.044685 [Crossref] [ Google Scholar]
40. Gimbel S, Mwanza M, Nisingizwe MP, Michel C, Hirschhorn L. Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research (CFIR): results from the African Health Initiative. BMC Health Serv Res 2017; 17(suppl 3):828. doi: 10.1186/s12913-017-2660-y [Crossref] [ Google Scholar]
41. Piette JD, Lun KC, Moura LA Jr. Impacts of e-health on the outcomes of care in low- and middle-income countries: where do we go from here?. Bull World Health Organ 2012; 90(5):365-372. doi: 10.2471/blt.11.099069 [Crossref] [ Google Scholar]
42. MEASURE Evaluation. Health Management Information System: Facilitator’s Guide for Training Hospital Staff. MEASURE Evaluation; 2013.
43. Nwankwo B, Sambo MN. Can training of health care workers improve data management practice in health management information systems: a case study of primary health care facilities in Kaduna State, Nigeria. Pan Afr Med J 2018; 30:289. doi: 10.11604/pamj.2018.30.289.15802 [Crossref] [ Google Scholar]
44. Ahsan KZ, Tahsina T, Iqbal A. Production and use of estimates for monitoring progress in the health sector: the case of Bangladesh. Glob Health Action 2017; 10:1298890. doi: 10.1080/16549716.2017.1298890 [Crossref] [ Google Scholar]
45. Nicol E, Dudley L, Bradshaw D. Assessing the quality of routine data for the prevention of mother-to-child transmission of HIV: an analytical observational study in two health districts with high HIV prevalence in South Africa. Int J Med Inform 2016; 95:60-70. doi: 10.1016/j.ijmedinf.2016.09.006 [Crossref] [ Google Scholar]
46. Nyangara FM, Hai T, Zalisk K. Assessment of data quality and reporting systems for underserved populations: the case of integrated community case management programs in Nigeria. Health Policy Plan 2018; 33(4):465-473. doi: 10.1093/heapol/czy003 [Crossref] [ Google Scholar]
47. Chen H, Yu P, Hailey D, Wang N. Methods for assessing the quality of data in public health information systems: a critical review. Stud Health Technol Inform 2014; 204:13-18. [ Google Scholar]