Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review

Document Type : Review Article

Authors

1 Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

2 National Journal, Washington, DC, USA

3 Johns Hopkins Berman Institute of Bioethics, Baltimore, MD, USA

4 Division of General Internal Medicine, Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA

Abstract

Background
Patient, public, consumer, and community (P2C2) engagement in organization-, community-, and systemlevel healthcare decision-making is increasing globally, but its formal evaluation remains challenging. To define a taxonomy of possible P2C2 engagement metrics and compare existing evaluation tools against this taxonomy, we conducted a systematic review.

 
Methods
A broad search strategy was developed for English language publications available from January 1962 through April 2015 in PubMed, Embase, Sociological Abstracts, PsycINFO, EconLit, and the gray literature. A publication was excluded if: (1) the setting was not healthcare delivery (ie, we excluded non-health sectors, such as urban planning; research settings; and public health settings not involving clinical care delivery); (2) the P2C2 engagement was episodic; or (3) the concept of evaluation or possible evaluation metrics were absent. To be included as an evaluation tool, publications had to contain an evaluative instrument that could be employed with minimal modification by a healthcare organization.

 
Results
A total of 199 out of 3953 publications met exclusion and inclusion criteria. These were qualitatively analyzed using inductive content analysis to create a comprehensive taxonomy of 116 possible metrics for evaluating P2C2 engagement. 44 outcome metrics were grouped into three domains (internal, external, and aggregate outcomes) that included six subdomains: impact on engagement participants, impact on services provided by the healthcare organization, impact on the organization itself, influence on the broader public, influence on population health, and engagement cost-effectiveness. The 72 process metrics formed four domains (direct process metrics; surrogate process metrics; aggregate process metrics; and preconditions for engagement) that comprised sixteen subdomains. We identified 23 potential tools for evaluating P2C2 engagement. The identified tools were published between 1973-2015 and varied in their coverage of the taxonomy, methodology used (qualitative, quantitative, or mixed), and intended evaluators (organizational leaders, P2C2 participants, external evaluators, or some combination). Parts of the metric taxonomy were absent from all tools.

 
Conclusions
By comprehensively mapping potential outcome and process metrics as well as existing P2C2 engagement tools, this review supports high-quality P2C2 engagement globally by informing the selection of existing evaluation tools and identifying gaps where new tools are needed.


 
Systematic Review Registration
PROSPERO registration number CRD42015020317

Highlights

Supplementary File 1 (Download)

Supplementary File 2 (Download)

Supplementary File 3 (Download)

Supplementary File 4 (Download)

Supplementary File 5 (Download)

 

Commentaries Published on this Paper

  • Using the Taxonomy and the Metrics: What to Study When and Why; Comment on “Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review”

           Abstract | PDF

  • Metrics of Patient, Public, Consumer, and Community Engagement in Healthcare Systems: How Should We Define Engagement, What Are We Measuring, and Does It Matter for Patient Care?; Comment on “Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review”

           Abstract | PDF

  • From Craft to Reflective Art and Science; Comment on “Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review”

           Abstract | PDF

  • Delving Into the Details of Evaluating Public Engagement Initiatives; Comment on “Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review”

           Abstract | PDF

  • Patient Engagement and its Evaluation Tools – Current Challenges and Future Directions; Comment on “Metrics and Evaluation Tools for Patient Engagement in Healthcare Organization- and System-Level Decision-Making: A Systematic Review”

           Abstract | PDF

 

Authors' Response to the Commentaries

  • The Why, Who, What, How, and When of Patient Engagement in Healthcare Organizations: A Response to Recent Commentaries

           Abstract | PDF

Keywords

Main Subjects

References

  1. Declaration of Alma-Ata. WHO Chron. 1978;32(11):428-430.
  2. Carman KL, Dardess P, Maurer M, et al. Patient and family engagement: A framework for understanding the elements and developing interventions and policies. Health Aff. 2013;32(2):223-231. doi:10.1377/hlthaff.2012.1133
  3. Charles C, DeMaio S. Lay participation in health care decision making: a conceptual framework. J Health Polit Policy Law. 1993;18(4):881-904. doi:10.1215/03616878-18-4-881
  4. An act to promote cost containment, transparency and efficiency in the delivery of quality health care, Enacted in the Acts of 2008, The General Laws of Massachusetts, Ch. 305, Sect. 11/53F (2008).
  5. DeCamp M, Sugarman J, Berkowitz S. Meaningfully Engaging Patients in ACO Decision-making. Am J Accountable Care. 2015;3(2):30-33.
  6. Pearson SD, Sabin JE, Emanuel EJ. No Margin, no Mission: Health-care Organizations and the Quest for Ethical Excellence. Oxford, New York: Oxford University Press; 2003.
  7. Egener BE, Mason DJ, McDonald WJ, et al. The charter on professionalism for health care organizations. Acad Med. 2017. doi:10.1097/ACM.0000000000001561
  8. Danis M, Solomon M. Providers, payers, the community, and patients are all obliged to get patient activation and engagement ethically right. Health Aff. 2013;32(2):401-407. doi:10.1377/hlthaff.2012.1081
  9. Galiher CB, Needleman J, Rolfe AJ. Consumer participation. HSMHA Health Rep. 1971;86(2):99-106.
  10. Martin GP. Whose health, whose care, whose say? Some comments on public involvement in new NHS commissioning arrangements. Crit Pub Health. 2009;19(1):123-132. doi:10.1080/09581590802385672
  11. Anton S, McKee L, Harrison S, Farrar S. Involving the public in NHS service planning. J Health Organ Manage. 2007;21(4-5):470-483. doi:10.1108/14777260710778989
  12. Culyer AJ. Involving stakeholders in healthcare decisions--the experience of the National Institute for Health and Clinical Excellence (NICE) in England and Wales. Healthc Q. 2005;8(3):56-60. doi:10.12927/hcq..17155
  13. Frankish CJ, Kwan B, Ratner PA, Higgins JW, Larsen C. Challenges of citizen participation in regional health authorities. Soc Sci Med. 2002;54(10):1471-1480. doi:10.1016/S0277-9536(01)00135-6
  14. Thurston WE, MacKean G, Vollman A, et al. Public participation in regional health policy: A theoretical framework. Health Policy. 2005;73(3):237-252. doi:10.1016/j.healthpol.2004.11.013
  15. Church J, Saunders D, Wanke M, Pong R, Spooner C, Dorgan M. Citizen participation in health decision-making: Past experience and future prospects. J Public Health Policy. 2002;23(1):12-32. doi:10.2307/3343116
  16. Preston R, Waugh H, Larkins S, Taylor J. Community participation in rural primary health care: Intervention or approach? Aust J Prim Health. 2010;16(1):4-16. doi:10.1071/PY09053
  17. Bath J., Wakerman J. Impact of community participation in primary health care: What is the evidence? Aust J Prim Health. 2015;21(1):2-8.  doi:10.1071/PY12164
  18. Coney   S.   Effective   Consumer   Voice   and   Participation for New Zealand:  A Systematic Review of the Evidence:  Discussion Document. Auckland: New Zealand Guidelines Group; 2004.
  19. Bolsewicz Alderman K, Hipgrave D, Jimenez-Soto E. Public engagement in health priority setting in low- and middle-income countries: current trends and considerations for policy. PLoS Med. 2013;10(8):e1001495. doi:10.1371/journal.pmed.1001495
  20. Kohler JC, Martinez MG. Participatory health councils and good governance: healthy democracy in Brazil? Int J Equity Health. 2015;14(1):21. doi:10.1186/s12939-015-0151-5
  21. O'Meara WP, Tsofa B, Molyneux S, Goodman C, McKenzie FE. Community and facility-level engagement in planning and budgeting for the government health sector--a district perspective from Kenya. Health Policy. 2011;99(3):234-243. doi:10.1016/j.healthpol.2010.08.027
  22. Meads GD, Griffiths FE, Goode SD, Iwami M. Lessons from local engagement in Latin American health systems. Health Expect. 2007;10(4):407-418. doi:10.1111/j.1369-7625.2007.00468.x
  23. Mitton C, Smith N, Peacock S, Evoy B, Abelson J. Public participation in health care priority setting: A scoping review. Health Policy. 2009;91(3):219-228. doi:10.1016/j.healthpol.2009.01.005
  24. Agency for Healthcare Research and Quality. Strategy 1: Working With Patients and Families as Advisors. http://www.ahrq.gov/professionals/systems/hospital/engagingfamilies/strategy1/index.html.   Accessed September 5, 2015.
  25. Jeppson ES, Thomas J. Essential Allies: Families as Advisors. Washington, DC: Institute for Family-Centered Care; 1995.
  26. Coulter A. Patient engagement-what works? J Ambul Care Manage. 2012;35(2):80-89. doi:10.1097/JAC.0b013e318249e0fd
  27. Conklin A, Morris Z, Nolte E. What is the evidence base for public involvement in health-care policy?: Results of a systematic scoping review. Health Expect. 2015;18(2):153-165. doi:10.1111/hex.12038
  28. Crawford MJ, Rutter D, Manley C, et al. Systematic review of involving patients in the planning and development of health care. Br Med J. 2002;325(7375):1263-1265. doi:10.1136/bmj.325.7375.1263
  29. Abelson J, Montesanti S, Li K, Gauvin F, Martin E. Effective Strategies for Interactive Public Engagement in the Development of Healthcare Policies and Programs. Ottawa, ON: Canadian Health Services Research Foundation; 2010.
  30. Corbin JM, Strauss AL, Strauss AL. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Thousand Oaks, CA: Sage Publications, Inc.; 2008.
  31. Crabtree BF, Miller WL. Doing Qualitative Research. Newbury Park, CA: Sage Publications, Inc.; 1992.
  32. Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23(4):334-340. doi:10.1002/1098-240X(200008)23:43.0.CO;2-G
  33. Metsch JM, Veney JE. Measuring the Outcome of Consumer Participation. J Health Soc Behav. 1973;14(4):368-374.
  34. Steckler A, Dawson L. Determinants of consumer influence in a health systems agency. Health Educ Monogr. 1978;6(4):378-393.
  35. Rifkin SB, Muller F, Bichmann W. Primary health care: On measuring participation. Soc Sci Med. 1988;26(9):931-940.
  36. Schmidt DH, Rifkin SB. Measuring participation: Its use as a managerial tool for district health planners based on a case study in Tanzania. Int J Health Plann Manage. 1996;11(4):345-358.
  37. Kent H, Read J. Measuring consumer participation in mental health services: Are attitudes related to professional orientation? Int J Soc Psychiatry. 1998;44(4):295-310. doi:10.1177/002076409804400406
  38. El Ansari AW, Phillips C.J. Interprofessional collaboration: A stakeholder approach to evaluation of voluntary participation in community partnerships. J Interprof Care. 2001;15(4):351-368. doi:10.1080/13561820120080481
  39. Shortell SM, Zukoski AP, Alexander JA, et al. Evaluating partnerships for community health improvement: Tracking the footprints. J Health Polit Policy Law. 2002;27(1):49-91. doi:10.1215/03616878-27-1-49
  40. Halliday J, Asthana SNM, Richardson S. Evaluating partnership: the role of formal assessment tools. Evaluation. 2004;10(3):285-303. doi:10.1177/1356389004048279
  41. Jarrett L, Patient Involvement Unit. A report on a study to evaluate patient/carer membership of the first NICE Guideline Development Groups. London: National Institute for Clinical Excellence (NICE); 2004. Accessed September 5, 2015.
  42. Institute for Family-Centered Care. Patient- and Family-Centered Care: A Hospital Self-Assessment Inventory. Bethesda, MD: Institute for Family-Centered Care; 2004. Accessed September 5, 2015.
  43. South J, Fairfax P, Green E. Developing an assessment tool for evaluating community involvement. Health Expect. 2005;8(1):64-73. doi:10.1111/j.1369-7625.2004.00313.x
  44. Grant J. The participation of mental health service users in Ontario, Canada: A Canadian Application of the Consumer Participation Questionnaire. Int J Soc Psychiatry. 2007;53(2):148-158. doi:10.1177/0020764006074557
  45. Health and Social Care Regulatory Forum. Framework for Public and Service User Involvement in Health and Social Care Regulation in Ireland. http://www.mhcirl.ie/File/Framework_Pub_SU.pdf.  Accessed September 5, 2015.
  46. Draper AK, Hewitt G, Rifkin S. Chasing the dragon: Developing indicators for the assessment of community participation in health programmes. Soc Sci Med. 2010;71(6):1102-1109. doi:10.1016/j.socscimed.2010.05.016
  47. Consumer Health Quality Council Health Care For All, Massachusetts. Health Care for All: Patients and Families Improving Care. Patient and Family Advisory Councils: A Review of 2011 PFAC Reports. http://www.ipfcc.org/bestpractices/Review-of-PFAC-2011-Reports.pdf.  Accessed September 5, 2015.
  48. National Institute for Children's Health Quality. Creating a Patient and Family Advisory Council: A Toolkit for Pediatric Practices. http://www.nichq.org/~/media/files/resources/pfac%20toolkit/pfac_resources.ashx.  Accessed September 5, 2015.
  49. Health Care for All, Massachusetts. PFAC 2014. A Review of 2013 Massachusetts Patient & Family Advisory Council Reports. https://www.hcfama.org/sites/default/files/2014_pfac_report_final.pdf.  Accessed September 5, 2015.
  50. Brigham and Women’s Hospital Center for Patients and Families. Patient and Family Advisory Council (PFAC) Report Submitted September 30, 2014. https://www.hcfama.org/sites/default/files/brigham_and_womens_dph_report_sept_2014.pdf.  Accessed September 5, 2015.
  51. Abelson J, PPEET Research-Practice Collaborative. The Public and Patient Engagement Evaluation Tool. https://fhs.mcmaster.ca/publicandpatientengagement/ppeet.html.  Accessed September 5, 2015.
  52. Consumer Confidence Project Steering Committee. Consumer Confidence Project: Report of Pilot Results. http://www.mentalhealthportland.org/wp-content/uploads/2015/08/Consumer-Confidence-Project-Report-of-Pilot-Results.pdf.  Accessed September 5, 2015.
  53. Shortell SM, Sehgal NJ, Bibi S, et al. An early assessment of accountable care organizations’ efforts to engage patients and their families. Med Care Res Rev. 2015;72(5):580-604. doi:10.1177/1077558715588874
  54. Rowe G, Frewer LJ. Evaluating public-participation exercises: a research agenda. Sci Technol Hum Val. 2004;29(4):512-556. doi:10.1177/0162243903259197
  55. Abelson J. Understanding the role of contextual influences on local health-care decision making: case study results from Ontario, Canada. Soc Sci Med. 2001;53(6):777-793. doi:10.1016/S0277-9536(00)00386-5
  56. Arnstein SR. A Ladder of Citizen Participation. J Am Inst Plann. 1969;35(4):216-224. doi:10.1080/01944366908977225
  57. Rowe G, Marsh R, Frewer LJ. Evaluation of a Deliberative Conference. Sci Technol Hum Val.  2004;29(1):88-121. doi:10.1177/0162243903259194
  58. Tenbensel T. Interpreting public input into priority-setting: the role of mediating institutions. Health Policy. 2002;62(2):173-194.
  59. Abelson J, Forest PG, Eyles J, Smith P, Martin E, Gauvin FP. Deliberations about deliberative methods: issues in the design and evaluation of public participation processes. Soc Sci Med. 2003;57(2):239-251. doi:10.1016/S0277-9536(02)00343-X
  60. Worley CG, Mohrman SA, Nevitt JA. Large group interventions: An empirical field study of their composition, process, and outcomes. J Appl Behav Sci. 2011;47(4):404-431. doi:10.1177/0021886311410837
  61. Abelson J, Li K, Wilson G, Shields K, Schneider C, Boesveld S. Supporting quality public and patient engagement in health system organizations: development and usability testing of the public and patient engagement evaluation tool. Health Expect. 2016;19(4):817. doi:10.1111/hex.12378
  62. Werlin SH, Walcott A, Joroff M. Implementing formative health planning under PL 93-641. N Engl J Med. 1976;295(13):698-703. doi:10.1056/NEJM197609232951304

Files

History

  • Receive Date: 04 October 2017
  • Revise Date: 26 April 2018
  • Accept Date: 28 April 2018
  • First Publish Date: 01 October 2018

Share

How to cite

Statistics