Barriers to comparing the usability of electronic health records

Corresponding Author: Raj Ratwani, PhD, National Center for Human Factors in Healthcare, MedStar Health, 3007 Tilden St, Suite 7M, Washington, DC 20009, USA. E-mail: raj.ratwani@medicalhfe.org; Tel: 202-244-9815

National Center for Human Factors in Healthcare, MedStar Health, and Department of Emergency Medicine, Georgetown University School of Medicine, Washington, DC

National Center for Human Factors in Healthcare, MedStar Health, and Department of Emergency Medicine, Georgetown University School of Medicine, Washington, DC

Journal of the American Medical Informatics Association, Volume 24, Issue e1, April 2017, Pages e191–e193, https://doi.org/10.1093/jamia/ocw117

Cite

Raj M Ratwani, A Zachary Hettinger, Rollin J Fairbanks, Barriers to comparing the usability of electronic health records, Journal of the American Medical Informatics Association, Volume 24, Issue e1, April 2017, Pages e191–e193, https://doi.org/10.1093/jamia/ocw117

Despite the widespread adoption of electronic health records (EHRs), usability of many EHRs continues to be suboptimal, with some vendors failing to meet usability standards, resulting in clinician frustration and patient safety hazards. In an effort to increase EHR vendor competition on usability, recommendations have been made and legislation drafted to develop comparison tools that would allow purchasers to better understand the usability of EHR products prior to purchase. Usability comparison can be based on EHR vendor design and development processes, vendor usability testing as part of the Office of the National Coordinator for Health Information Technology certification program, and usability of implemented products. Barriers exist within the current certified health technology program that prevent effective comparison of usability during each of these stages. We describe the importance of providing purchasers with improved information about EHR usability, barriers to making usability comparisons, and solutions to overcome these barriers.

INTRODUCTION

The usability of electronic health records (EHRs) continues to be a source of front-line clinician frustration and increased workload, and presents patient safety hazards. 1 , 2 In an effort to improve the usability of EHRs, recent legislation drafted by the US Senate Health, Education, Labor, and Pensions Committee and recommendations from the federal Office of the National Coordinator for Health Information Technology (ONC) Certified Technology Comparison Task Force have highlighted the need for EHR usability comparison tools. 3 , 4 Product comparison tools would support market forces by better informing purchasers about the usability of EHRs prior to making a purchase decision, and would consequently push EHR vendors to compete more on usability. Although many providers have already adopted EHR technology, there has been an increase in the number of providers switching EHR products. 5 However, until robust EHR usability comparison tools exist, purchasers do not have the usability information they need to inform their decision-making. Thus, vendors may be less likely to aggressively compete on usability, since it is not a major factor in the purchasing process.Recognizing the importance of usability, the ONC enacted certification requirements to promote the usability of EHRs through a safety enhanced design certification requirement. 6 The certification program requires vendors to provide a statement attesting to use of a user-centered design (UCD) process, which puts the needs of users at the core of the design and development process, and they are also required to conduct a formal usability test, called a summative test, on their products once development is complete. Both the vendor attestation and the usability testing results are publicly available on the ONC’s certified health product list and are a starting point for examining EHR vendor usability processes. However, our analysis of these reports suggests that many vendors are not adhering to the certification requirements. 7 , 8 An examination of 41 vendor reports showed that 34% did not contain a statement of the UCD process as required by the ONC, yet the products were still certified. 7 Most purchasers are unaware of the lack of vendor adherence. The impact of the ONC’s certification program on influencing EHR vendor usability processes is limited and underscores the need for alternative methods to promote EHR usability.

Critics of the ONC’s certification program have said that the market, not regulation, should drive usability. However, the ability of market forces to impact EHR usability are significantly impaired for 3 reasons: (1) Usability has the most dramatic effect on front-line clinicians, who often have the least influence on the purchasing decision; (2) EHR usability is dependent on both the EHR product design and implementation decisions; and (3) most important, there are no tools to allow comparison of the usability of EHR products prior to purchase. It is this last reason that is most easily addressed to help both health care organizations make more informed decisions and EHR vendors with highly usable products gain recognition for their significant investments, as well as place pressure on those vendor products with poor usability.

Usability comparisons can be focused on different aspects of EHR design, development, and use, including the vendor UCD process, vendor usability testing, and usability of the product as used by front-line clinicians. For each of these areas, major barriers exist within the current regulatory framework preventing effective comparison. Without changes to the certification program and a major investment in facilitating the development of comparison tools, the initiatives to compare EHR usability are unlikely to be effective. We describe the major barriers to EHR usability comparison and propose solutions to overcome those barriers.

REVIEW OF BARRIERS TO EHR USABILITY COMPARISON

Barriers to comparing the user-centered design process

Although many mistakenly believe that usability is simply determined by the design of the visual display, or user interface, a rigorous user-centered design process is based on a deep understanding of how front-line clinicians conduct their cognitive and task-oriented work, and leveraging this knowledge to guide design and development of the product. 9 Engaging in a rigorous UCD process lays the foundation for a product that supports the needs of users. 10 There are several UCD process guidelines from different standards organizations to guide vendor design and development. 11 , 12 Comparing the UCD processes of different vendors is one method for gaining insight into the investments vendors are making in UCD and how well they are executing the UCD processes they employ.

Within the current regulatory framework, ONC certification requires each vendor to attest to using a UCD process by providing a written statement describing the process. However, the attestation alone does not lend itself to a meaningful comparison of how the UCD process was actually utilized during design and development. For example, 2 vendors may both attest that they follow National Institute for Standards (NIST) internal/interagency report 7741, a common UCD process method. 11 However, vendor A may have conducted numerous iterations of early user testing with a diverse set of clinicians, while vendor B may have conducted a single iteration of testing with a small sample of nonmedical users. These differences in UCD process rigor are not evident in the certification attestations provided by vendors, making it impossible to compare how UCD is being used across vendors.

To overcome this barrier to comparison, the certification requirements could be modified to require that vendors provide evidence of their UCD process. The evidence would provide insight into the rigor of the UCD process, as is done by the Food and Drug Administration with the testing of medical devices, and this information would support comparison across vendors. 13 The byproducts of utilizing a UCD process can serve as evidence that could reduce any burden on vendors to report this information. The evidence of a rigorous UCD process might include user stories, personas, use test cases, and documentation of user feedback.

Barriers to comparing vendor usability test results

Since vendors are required to conduct summative usability tests, one might hope that comparing metrics from these tests could serve the purpose of providing purchasers with greater insight on the usability of the products. The metrics from these tests include error rates along with measures of user efficiency, effectiveness, and satisfaction with the product being tested. These usability tests are intended to serve as a final safety check once the product has been designed and developed.

There are, however, several barriers to comparing the summative test results from vendors. There are recommended testing scenarios, but no standard testing scenarios are required for certification. Consequently, each vendor creates its own testing scenarios, which prevents effective comparison of test results. Participants in the usability tests are completing entirely different tasks, despite using the same feature, such as computerized provider order entry. For example, a vendor may use a very complex test scenario where a physician must place multiple medication orders, cancel 1 of the orders, and place a new order, while another vendor may use a simple scenario of having the physician place 1 straightforward medication order. The resulting error rates and task times between these 2 test scenarios cannot be fairly compared. In addition, the certification requirements only set a minimum number of participants and do not stipulate the background of participants, which also prevents meaningful comparison. Finally, some vendors are not adhering to the required metrics, which prevents direct comparison across vendors. 7

To overcome the barriers to comparing the EHR certification testing results, standard test scenarios would need to be adopted by each vendor, or a third party could conduct summative tests using a standard set of test scenarios. A standard number of participants, with specific backgrounds representative of the typical end users of the feature, and standard metrics would need to be established. Current certification requirements could be modified to overcome this barrier.

Barriers to comparing the usability of products post-implementation

Comparing the usability of EHR products as they are actually used by front-line clinicians would provide insight on which products are able to best support the needs of clinicians in context. Currently, there are survey-based comparisons that rely on clinician feedback or information technology leaders’ perceptions of the usability of implemented EHRs; however, there are no objective test-based assessments of implemented EHRs. While survey-based comparisons provide some insight, these methods are not formal assessments and often underrepresent actual usability challenges.

Because EHRs often go through an intense phase of customization and configuration to integrate with other clinical systems during implementation, the same EHR product used at different provider sites is often dramatically different. These differences are a major barrier to comparing the usability of products. Usability testing at a provider site will reflect all the unique customization choices made by that organization. Testing the same vendor product at 2 different providers would likely result in different measures of effectiveness, efficiency, and satisfaction. Consequently, it would be difficult to attribute usability shortcomings directly to the vendor; rather, the shortcomings reflect the joint choices about the EHR product made between the health care organization and the EHR vendor. In addition, gaining access to conduct rigorous usability tests in the clinical environment would require tremendous investment and coordination.

The options for overcoming the barriers to comparing the usability of implemented products are limited. For vendors that limit customization or are prescriptive during the implementation process by providing best practices, more meaningful usability testing comparisons could be conducted, since the outcomes of the testing would reflect standard vendor practices. One possibility is to compare user interaction data that many vendors collect during routine use of their products, such as time to place a medication order. The validity of this method would need to be determined. In addition, the ONC’s proposed efforts to improve post-market surveillance of EHR products may also support improved comparison of vendor products. 14

CONCLUSION

The current regulatory framework and implementation process prevent meaningful comparison of usability design processes, usability certification testing results, and usability testing of implemented products as used by front-line clinicians. These barriers may be one reason why third-party rating agencies are not as prominent here as in other industries. Some of the identified barriers can be overcome by making modifications to current certification requirements, and the ONC’s 2015 certification requirements are a step in the right direction, with the establishment of a 10-participant minimum for usability testing and a recommended set of testing scenarios provided by NIST. However, further evolution of the requirements is necessary and would benefit from requiring vendors to provide evidence of their UCD process, which would allow for a more accurate assessment of the rigor of that process. To facilitate comparison of the certification usability testing results, a requirement for common testing scenarios, an established number of participants and participant backgrounds, and common metrics must be established.Developing EHR usability comparison tools is a critical step toward supporting more informed purchaser decisions and increasing vendor competitiveness around usability. In addition to influencing purchase decisions, usability comparison tools may also bring awareness to vendor reputations and become a major point of competition for vendors, resulting in improved products. Overcoming the barriers described here will require cooperation from numerous stakeholders, including the ONC, EHR vendors, and providers. Until free market conditions can push vendors to aggressively compete on usability, ONC certification requirements will remain the primary lever for promoting usability, even though vendor adherence to these requirements remains subpar. 7

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector.

COMPETING INTERESTS

The authors have no competing interests to declare.

CONTRIBUTORS

R.R. led the conceptualization of the manuscript and the first draft. A.Z.H. and R.J.F. made contributions to the conceptualization and to revisions of the manuscript.