Real World Test Plan

Under the ONC Health IT Certification Program, Health IT Developers are required to conduct

Real World Testing of their Certified Health IT.

Year 2024

Background & Instructions

Under the ONC Health IT Certification Program (Program), Health IT Developers are required to conduct Real World Testing of their Certified Health IT (45 CFR 170.556 and 170.523(i)). The Office of the National Coordinator for Health Information Technology (ONC) issues Real World Testing resources to clarify Health IT Developers’ responsibilities for conducting Real World Testing, to identify topics and specific elements of Real World Testing that ONC considers a priority, and to assist Health IT Developers to develop their Real World Testing plans.

Health IT Developers have maximum flexibility to develop innovative plans and measures for Real World Testing. As developers are planning for how they will execute Real World Testing, they should consider the overall complexity of the workflows and use cases within the care settings in which they market their Certified Health IT to determine which approaches they will take. This Real World Testing plan template was created to assist Health IT Developers in organizing the required information that must be submitted for each element in their Real World Testing plan. Health IT Developers must submit one plan for each year of Real World Testing (see resources listed below for specific timelines and due dates). ONC does not encourage updating plans outside the submission timeline and will not post updates on the Certified Health IT Product List (CHPL). If adjustments to approaches are made throughout Real World Testing, the Health IT Developer should reflect these adjustments in their Real World Testing results report. ONC would expect that the Real World Testing results report will include a description of these types of changes, the reasons for them, and how intended outcomes were more efficiently met as a result. This resource should be read and understood in conjunction with the following companion resources, which describe in detail many of the Program requirements referenced in this resource.

Health IT Developers should also review the following regulatory materials, which establish the core requirements and responsibilities for Real World Testing under the Program.

21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program final rule, 85 FR 25642 (May 1, 2020) (Century Cures final rule). Section VII.B.5 — “Real World Testing”

General Information

Plan Report ID Number: [For ONC-Authorized Certification Body use only]

Developer Name: EDENLAB OÜ

Product Name(s): Kodjin FHIR Server

Version Number(s): Version 3

Certified Health IT Product List (CHPL) ID(s): 15.04.04.3148.Kodj.03.00.0.230330

Developer Real World Testing Page URL: 

Justification for Real World Testing Approach

Consistent with the ONC’s recommendation that “Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange”, this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.

It is important to note that Real World Testing is only one component of the Health IT Certification program used to demonstrate compliance with the program requirements. Real World Testing should augment and support testing that was conducted prior to certification being granted. It is not intended to duplicate the methods or results previously demonstrated. Instead, this test plan was developed to demonstrate that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.

Kodjin FHIR Server doesn’t have narrow care settings, it can be used by any practices. All measures reasonably align with the §170.315(g)(10) certification criterion.

Standards Updates (Including Standards Version Advancement Process-SVAP and USCDI)

EDENLAB OÜ has not updated Kodjin FHIR Server to any new standards as part of SVAP or the Cures Update criteria as of this date nor plan to prior to the execution of the 2024 Real World Test.

Care Settings

Kodjin FHIR Server doesn’t have narrow care settings, it can be used by any practices.

Measures Used In Overall Approach

For each measurement/metric, describe the elements below: 

  • Associated certification criteria 
  • Description of each measurement/metric 
  • Justification for each measurement/metric 
  • Expected Outcomes for each measurement/metric 
  • Care Settings Targeted will be ambulatory care settings We are going to use de-identified Kodjin FHIR Server logs to collect all necessary data and build the report.

Measure 1 

Associated Certification Criterion: §170.315(g)(10) Standardized API for patient and population services Measurement: Count of FHIR API requests by smart apps and error rate Description: Using FHIR API by smart apps proves (i)(A) Respond to requests for a single patient’s data according to the standard adopted in § 170.215(a)(1) and implementation specification adopted in § 170.215(a)(2) Justification: Demonstrating the ability Kodjin FHIR Server module to respond on (i)(A) requests Expected Outcomes: It is expected that the FHIR API will be conformant to § 170.315(g)(10) with less than 1 percent error rate experienced by users.

Measure 2 

Associated Certification Criterion: §170.315(g)(10) Standardized API for patient and population services Measurement: Count of bulk export requests and error rate Description: Using bulk export operation proves (i)(B) Respond to requests for multiple patients’ data as a group according to the standard adopted in § 170.215(a)(1) Justification: Demonstrating the ability Kodjin FHIR Server module to respond on (i)(B) requests Expected Outcomes: We do not expect this API often used, so we do not expect any errors here and will review them if they will occur.

Measure 3

Associated Certification Criterion: §170.315(g)(7) Application access — patient selection.
Metrics:
  1. 1. Number of requests for a patient ID or token
  2. 2. Number of requests that provided sufficient information to provide a valid response
  3. 3. Number of follow-up requests made using the provided patient ID or token
Justification: This criterion requires the certified Health IT module to provide an API and supporting documentation that enables external applications to request a unique patient identifier from the certified Health IT module that can be used to request additional patient data. We intend to record the frequency that patient ID requests received by providers via API to demonstrate the certified capability is available and effective, regardless of the frequency it is used. Our expectation is there will be low utilization by providers with a high success rate.

Schedule of Key Milestones

Develop a list of clients to assist with Real World Testing: Q1 2024

Preparation and scheduling testing with clients: Q2 2024

Collection of information as laid out by the plan: Q2 and Q3 2024

Real World Testing plan will be completed according to ONC and ONC-ACB requirements and expectations: Q4 2024

Document our 2024 test results: January 2025

Submit Real World Testing Report to ONC-ACB: February 2025

Attestation

This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.

Authorized Representative Name: Andrew Krylov

Authorized Representative Email: [email protected]

Authorized Representative Phone: +380508818737

Authorized Representative Signature: Date: 20/10/2023

Let`s chat

We would be glad to share more details about our enterprise-level FHIR software solutions and other cases based on the HL7 FHIR standard.

    Your form has been submitted successfully

    We will contact your shortly

    Kodjin White Paper

    Please leave your email to get Kodjin White Paper

      By downloading files from this site you agree to the Policy

      The Kodjin White Paper has been successfully sent to your email

      We have sent a copy to your email

      Back to website content