Under the ONC Health IT Certification Program, Health IT Developers are required to conduct
Real World Testing of their Certified Health IT
Year 2024
Background & Instructions
Under the ONC Health IT Certification Program (Program), Health IT Developers are required to conduct Real World Testing of their Certified Health IT (45 CFR 170.556 and 170.523(i)). The Office of the National Coordinator for Health Information Technology (ONC) issues Real World Testing resources to clarify Health IT Developers’ responsibilities for conducting Real World Testing, to identify topics and specific elements of Real World Testing that ONC considers a priority, and to assist Health IT Developers to develop their Real World Testing plans.
Health IT Developers have maximum flexibility to develop innovative plans and measures for Real World Testing. As developers are planning for how they will execute Real World Testing, they should consider the overall complexity of the workflows and use cases within the care settings in which they market their Certified Health IT to determine which approaches they will take. This Real World Testing plan template was created to assist Health IT Developers in organizing the required information that must be submitted for each element in their Real World Testing plan. Health IT Developers must submit one plan for each year of Real World Testing (see resources listed below for specific timelines and due dates). ONC does not encourage updating plans outside the submission timeline and will not post updates on the Certified Health IT Product List (CHPL). If adjustments to approaches are made throughout Real World Testing, the Health IT Developer should reflect these adjustments in their Real World Testing results report. ONC would expect that the Real World Testing results report will include a description of these types of changes, the reasons for them, and how intended outcomes were more efficiently met as a result. This resource should be read and understood in conjunction with the following companion resources, which describe in detail many of the Program requirements referenced in this resource.
Health IT Developers should also review the following regulatory materials, which establish the core requirements and responsibilities for Real World Testing under the Program.
21st Century Cures Act: Interoperability, Information Blocking, and the ONC Health IT Certification Program final rule, 85 FR 25642 (May 1, 2020) (Century Cures final rule). Section VII.B.5 — “Real World Testing”
General Information
Plan Report ID Number: [For ONC-Authorized Certification Body use only]
Developer Name: EDENLAB OÜ
Product Name(s): Kodjin FHIR Server
Version Number(s): Version 3
Certified Health IT Product List (CHPL) ID(s): 15.04.04.3148.Kodj.03.00.0.230330
Developer Real World Testing Page URL:
Justification for Real World Testing Approach
Consistent with the ONC’s recommendation that “Real World Testing verify that deployed Certified Health IT continues to perform as intended by conducting and measuring observations of interoperability and data exchange”, this test plan focuses on capturing and documenting the number of instances that certified capability is successfully utilized in the real world. In instances where no evidence exists due to zero adoption of a certified capability or the inability to capture evidence of successful use for other reasons, we will demonstrate the required certified capability in a semi-controlled setting as close to a “real world” implementation as possible.
It is important to note that Real World Testing is only one component of the Health IT Certification program used to demonstrate compliance with the program requirements. Real World Testing should augment and support testing that was conducted prior to certification being granted. It is not intended to duplicate the methods or results previously demonstrated. Instead, this test plan was developed to demonstrate that the certified capabilities have been successfully deployed for providers to use at their discretion in live settings.
Kodjin FHIR Server doesn’t have narrow care settings, it can be used by any practices. All measures reasonably align with the §170.315(g)(10) certification criterion.
Standards Updates (Including Standards Version Advancement Process-SVAP and USCDI)
EDENLAB OÜ has not updated Kodjin FHIR Server to any new standards as part of SVAP or the Cures Update criteria as of this date nor plan to prior to the execution of the 2024 Real World Test.
Care Settings
Kodjin FHIR Server doesn’t have narrow care settings, it can be used by any practices.
Measures Used In Overall Approach
For each measurement/metric, describe the elements below:
- Associated certification criteria
- Description of each measurement/metric
- Justification for each measurement/metric
- Expected Outcomes for each measurement/metric
- Care Settings Targeted will be ambulatory care settings We are going to use de-identified Kodjin FHIR Server logs to collect all necessary data and build the report.
Measure 1
Associated Certification Criterion: §170.315(g)(10) Standardized API for patient and population services Measurement: Count of FHIR API requests by smart apps and error rate Description: Using FHIR API by smart apps proves (i)(A) Respond to requests for a single patient’s data according to the standard adopted in § 170.215(a)(1) and implementation specification adopted in § 170.215(a)(2) Justification: Demonstrating the ability Kodjin FHIR Server module to respond on (i)(A) requests Expected Outcomes: It is expected that the FHIR API will be conformant to § 170.315(g)(10) with less than 1 percent error rate experienced by users.
Measure 2
Associated Certification Criterion: §170.315(g)(10) Standardized API for patient and population services Measurement: Count of bulk export requests and error rate Description: Using bulk export operation proves (i)(B) Respond to requests for multiple patients’ data as a group according to the standard adopted in § 170.215(a)(1) Justification: Demonstrating the ability Kodjin FHIR Server module to respond on (i)(B) requests Expected Outcomes: We do not expect this API often used, so we do not expect any errors here and will review them if they will occur.
Measure 3
- 1. Number of requests for a patient ID or token
- 2. Number of requests that provided sufficient information to provide a valid response
- 3. Number of follow-up requests made using the provided patient ID or token
Schedule of Key Milestones
Develop a list of clients to assist with Real World Testing: Q1 2024
Preparation and scheduling testing with clients: Q2 2024
Collection of information as laid out by the plan: Q2 and Q3 2024
Real World Testing plan will be completed according to ONC and ONC-ACB requirements and expectations: Q4 2024
Document our 2024 test results: January 2025
Submit Real World Testing Report to ONC-ACB: February 2025
Attestation
This Real World Testing plan is complete with all required elements, including measures that address all certification criteria and care settings. All information in this plan is up to date and fully addresses the health IT developer’s Real World Testing requirements.
Authorized Representative Name: Andrew Krylov
Authorized Representative Email: [email protected]
Authorized Representative Signature:
Date: 20/10/2023
