How to Perform Clinical Evaluation of Medical Software

Manufacturers must clinically evaluate a medical device before they can market it in Europe. To do so, the manufacturer has to examine clinical data in order to investigate whether the medical device is safe and performing. The clinical evaluation of a medical device extends over the entire product life cycle and is a complex process. It begins with initial explorative studies and product ideas and reaches all the way to the storage of regulatory documents. The manufacturer can use the results to assess whether the risks associated with the use of the medical device are appropriate to the expected benefit.The clinical evaluation of software is based on the same legal requirements as the clinical evaluation of all medical devices. In this article, we explain what you need to consider when clinically evaluating medical software.

Legal Basis for Medical Devices in Europe

The clinical evaluation of medical devices according to the EU MDR is a mandatory process for the manufacturer (Art. 61 and Annex XIV Part A). It aims to examine and evaluate clinical data to verify the clinical safety and performance of the medical device. Obviously, the kind and extent of the clinical evaluation depend on the intended purpose defined by the manufacturer. Furthermore, it includes a post-marketing clinical follow-up (PMCF, Art. 61 (11) and Annex XIV Part B).
The manufacturer must keep the clinical evaluation up to date throughout the entire life cycle. He must update the PMCF assessment report for Class III and implantable medical devices annually.

Clinical Data

Clinical data is any “information concerning safety or performance that is generated from the use of a device” (Art. 2 (48)). It can come from the following sources:
  • clinical trial of the device concerned or of an equivalent device,
  • clinical experience with the device concerned or an equivalent device, or
  • PMCF.
Note that the EU MDR calls a clinical trial “clinical investigation”.Manufacturers can use observations from post-market surveillance (e.g. vigilance data) as well as results from PMCF studies as additional sources for clinical evaluation. For this reason, some manufacturers have already started to collect data for their devices from other markets such as the US. This includes data from Investigator Initiated Studies (IIS). Whether these data are sufficient to demonstrate the performance, clinical safety and clinical benefit of a device depends on the quality and significance of the data. The manufacturer must conduct clinical studies (e.g. PMCF studies) if the available clinical data is insufficient.
Manufacturers should link and keep up to date risk management, quality management and clinical evaluation procedures on a regular basis.
Note that pre-clinical and clinical data must be part of the technical documentation (Annex II (6.1.)) including:

What Does Equivalence to Marketed Devices Mean?

The manufacturer can demonstrate equivalence to already marketed medical devices. Annex XIV (3) of EU MDR specifies 3 characteristics, that manufacturers must consider when demonstrating equivalence:
  • technical (e.g. conditions of use, properties and algorithms),
  • biological (not applicable for software) and,
  • clinical (e.g. clinical condition or purpose, population and performance).
To demonstrate equivalence, the characteristics of the compared devices must match. There must be no clinically significant difference in the safety and clinical performance of the devices. It is therefore important that the demonstration is based on adequate scientific justification. Additionally, the manufacturer must clearly show that he has sufficient access to the comparator data.

Practical Implementation of Equivalence

Obviously, the manufacturer has a considerable scope for interpretation as to how he can practically implement these requirements of the EU MDR. For instance, the manufacturer may refer to the clinical evaluation of equivalent devices from other manufacturers. What extent of data of medical devices from other manufacturers does he then need?Annex XIV (3) EU MDR, for instance, refers to non-implantable devices of risk classes I – IIb. It specifies, that manufacturers must provide clear evidence that they have sufficient access to the data of the device with which they claim equivalence. Art. 61 para. 5, in contrast, refers to implantable devices or devices in risk class III. It says that two manufacturers need a contract explicitly allowing the manufacturer of the second device full access to the technical documentation on an ongoing basis.How manufacturers can deal with the difference between “sufficient access” and “full access to technical documentation” in practice is not clear. However, the clinical evaluation of the equivalent device must comply with EU MDR requirements. The EU guidance document MEDDEV 2.7/1 Rev. 4 provides further information on demonstrating equivalence (see Appendix 1). However, this document refers to the “old” EU MDD and AIMDD.If the clinical evaluation reveals gaps, the manufacturer can conduct a clinical trial to generate the missing data. The clinical trial may be designed to demonstrate the comparability of the clinical safety and performance of the device with the device to be compared.

When Is a Clinical Trial Required?

In any case, the manufacturer must conduct a clinical trial if he:
  • launches completely new software with new features and functionality,
  • he has modified an existing software in such a way that clinical safety and efficacy may be affected, or
  • he uses an existing software for a new medical purpose.
In general, manufacturers must conduct a clinical trial when it comes to high risk medical devices like class III and implantable devices (Art. 61 (4)). Furthermore, the manufacturer may consult an expert panel (Art. 106) in case of class III and class IIb active devices intended to administer or remove a medicinal product (Art. 61 (2)). The expert panels give advice in product development and assist the EU Commission in the preparation of guidelines and common specifications.There is one exception concerning the high-risk devices mentioned above. If the clinical evaluation of a medical device marketed under the EU MDD is based on “sufficient clinical data”, no clinical study is required (Art. 61 para. 6). However, there is currently no general definition of “sufficient clinical data”, as their kind and extent are highly dependent on the type of medical device. In the future there will probably be EU-wide evaluation criteria. Respective guidelines or common specifications should be device-specific.

Guidance Documents

At present the two guidance documents MEDDEV 2.7/1 Rev. 4 from the European Commission and N41 for the clinical evaluation of stand-alone software from the International Medical Device Regulators Forum (IMDRF) provide supporting information for manufacturers and notified bodies regarding clinical evaluation. Both guidance documents are legally not binding and the MEDDEV 2.7/1 Rev. 4 still refers to the EU MDD and AIMDD. Nevertheless, for the time being we recommend that manufacturers use these documents.

The Clinical Evaluation Report

Annex 9 of MEDDEV 2.7/1 Rev. 4 proposes the content and structure of the clinical evaluation report (CER):
  • scope of the clinical evaluation and description of the medical device,
  • state-of-the-art in science and technology as well as clinical background, and
  • route of clinical evaluation/device under evaluation.
The device description contains the intended purpose, all relevant market data, and the implemented innovations during the product cycle. The state-of-the-art describes the medical device in the context of other medical devices and the underlying medical condition. Finally, the manufacturer must describe the chosen route of clinical evaluation. This could be the so-called “literature route” which uses existing data. It includes demonstration and analysis of device equivalence, device-specific data from preclinical and clinical studies or a combination of both. For medical software, manufacturers can also use technical tests and add comparable studies.The challenge for medical software manufacturers is to find starting points for complying with basic performance and safety requirements. Here, N41 introduces the following three main elements for the clinical evaluation of stand-alone software (or “Software as Medical Device” (SaMD) according to IMDRF).

Valid Clinical Association

First, there must be a valid clinical association between the output of the software and the clinical condition, disease, injury, or disability. Therefore, the manufacturer must carry out a comprehensive literature study by searching the relevant databases (e. g. PubMed) for systematic and non-systematic reviews as well as meta-analyses. Moreover, he can use guidelines of medical societies as a source.Here the key questions are:
  • Does the output of the medical software correspond to the current state of science and technology regarding its intended purpose?
  • Is the functionality of the software of clinical relevance in medical practice?
All results of the valid clinical association examination belong to the section “state-of-the-art in science and technology as well as clinical background” of the clinical evaluation report.

Analytical Validation

Second, the manufacturer must validate his device analytically or technically by carrying out validation and verification tests as well as comparative tests in the case of device equivalence. In addition, the device must comply with the relevant standards.Finally, the manufacturer has to demonstrate the risk-based approach in the clinical evaluation by addressing software risks, e. g. caused by incorrect data input, insufficient precision of output, inadequate upper/lower limits, technical failure of e.g. mobile devices or the software environment, missing operability, and compatibility.Here the key questions are:
  • Is the software algorithm working correctly and reliably?
  • Are all specifications fulfilled regarding the intended purpose?
Standard IEC 82304-1 should be harmonized under the EU MDR in the future and can be used for validation of stand-alone software. The results belong to the section “route of clinical evaluation/device under evaluation” of the clinical evaluation report.

Clinical Validation

Third, the manufacturer must measure the ability of the software to yield a clinically meaningful output. Clinical trials conducted with the own medical device or an equivalent one, validation and verification, post-market surveillance and usability studies are possible.Here, the key questions are:
  • Does the desired (precise and reliable) output of the medical software meets the intended purpose in the target population?
  • Does the output have the desired clinical significance for the corresponding patient population?
  • Is the result of the algorithm relevant to the diagnosis, treatment or prevention of a disease in a patient?
All results of the clinical validation belong to the section “route of clinical evaluation/device under evaluation” of the clinical evaluation report.Consider that the corresponding IMDRF guidance introduces a risk categorization for stand-alone software. These categories are different from the medical device risk classes of the EU MDR and the software safety classification of standard EN 62304. However, the EU MDR and the harmonized standards are finally relevant for placing medical devices on the market in the EU.

Documentation

The overall documentation of the clinical evaluation is getting more complex according to EU MDR. It requires the following documents:
  • clinical evaluation plan (CEP, Annex XIV, Part A, 1.) describing the procedure for clinical evaluation to demonstrate the benefit-risk ratio based on the state of the art,
  • clinical evaluation report (CER, Art 61 (12)),
  • post-market surveillance plan (PMS, Art. 84, Annex III) covering proactive and passive activities to collect and analyze market-related experience including disclosure of methods and protocols to manage those post-market activities,
  • post-market clinical follow-up (PMCF, Annex XIV, Part B) or a justification as to why a PMCF is not applicable (outlined in the PMS plan),
  • post-market surveillance (PMS report for class I devices which summarizes the results and conclusions of the post-market surveillance data,
  • periodic safety update report for classes IIa, IIb and class III devices (PSUR, Art. 86), and
  • clinical evaluation assessment report (CEAR, Annex VII section 4.6; to be compiled by the notified body).
At present, there are no official templates for the documentation of the clinical evaluation of medical devices. Here it is worth looking at the corresponding documents of the pharmaceutical field.
The clinical evaluation report (CER) compiles the conclusions of the clinical evaluation. In addition, the manufacturer must provide a publicly available “Summary of safety and clinical performance” (Art. 32) for certain high-risk devices. The notified body in turn documents the results of the clinical evaluation assessment in the clinical evaluation assessment report (CEAR). In the case of certain high-risk devices the expert panel, the competent authorities, the authority responsible for notified bodies and the EU Commission have these documents at their disposal.The manufacturer commits himself to the collection of post-market surveillance data in the PMS plan. The periodic safety update report (PSUR) summarizes this data and contain among others also the main findings of the post-market clinical follow-up (PMCF). Notably, the EU MDR does not say something about the frequency of clinical evaluation updates. MEDDEV 2.7/1 Rev. 4 includes such information.

Our Recommendations

  • Monitor the implementing legislation of the EU commission and guidance documents released by the Medical Device Coordination Group (MDCG).
  • Constantly watch the market for existing and new equivalent medical devices and have a respective process in place. Are new findings (clinical study and scientific results) published in professional journals? Are any warning published by competent authorities?
 

Write a comment or suggest a term!