It is a good idea to consult a statistician already during the planning phase to enable the production of correctly analysed data and publication of the result. Early preparation is crucial for producing a good study design, the correct number of study participants and the right outcome measures to answer your research question.
Planning before the study begins determines how data is handled and analysed
Before the clinical study begins, a plan for data management and analysis should have been produced. The plan can vary in the level of detail, and may consist of a few short lines formulated in the research protocol, or of a longer document (statistical analysis plan, SAP), with more detailed contents. It is more important that the data management and analysis has been described in detail in a study aimed at producing a confirmatory result than in an explorative study that aims to generate preliminary results.
The research question or hypothesis, definition of outcome measures, choice of study design and subsequent analysis methods shall all link up, and shall be understandable as a whole in the protocol or analysis plan. It shall be clear that what is measured in the study will be possible to use to answer the question. Any interim analyses, planned adjustment analyses for confounders or subsidiary group analyses should also be described. Depending on the level of detail that is justified, the statistical methods to be used should be described.
Studies with a confirmatory purpose
A confirmatory study shall be based on a hypothesis that is defined before the study start, and that can be tried after the completion of the study with the help of the data collected, and with the methods described in the research protocol or analysis plan. It is important to define the primary outcome measure, that is to say which variable is to be analysed to test the pre-defined hypothesis. The primary outcome measure is the variable on which you base your sample calculation, and determines how many individuals must participate in the study in order to achieve a certain level of statistical power in the study. The method used to calculate sample size is the one that then determines the method used for analysing the hypothesis.
In a confirmatory study, it is particularly important to take measures to prevent as far as possible any bias that may affect the results of the study. If it is a comparative therapy study, there should be a control group receiving treatment using placebo or “golden-standard” therapy. Which type of therapy allocated to each study participant should be determined in an entirely randomised way, and independent of the wishes of the study participant or researcher. To the extent it is possible, the study should also be double-blind. Double-blinding means that neither the study participant nor the researcher knows which type of therapy has been allocated to the study participant.
Studies with an exploratory purpose
In exploratory studies, one or several relatively broad questions are asked, aimed at generating information in an area where there is a lack of previous knowledge. In this case, it is not relevant to formulate a hypothesis to be tried. As there is no hypothesis to be tried, there is also no reason to calculate a sample size. Therefore, no primary outcome measure needs to be defined; instead, the data to be collected can be described in the research plan or protocol without any ranking. Exploratory studies are often conducted without a control group, and without using various methods such as blinding and randomising to reduce bias. The results of exploratory studies are mainly reported using descriptive measures, but also with confidence intervals, for example. Statistical tests may also be used. The purpose is to generate information that can form the basis for confirmatory hypothesis trials in future studies.
There are also confirmatory studies that include an explorative part.
Data handling when the study is concluded
When the practical implementation of the study has been concluded and all the data collected in a database or a data file, you should check that they are complete and that there are no outstanding questions. If any correction needs to be made, then information about the corrections shall be documented and saved.
If the study is blind, the corrections should be made without uncovering the blinding, to avoid the risk of certain study data being removed or corrected on the wrong grounds. If it turns out there is a need to change the analysis plan, it is suitable to do this before the blinding is uncovered. It is important to document the reasons why the analysis plan needs changing, and to be clear with this information when the study results are reported and published. Remember that a changed analysis plan may be considered as a significant change that requires approval by the Ethical Review Authority. If data from the study are not blinded, you could let an independent party carry out corrections and any checks of data.
Thereafter, the study data should be “closed” or “locked”, that is to say nothing more must be changed in the raw data that forms the basis for the analysis. The closure of the database, or the datafile as applicable, is particularly important in studies with a confirmatory purpose, but can be copied when applicable in exploratory studies too. In studies where you do not have access to a database with a locking function, it is important you ensure there is an exact copy of the raw datafile saved in unchanged condition, that is not corrupted by ongoing analysis work.
The raw datafile or database should be archived for possible future scrutiny and control.
Analysis of study data
Before the questions asked in the study are analysed, the data may need further processing. For example, some data may need to be used to generate the variable to be used in the analysis. One of the simpler examples are weight and height collected in the raw datafile, but which must be used to calculate the BMI when this is the variable to be included in the statistical analysis. Calculations to replace missing values in data (imputation) may also need to be made. If imputation methods are used, these should be pre-specified in the research protocol or analysis plan.
The method choice that is then used for data to conduct the statistical analysis is determined by the research question and the study design described in the research plan or analysis plan. If a primary question has been defined in the plan, the analysis of the primary variable should be clearly separated from the analyses of the other variables, which is also the case when the data is later presented. Without constituting a full list, some examples of aspects to be considered in the analysis phase are presented below.
If many different therapies or other interventions are to be analysed in relation to the same outcome measure, the problem with multiple comparisons should be considered, and an adjustment of the significance level may be required.
In studies that do not include a randomising process, it is important to identify and define in advance any confounders for which the main analyses must be adjusted. A confounder is a variable that affects both the primary outcome measure and the therapy groups in the study. There are various ways of identifying confounders, for example through DAGs (Directed Acyclic Graphs). If a randomising process is included in the study, no adjustments for confounders are normally needed afterwards.
Subsidiary group analyses and interactions
If the selected outcome measure co-varies (interacts) with another variable apart from the study therapy, then it is often interesting to report such interactions. The interaction variable may be a variable that divides up persons into subsidiary groups (such as gender or age groups), or a continuous variable (such as age or weight). You should as far as possible state the variable and methods you intend to use for interaction analyses in your analysis plan. Subsidiary group analyses can also be of an exploratory nature, if during the analysis work it turns out that there are previously unknown interactions.
Interim analysis is analysis that is carried out during while the study is in progress, and aims to provide a recommendation whether the study should continue, or be terminated early. As the performance of an interim analysis impacts on the sample size of the study, it is important that they are planned carefully already in the study protocol. This type of analysis is usually carried out by the study’s Data Safety and Monitoring Board (DSMB), which is an independent monitoring group. Any uncovering of blinding shall be kept within the independent group, and all personnel working with the study shall remain blind to the result of such analysis. Interim analysis that is carried out in an unplanned manner should be avoided, as it can weaken the trust in the conclusions drawn.
Specific rules for medicines or medical devices
There are specific rules to attend to when analysing medical device-studies that have a permit from the Medical Product Agency.
Specific rules for medical devices
The principles for data management and analysis presented in the section above also apply for studies involving medical devices. A specific requirement for clinical studies involving medical devices that require notification to the Swedish Medical Products Agency is that the standard for good clinical practice, ISO 14155, shall apply. This means, among other things, that there are a number of statistical considerations to be described in the clinical investigation plan, see Annex A.5–A.8 of the standard. Any divergence from the investigation plan in terms of the statistical aspects of the study must be evaluated in relation to whether they should be considered as significant changes that need approval from the Ethical Review Authority and the Medical Products Agency before they can be implemented.
Confirmatory studies of the safety and performance of medical devices are often based on the product being compared with an existing method or therapy, and in this way it can be shown that the device works equally well or better. This type of study should be randomised and blinded, to avoid bias. For studies involving medical devices, it can be difficult to design the study based on these criteria, for example because it may be impossible to develop an identical placebo or comparison device, and it may also be unsuitable to use a prospective control group for various reasons. Some of the differing ways of handling the challenges involved in medical device studies are presented below.
Medical device with a therapeutic purpose
Control group: If it is possible to use a prospective control group that is treated with a standard therapy, then this is to be preferred. Otherwise, it may be possible to use a historical control, where you know that a certain therapy or lack of therapy has a certain outcome, which can then be compared with the outcome of the study.
Randomising and blinding: If you use a historical control, then you do of course lose the opportunity of randomising and blinding the therapy. But if you use a prospective control group, it can still be difficult to keep patients and the physicians providing treatment blinded, if it has not been possible to develop a placebo therapy. In this case, an independent evaluating physician may conduct a follow-up evaluation of the therapy result in a blinded way (that is to say, without knowing what therapy the patient has been randomised to).
Medical device with a diagnostic purpose
Control group: Often there is a standard method for making a diagnosis in the area. This method has a known ability to make a specific diagnosis (known as sensitivity and specificity, which can be used for comparison with the results of the trial device.
Randomising and blinding: In some cases, patients in diagnostic studies can be their own control, that is to say that they are subjected to two types of investigation, where one is an investigation using the standard method, and one is an investigation using the trial device. To increase safety for the study participant, the patient is diagnosed using the standard method. An independent physician can, however, make a diagnosis using the standard method and the trial device in a blinded and randomised way, to see whether the diagnoses generated by each method correspond.
As for all clinical studies, it is recommend that the study design, data management and analysis are discussed with a statistician as necessary. There is guidance information about medical device study design and statistical considerations on the EU Commission’s website (MEDDEV 2.7/4). In the guidance document MEDDEV 2.7/2, you can read about how public agencies assess applications for clinical studies involving medical devices they receive in terms of the study design in particular. The International Medical Device Regulators Forum has produced a guidance document for how medical device software can be evaluated. ICH E9 Statistical Principles for Clinical Trials is a document primarily produced for clinical medicine studies, but the statistical principles can be applied for medical device studies as well.
ECRIN has produced a database of outcome measurements that have been used in studies of different types of medical devices. Here you can get tips of how others have chosen to measure the effect and performance of medical devices. Harmonisation of outcome measurements can also facilitate future meta-analyses and summaries of the evidence situation for devices that have been evaluated in the same way.