Statistical analysis

Introduction

Statistics is a science derived from mathematics which can be divided into two branches: descriptive and inferential. Descriptive statistics are commonly used as the first step in data analysis. It refers to measures that summarize and characterize a set of data – it answers questions like: How many people have the disease? How often did the event occur? What is the spread of test results in the population? Descriptive statistics can point to similarities and differences between groups, but on their own are not enough to confirm or refute a hypothesis. Inferential statistics allows hypothesis testing based on the probability theory. It answers the key question: How likely is it that this difference (that we observed between two groups) is due to chance alone?

Statistical Analysis Plan

Before analyzing your data, you should have a Statistical Analysis Plan (see below for links to templates, guidelines and examples). This document describes what statistical tests you will perform and assigns them to a hierarchy of primary, secondary and, sometimes, exploratory analyses. Many researchers will publish the statistical analysis plan, either summarized as part of a design paper or study protocol, or by making it available online. This ensures transparency and encourages rigorous scientific method.

Primary, Secondary and exploratory analyses

Your primary analysis relates to your primary study outcome. It is essential to spell this out from the start. Remember that a P-value of 0.05 just means a likelihood of one in twenty, meaning that if you do twenty statistical tests, the chances are that one of them will be ‘positive’ with a P-value < 0.05. So it is important to specify your primary outcome before doing the analysis so that readers know that you have not simply performed many statistical tests and chosen the one with the significant P-value. Analyses that you plan to do in advance, alongside the primary analysis are called secondary analyses. Any analysis that you decide to do only after looking at the data is always considered exploratory meaning that it can only ever be considered as hypothesis generating for further research and should never be used as proof.

Types of Data and Variables

An important step before applying any statistical test is to identify the dependent and independent variables. In clinical trials, the dependent variable is the outcome(s) of the study (e.g. mortality rate; change in renal function) and the independent variables are the factors under investigation that could possibly modify the outcome, the most important of which is the randomized treatment allocation (i.e. intervention vs. control). Other independent variables (e.g. proteinuria, blood pressure) may be used in multivariable analysis, however this is almost always a secondary analysis in a clinical trial because randomization has been used to balance all other factors between groups.

Each variable should be classified by type (eg. continuous or categorical, including ordinal and dichotomous categories) and distribution (normal [also known as parametric or Gaussian] or non-normal). This will help you determine the right statistical test to use. This should be considered when planning the study as the type of data will result in strengths and limitations in terms of the possible mathematical tests and the interpretation of the obtained results. For example, if CKD stage is collected then one must use a categorical data analysis, however if eGFR is collected then one can use a continuous data analysis (usually more powerful) or convert them to CKD categories to use a categorical analysis. Therefore, a careful plan to adjust the study design according to the research question and the characteristics of the study variables is always desirable.

Other Considerations

Intention-to-treat vs. per protocol analyses

Intention to treat (ITT) analyses divide study participants according to their randomised treatment allocation (e.g. intervention or control) – regardless of whether the participant received their intended treatment. This preserves the equal distribution of both known and unknown confounding factors created by the randomisation, and so serves as the least biased test of the overall effect of an intervention.

A per protocol analysis divides study participants according to what treatment they actually received (i.e. by whether they followed the study protocol). This approach, although appearing to better reflect what actually occurred during the trial, is subject to biasing the results. This occurs because deviation from the protocol (whether non-adherence or crossing over to receive the alternative treatment) is frequently associated with clinical trial outcomes (e.g. patients in SGLT2 inhibitor trials who fail to take their study medication may be more likely to be non-adherent to their other prescribed medications, such as angiotensin receptor blockers or statins, leading to a higher risk of adverse kidney and cardiovascular outcomes).

As a rule, although per protocol analyses may be of interest as an indicator of treatment efficacy under optimal conditions, ITT analyses are the preferred measure of treatment effect. An exception to this is a non-inferiority trial, which aims to demonstrate that one treatment is not meaningfully worse than another. In this case, a per protocol analysis – which may overestimate treatment differences – provides the more stringent test and may be preferred to ITT. Because an ITT analysis typically provides a conservative estimate of treatment effect, a modified version is sometimes applied in which participants who never receive study treatment (neither intervention nor control) are excluded from analysis. This ‘mITT’ analysis can be unbiased in double-blind trials where the participant’s decision not to proceed with treatment cannot have been influenced by knowledge of their randomised allocation.

For more information see to following articles:

Cole et al. Reducing Bias in Estimates of Per Protocol Treatment Effects: A Secondary Analysis of a Randomized Clinical Trial. JAMA Netw Open. 2023;6(7):e2325907. doi:10.1001/jamanetworkopen.2023.25907

Kahan, et al. Using modified intention-to-treat as a principal stratum estimator for failure to initiate treatment. Clin Trials. 2023 Jun;20(3):269-275. doi: 10.1177/17407745231160074.

P-values are not the only way to assess a trial result

Many in the scientific community have suggested that too much emphasis is placed on P-values. In short, there is more than one factor in determining how meaningful a result actually is. Further discussion of this issue can be found in the following articles:

Useful links