The win statistics are a family of flexible, nonparametric statistics used to quantify effect size in comparisons of hierarchical composite outcomes between two groups. The win statistics – comprising the win ratio, the win odds, and the net treatment benefit (also called the win difference) – are primarily used as measures of effect size in the generalised pairwise comparisons (GPC) framework, a generalisation of the theory underpinning the Finkelstein-Schoenfeld test.
The Finkelstein-Schoenfeld test and, by extension, the GPC framework were developed to address limitations in composite outcome modelling when using conventional statistical methods. Composite outcomes are typically analysed using survival analysis on a time-to-first composite event, but doing so gives equal weighting to all outcomes and typically results in comparisons driven by the more frequent, less clinically relevant events (e.g., hospitalisations instead of deaths). To remedy this, the GPC framework considers an ordered hierarchy of outcomes that readily incorporates comparisons of recurrent events and continuous outcomes.
Methodology
Outcome hierarchy The first step in calculating the win statistics is to define a hierarchy of outcomes ranked in decreasing order of clinical relevance. This hierarchy dictates the sequence in which outcomes are considered when evaluating the data, with greater priority given to the more clinically important outcomes.
Pairwise comparison procedure
Forming pairs Win statistics are built on subject-level comparisons of between-treatment patient pairs. Two approaches exist for forming these pairs: matched and unmatched.
In a matched comparison, pairs are formed by matching each patient in the treatment arm to one patient in the control arm; Pocock et al. recommended that this matching be performed on baseline risk. If there are patients in the treatment arm and in the control arm, this results in pairs, with any "excess" patients in either arm excluded from the analysis.
In an unmatched comparison, every patient in the treatment arm is paired with every patient in the control arm. If there are patients in the treatment arm and in the control arm, this produces total pairs. The construction of all possible between-arm patient pairs forms the basis for Buyse's GPC framework.
Pair-level comparisons Each pair is then evaluated sequentially against the outcome hierarchy. If one patient in the pair has a more favorable result than the other, that patient is declared the "winner", the pair is deemed untied, and the comparison terminates; if neither demonstrates a better result, the pair proceeds onto the next outcome in the hierarchy. Subsequent outcomes are considered similarly, with this process repeating until the pair is untied or the hierarchy exhausted.
In theory, one has infinite flexibility to define what constitutes a favorable outcome; in practice, almost all comparisons fall into one of three comparison types: time-to-first event, recurrent event count, or a generic quantitative comparison. The two event-based outcomes consider comparisons over the pair's shared follow-up, the period over which both patients are fully observed, with events occurring outside this period not considered when performing the comparison.
All three approaches allow the use of "margins of win", whereby the difference between patients' outcomes must exceed some minimally clinical relevant threshold to declare a "winner".
Note that "wins" for the control arm are sometimes referred to as "losses" for the active arm.
Time-to-first event
Within each pair, patients are compared on the time until they first experience an event: if both patients experience the event, the one with the longer time-to-event is deemed the "winner"; if one patient experiences the event within the shared follow-up window and the other does not, the patient without the event is deemed the "winner"; if neither experiences an event during their shared follow-up period, the comparison is deemed indeterminate.
Events may also be considered successes (e.g., complete response in oncology studies), in which case the approach described above is reversed.
Recurrent event count
Patients are compared on the total number of repeat events experienced during their shared follow-up: if one patient experiences more events than the other, the patient with the fewer events is declared the "winner"; otherwise, the comparison is said to be indeterminate. As with time-to-first outcomes, these events may be considered positives, in which case the patient with the greater number of events is deemed the "winner".
Quantitative Generic quantitative comparisons involve comparing patients in each pair with respect to some single quantitative variable (e.g., a quality of life score), where the patient with the higher value is deemed the "winner". The direction of comparisons may be reversed if lower scores are considered more favorable.
Calculation of the win statistics After all pairs have been evaluated, the total number of "winners" in each arm is determined. Pairs that are not untied are deemed "tied" for the purpose of calculating the win statistics.
Win ratio The win ratio is calculated as the ratio of the number of "winners" in the treatment arm to the number of "winners" in the control arm:The win ratio can be interpreted as the odds of having a "better" outcome, conditional on being untied by the hierarchy.
Win odds The win odds are calculated as the number of "winners" in the treatment arm plus half the number of ties, divided by the number of "winners" in the control arm plus half the number of ties:
Net treatment benefit The net treatment benefit is defined as the difference between the number of "winners" in the two arms divided by the total number of patient pairs, calculated as follows:
Example To demonstrate, consider a two-arm clinical trial comparing a new treatment () with a control (), with respect to the following hierarchy of outcomes assessed 12 months after randomisation:
Time-to-death Number of hospitalisations Quality of life score (higher scores indicate better quality of life)
Let denote the patients in the treatment arm and the patients in the control arm. An unmatched analysis would utilise all patients' data, yielding pairwise comparisons; a matched approach would give comparisons.
Starting at the top of the hierarchy, patient pairs are first compared according to their time-to-death over their shared follow-up period. Without loss of generality, consider a pair comprising patients and .
For component (1), a "favorable" outcome is defined as being observed to survive beyond the other patient in the pair. If was censored 9 months after randomisation while was followed up until their death at 12 months post-randomisation, the comparison would be declared a tie because the event happened outside the pair's shared follow-up period (the first 6 months after randomisation during which was observed). If instead had been censored at 24 months, 's death would occur within their shared follow-up period and the comparison would be declared a "win" for the active arm.
For component (2), a "favorable" outcome is defined as experiencing fewer hospitalisations than the other patient in the pair over the pair's shared follow-up period.
To illustrate, assume and were tied on component (1); was hospitalised at 3 and 5 months and later censored at 6 months; and that was hospitalised at 4, 8, and 10 months, before being administratively censored at 12 months. The pair's shared follow-up period is then the first 6 months of follow-up (the point at which was censored); during this period, experienced 2 hospitalisations (at 3 and 5 months), while experienced just 1 (at 4 months). As such, this comparison would then be declared a "win" for the control group.
The subset of pairs who remain tied after comparisons on components (1) and (2) then proceed to comparisons of component (3). For component (3), a "favorable" outcome is defined as having a greater quality of life score.
Suppose and were tied on components (1) and (2) and continued on to a comparison of (3). Assume 's quality of life score was 5 and C1's was 4: in this instance, has a higher score and is declared the "winner".
We might have introduced a "margin of win", stipulating that the difference between the patients' values must be greater than the some threshold to declare a "winner": had we used a margin of two points in the quality of life score, this pair would remain because the difference did not exceed 2 points. Here, without any remaining outcomes left to consider, no further attempts are made to untie the pair and they are considered "tied".
Variance estimation and confidence intervals Variance estimation differs significantly between the matched and unmatched analyses due to the patient-level clustering in the latter.
Matched approach To estimate the variance in a matched analysis, patient pairs that were not untied by the hierarchy are first discarded, leaving only the "winners" in each arm. The probability of a given winner belonging to the treatment arm, , is then:Where and denote the number of winners in the active and control arms, respectively. The variance of follows from the Binomial distribution and the bounds of its 95% confidence interval, and , are asymptotically given by: , andThe confidence intervals for the matched win ratio (WR) and net treatment benefit (NTB) are then given by: , and
, for The variance of the win odds is not well defined under this approach.
Unmatched approach In 2023, Dong et al. formally described the relationship between the variance estimators for the unmatched win ratio, win odds, and net treatment benefit: Deriving the variance of one of the win ratio, win odds, or net treatment benefit is then sufficient to find all three.
U-statistic In 2016, Dong et al. and Bebu and Lachin developed U-statistic-based asymptotic variance estimators for the win ratio. Define kernel functions , which takes value 1 if patient in the active arm "wins" over patient in the control arm (and 0 otherwise), and , which takes value 1 if patient in the control arm "wins" over patient in the active arm (and 0 otherwise). The U-statistics and are then unbiased estimators of the win and loss probabilities:Where denotes the number of patients in the active arm and the number of patients in the control arm. It then follows that and are asymptotically normally distributed:Where and are equal. By the delta method, the variance of the log win ratio is asymptotically given by:The components of the variance-covariance matrix, , , , are defined as:It is the definition of the and terms that vary between the Dong and Bebu and Lachin approaches.
Bebu and Lachin Define , , , and , the patient-specific win and loss counts, as:It then follows that:
Dong Let be defined as:Then:
Yu and Ganju
In 2021, Yu and Ganju developed an approximate variance estimator, valid under the null hypothesis that the win ratio is 1, that is a function of the treatment allocation ratio , the number of patients in each arm and , and the number of ties :, where The primary use of Yu and Ganju's variance estimator is in sample size calculation: Dong and Bebu and Lachin's estimators remain the preferred approaches for real-world application due to their superior accuracy, particularly when there is evidence of a treatment effect.
Accounting for covariate imbalance While matched methods directly address the issue of "unfair" comparisons between patients with different risk profiles, covariate imbalance in the unmatched approach can lead to a loss of precision and biased estimates. Several approaches have been developed to address this, to varying degrees of abstraction.
Stratification