How hazardous are hazard ratios? An empirical investigation of individual patient data from 27 large randomized clinical trials

The use of hazard ratios as the standard treatment effect estimators for randomized trials with time-to-event outcomes has been the subject of repeated criticisms in recent years, e.g., for its non-collapsibility or with respect to (causal) interpretation. Another important issue is the built-in selection bias, which arises when the treatment is effective and when there are unobserved or not included prognostic factors that influence time-to-event. In these cases, the hazard ratio has even been termed “hazardous” because it is estimated from groups that increasingly differ in their (unobserved or omitted) baseline characteristics, yielding biased treatment estimates. We therefore adapt the Landmarking approach to assess the effect of ignoring a gradually increasing proportion of early events on the estimated hazard ratio. We propose an extension called “Dynamic Landmarking”. This approach is based on successive deletion of observations, refitting Cox models and balance checking of omitted but observed prognostic factors, to obtain a visualization that can indicate built-in selection bias. In a small proof-of-concept simulation, we show that our approach is valid under the given assumptions. We further use “Dynamic Landmarking” to assess the suspected selection bias in the individual patient data sets of 27 large randomized clinical trials (RCTs). Surprisingly, we find no empirical evidence of selection bias in these RCTs and thus conclude that the supposed bias of the hazard ratio is of little practical relevance in most cases. This is mainly due to treatment effects in RCTs being small and the patient populations being homogeneous, e.g., due to inclusion and exclusion criteria.