Inference for the conditional association between an exposure and a time-to-event endpoint, given covariates, is routinely based on partial likelihood estimators for hazard ratios indexing Cox proportional hazards models. This approach is flexible and makes testing straightforward, but is nonetheless not entirely satisfactory. First, there is no good understanding of what it infers when the model is misspecified. Second, it is common to employ variable selection procedures when deciding which model to use. However, the bias and uncertainty that imperfect variable selection adds to the analysis is rarely acknowledged, rendering standard inferences biased and overly optimistic. To remedy this, we propose a nonparametric estimand which reduces to the main exposure effect parameter in a (partially linear) Cox model when that model is correct, but continues to capture the (conditional) association of interest in a well understood way, even when this model is misspecified in an arbitrary manner. We achieve an assumption-lean inference for this estimand based on its influence function under the nonparametric model. This has the further advantage that it makes the proposed approach amenable to the use of data-adaptive procedures (e.g., variable selection, machine learning), which we find to work well in simulation studies and a data analysis. for this article are available online.