Approach summary: because both models are evaluated on the same cases, the null hypothesis is that model identity has no effect on score — equivalently, per-case pairing matters. The correct permutation for a paired AUC test is to randomly swap (or not) the two model predictions for each case (i.e., permute model labels within each pair), preserving the marginal distribution of predictions and the pairing with true labels. This simulates the null that either prediction could have come from either model.Procedure (step-by-step):1. Compute observed statistic: d_obs = AUC(preds_A, y) - AUC(preds_B, y).2. Permutation scheme: for each case i, with probability 0.5 swap preds_A[i] and preds_B[i]; equivalently sample a vector s of independent Rademacher ±1 and for s[i] = -1 swap.3. For each permutation k (K large, e.g., 10k–100k): - create preds_A_perm, preds_B_perm by applying swaps - compute d_k = AUC(preds_A_perm, y) - AUC(preds_B_perm, y)4. Null distribution = empirical distribution of d_k. For two-sided p-value: p = (1 + count(|d_k| >= |d_obs|)) / (K + 1) (add-one correction).5. Interpret: small p means observed difference unlikely under null.Why swap predictions (not permute labels)? Permuting labels (y) breaks the dependency between models and responses and changes class balance; swapping preserves the association of each case to its true label and keeps dependence structure between model outputs, which is essential for paired tests.Practicalities and approximations:- Monte Carlo: when the full 2^n permutation space is huge, sample K random swaps (randomization test). Use K ≥ 10k for stable p-values down to 1e-4.- Asymptotic alternative: DeLong’s test provides an analytic variance estimate for AUC difference under correlated ROC curves; faster and gives a z-test approximation—useful for very large datasets.- Bootstrap alternative: paired (casewise) bootstrap that resamples cases with replacement and recomputes AUC difference; yields CIs and p-values but relies on bootstrap assumptions.- Edge cases: ties in predictions or labels — ensure AUC implementation handles ties consistently. If class imbalance is extreme, power may be low; consider stratified resampling for bootstrap diagnostics.Example (Python sketch):python
import numpy as np
from sklearn.metrics import roc_auc_score
def paired_permutation_pvalue(y, a, b, K=20000, seed=0):
rng = np.random.default_rng(seed)
d_obs = roc_auc_score(y, a) - roc_auc_score(y, b)
count = 0
for _ in range(K):
swap = rng.integers(0, 2, size=len(y)).astype(bool)
a_p = a.copy(); b_p = b.copy()
a_p[swap], b_p[swap] = b_p[swap], a_p[swap]
d = roc_auc_score(y, a_p) - roc_auc_score(y, b_p)
if abs(d) >= abs(d_obs): count += 1
return (count + 1) / (K + 1), d_obs
