Does Mapping Non-Maximal Probabilities to GMM Components Matter for S-JEPA Encoder Representations?

S-JEPA uses soft Gaussian mixture model (GMM) posteriors instead of hard cluster labels to preserve uncertainty. It remains unclear whether the probability values alone are sufficient, or whether it also matters which GMM components receive the non-maximal probabilities. We test this with two matche...