The TWIML AI Podcast · Monday, July 27, 2026
Damian Borth highlights the challenge of permutation symmetries in neural network weights, where changing the order of neurons doesn't alter the network's function. His team developed techniques like contrastive loss and augmentation strategies to address these symmetries in weight space learning.
“One of the tricky things with weight spaces is, when you have a neural network and you have two layers, let's say for simplicity, fully connected, you can change the position of the neurons and, uh, it's actually changes the sequence of the neurons, the order of weights, but the function is the same.”
“So, you know, there are a lot of permutation symmetries and other, you know, symmetries that, you know, in the weight space that do not change the underlying function.”
“So we have already in the first paper, we had contrastive loss, and to build a contrast, you need to augment. So, I mean, it's simple to flip an image or rotate an image, but you know, what's the counterparty in weight space?”