Machine learning and artificial intelligence are dramatically changing the way businesses operate and people live. The TWIML AI Podcast brings the top minds and ideas from the world of ML and AI to a broad and influential community of ML/AI researchers, data scientists, engineers and tech-savvy business and IT leaders. Hosted by Sam Charrington, a sought after industry analyst, speaker, commentator and thought leader. Technologies covered include machine learning, artificial intelligence, deep learning, natural language processing, neural networks, analytics, computer science, data science and more.
Chris Potts, a Stanford professor and co-founder of Big Spin, discusses the growing need to consider the economic sustainability of AI models. As reasoning models consume more tokens and agents become integrated into workflows, the costs and incentives associated with AI usage are becoming impossible to ignore, leading to a focus on 'tokenomics.' This shift prompts questions about the return on investment for AI token purchases.
Stanford professor Chris Potts shares how his academic background in linguistics, particularly his PhD research on swearing, unexpectedly led him into Natural Language Processing (NLP). He explains that his desire to analyze the context and intent behind people's language, especially in the context of profanity, drove him to explore corpora and NLP tools.
Chris Potts reflects on how the rise of AI and large language models has impacted the field of linguistics and NLP. While traditional linguistics is invigorated by AI's ability to model language, NLP faces an uncertain future due to the dominance of large, pre-trained models, making independent research more challenging.
Chris Potts explains that due to the uncertainty and resource demands of foundational AI research post-GPT-3, his team has shifted focus towards interpretability. This area is seen as relatively inexpensive to pursue and offers a promising path for understanding how complex AI models achieve their capabilities.
Jul 27 · Why Models Are AI’s Next Training Dataset with Damian Borth - #7726 stories
Damian Borth, a professor of AI and Machine Learning, suggests a novel approach where the weights of already trained neural networks can serve as input data for training new models. This 'weight space learning' could accelerate and refine the process of creating new AI models.
Damian Borth describes weight space learning as a promising direction for machine learning, aiming to solve current challenges faced by the AI community. This approach views trained model weights not just as an output, but as a new input modality for further learning and analysis.
Borth draws an analogy between weight space learning and language models. Just as language models are trained on text data to understand and generate language, a model trained on neural network weights could potentially analyze and generate new weights.
The initial research in weight space learning, starting around 2020-2021, focused on the idea of 'fingerprinting' neural networks. The goal was to determine if neural network weights could be analyzed similarly to software code to identify differences and changes.
Early work in weight space learning involved using autoencoders to compress neural network weights into a lower-dimensional latent space. These embeddings were then used with linear regression to predict a neural network's accuracy without needing test data.
Damian Borth highlights the challenge of permutation symmetries in neural network weights, where changing the order of neurons doesn't alter the network's function. His team developed techniques like contrastive loss and augmentation strategies to address these symmetries in weight space learning.
