Lex Fridman Podcast
Reformatted for readability — timestamps removed, lightly restructured. Not verbatim.
Note: full verbatim transcript not available for this episode. Content below represents section summaries and selected quotes from the source.
Lex Fridman: Let's start with the DeepSeek moment from January 2025, when they released DeepSeek R1 with near state-of-the-art performance using allegedly much less compute. The competition has accelerated dramatically. Who's winning — China or the United States?
Sebastian Raschka
Winning
is broad. DeepSeek wins hearts in the open-weight community through transparency. However, winning operates on different timescales. Currently, I don't believe any company possesses exclusive technology access. Researchers frequently change positions across labs. The differentiating factor will be budget and hardware constraints rather than proprietary ideas. I don't foresee a winner-takes-all scenario. Nathan Lambert: The hype around Anthropic's Claude Opus 4.5 has been intense and organic. Gemini 3 from Google had strong marketing momentum but seems overshadowed. Anthropic's cultural focus on coding is serving them well. However, China has numerous frontier labs beyond DeepSeek. DeepSeek initiated a movement similar to ChatGPT's impact, with companies like Z.ai, MiniMax, and Moonshot releasing strong open-weight models. The business model question remains: American companies serve paying users well, while Chinese companies leverage open-weight models for influence and market access despite US security concerns. Lex Fridman: How long will Chinese companies maintain open-weight releases? Nathan Lambert: Likely several years. They realize, as US companies do, that open models lack clear business models. However, they strategically see open-weight releases as accessing the substantial US AI expenditure market despite security barriers. Government incentives supporting international influence will probably maintain support. Consolidation will eventually occur, but expect more open-model builders throughout 2026 than 2025, notably from China. Sebastian Raschka: DeepSeek hasn't deteriorated; competitors now implement DeepSeek's innovations. This leapfrogging pattern means the most recent model typically performs best. There won't be a decisive winner — continuous cycling of releases occurs.
Lex Fridman: Which model won 2025, and which will win 2026? Nathan Lambert: Betting on Gemini over ChatGPT feels risky given OpenAI's incumbent advantages. Gemini gained 2025 momentum but started from weakness. They deserve credit for organisational persistence. Yet OpenAI, despite apparent chaos, consistently delivers. GPT-5's router feature saves them substantial GPU costs. Dissociating personal preferences from market differentiators is challenging. For 2026, Gemini will likely continue gaining ground. Google possesses scale advantages at extreme operational scales and separates research from product better than OpenAI's startup-like culture. Anthropic will maintain enterprise success. Lex Fridman: Do TPUs provide Google advantages? Nathan Lambert: Absolutely. NVIDIA's chip margins are substantial; Google develops end-to-end solutions without paying intermediary margins. Historical data centre expertise creates advantages. New paradigms likely emerge from OpenAI's research division, which has repeatedly landed innovations like Deep Research, Sora, and o1 thinking models. Sebastian Raschka: I appreciate having toggle options between intelligence and speed. For quick queries, I use fast models. For thorough reviews of my writing — checking references, formatting, figures — I use slower models without time pressure. Nathan Lambert: I'm losing my mind that you use non-thinking models. I exclusively use thinking variants or pro versions for information queries — research papers, code references. Lex Fridman: I use Grok 4 Heavy for hardcore debugging — it excels there. ChatGPT dominates for interface familiarity, though Gemini's context window strength appeals to me for information retrieval. You choose models based on particular strengths until they fail. Sebastian Raschka: Exactly. You use models until they break, then explore alternatives. It mirrors how we choose text editors or browsers — default until functionality fails.
Lex Fridman: On programming, I split time between Cursor and Claude Code — fundamentally different but both valuable. What do you use? Sebastian Raschka: I use Codeium with VS Code. It's convenient with repository access through chat. Claude Code is more agentic and comprehensive, but I prefer maintaining control. Codeium strikes my balance between assistance and oversight. Lex Fridman: Claude Code teaches programming through English rather than micromanaging diffs. You design macroscopically while understanding code deeply. Nathan Lambert: Claude Code performs remarkably better in that domain. Using both simultaneously with identical prompts shows clear differentiation.
Lex Fridman: Describe the open LLM landscape. Which models interest you most? Nathan Lambert: DeepSeek, Kimi, MiniMax, Z.ai, Antlang... Sebastian Raschka: Mistral AI, Gemma, gpt-oss, NVIDIA's Nemotron 3, Qwen... Nathan Lambert: OpenAI's first open model since GPT-2 is gpt-oss — very strong with underutilised strengths. Western companies including the Allen Institute for AI with OLMo, the Institute for Foundation Models with K2 models, Apertus, Hugging Face's SmolLM, NVIDIA's Nemotron, and Stanford's MiniCommunity Project all release fully transparent models. Chinese open-language models tend toward larger sizes, delivering higher peak performance as mixture-of-experts, while US models have favoured smaller approaches, though this is changing. Lex Fridman: You neglected mentioning LLaMA. Nathan Lambert: RIP. Sebastian Raschka: Qwen 3 stands out, as does gpt-oss. What makes gpt-oss particularly interesting is being the "first open-weight model really trained with tool use in mind," representing a paradigm shift for ecosystem readiness. Sebastian Raschka: Tool use — where LLMs perform web searches or call Python interpreters — directly addresses hallucinations. Rather than memorising information, models can search for facts or use calculators, ensuring reliability. Lex Fridman: What explains the open-model explosion? Nathan Lambert: Companies releasing open models want adoption foremost, then transparency and trust. Chinese companies globally distribute models since many decline software payments but might host models locally. Data privacy preferences also drive local deployment. Sebastian Raschka: Customisation possibilities unlock value for companies. You can fine-tune, specialise for law or medicine, and modify post-training. Chinese licences prove friendlier than restrictions limiting user bases.
Lex Fridman: Let's discuss transformer architecture broadly. Sebastian Raschka: The "Attention Is All You Need" paper presented encoder-decoder transformers. GPT focused on decoders — neural networks with attention mechanisms predicting tokens sequentially through embedding, transformer blocks with attention modules, fully connected layers, and normalisation. Moving from GPT-2 to gpt-oss-120b introduces mixture-of-experts, replacing single expensive fully-connected layers with multiple feedforward networks. A router selects relevant "experts" based on input tokens, packing more knowledge without using all simultaneously. Math-heavy inputs might consult different experts than translation tasks. Lex Fridman: How many genuinely new ideas exist between GPT-2 and today? Sebastian Raschka: Mixture-of-experts, group query attention (tweaking multi-head attention), RMSNorm replacing LayerNorm, different activation functions. These constitute modifications rather than fundamental transformations. You can convert GPT-2 to modern architectures through incremental adjustments. Lex Fridman: Fundamentally the same architecture? Sebastian Raschka: Correct.
Lex Fridman: Do scaling laws persist across pre-training, post-training, inference, context, data, and synthetic data? Nathan Lambert: Scaling laws represent power-law relationships between compute/data and held-out prediction accuracy. This relationship remains predictable. Three scaling dimensions now exist: traditional pre-training scaling (model and dataset size), reinforcement learning scaling, and inference-time compute. All continue working, though low-hanging fruit diminished, particularly in 2025 reinforcement learning with verifiable rewards and inference scaling. These changes fundamentally alter model experience — generating hidden thoughts before responses rather than immediate outputs — enabling tool use and superior engineering capabilities. Models rapidly learn skills through reinforcement learning with verifiable rewards, picking up tool usage, CLI commands, Git handling, and information searches that seemed impossible recently. Lex Fridman: Are you bullish on every scaling type? Nathan Lambert: Pre-training scaling grew extremely expensive. Serving massive models costs billions for hundreds of millions of users, whereas training costs five million dollars. DeepSeek reported five million dollars at cloud rates; OLMo 3 required two million dollars. Recurring serving costs dwarf training costs. Lex Fridman: Do models get smarter with pre-training scaling? Nathan Lambert: Fundamentally yes. Leadership notes this persisted across thirteen compute orders of magnitude — why would it stop? 2026 features gigawatt-scale NVIDIA Blackwell clusters coming online from 2022–2023 contracts. Lex Fridman: xAI targets one gigawatt early 2026, two gigawatts by year's end. Nathan Lambert: All approaches are involved. Pre-training decisions underpin all other training. Mixture-of-experts models enable efficient generation, essential for post-training. Architecture readiness determines compute scalability. Most compute remains in pre-training, enabling future RL scaling. Eventually RL scales longer as pre-training saturates. Sebastian Raschka: Training and inference offer two scaling knobs. Unlimited resources would enable both. Pre-training provides knowledge and capability unlocks. But mid and post-training capabilities require unlocking that base model knowledge. Claude Opus 4.5 shows inference scaling exceeded pre-training scaling benefits currently. Smaller models, through inference scaling, surpassed larger pre-trained models. Pre-training scaling isn't dead — merely less immediately attractive. Eventually, pre-training progress becomes necessary. Pre-training provides fixed training costs with permanent capabilities. Inference scaling costs accrue per query. Financial models determine worthiness based on model lifetime and user count.
Lex Fridman: Define pre-training, mid-training, and post-training. Sebastian Raschka: Pre-training involves classic next-token-prediction training across vast data corpuses. Pre-training focuses on cross-entropy loss via next-token prediction on internet data, books, and papers. Synthetic data evolved beyond raw scraping toward rephrasing Wikipedia articles as Q&A, summarising, and improving quality. Like humans learning from books versus messy Reddit posts, refined data enables faster learning. Mid-training resembles pre-training but focuses on specialised content like long-context documents. It avoids catastrophic forgetting — teaching new information while forgetting learned material. Mid-training selectively maintains quality, ensuring final exposures provide premium content. Post-training encompasses supervised fine-tuning, DPO, and reinforcement learning with human feedback — refinement stages. Pre-training demands substantial investment but provides permanent capabilities. Reinforcement learning costs less, unlocking rather than teaching knowledge — skill learning from existing knowledge. Nathan Lambert: Synthetic data isn't solely AI-generated. Optical character recognition (OCR) papers by DeepSeek, AI2, and others extract trillions of tokens from PDFs and documents. Pre-training datasets measure in trillions of tokens — smaller researcher models use five to ten trillion, Qwen reaches fifty trillion, rumours suggest closed labs approach 100 trillion. Data extraction represents massive funnelling, with actual training data comprising small percentages. Sebastian Raschka: OLMo 3 achieved better performance despite using less data than some competitors, suggesting data quality drove improvements. Nathan Lambert: Primarily data quality. With additional compute, we'd train longer. Bigger models absorb more from data, receiving greater benefits logarithmically. We prefer highest-quality data acquisition. Lex Fridman: Are data quality improvements low-hanging fruit? Nathan Lambert: It resembles turning a crank. Canonical best pre-training datasets shift between recent efforts. AI2's Dolma, Hugging Face's FineWeb, and DCLM projects have each held distinctions. The internet's closure increasingly limits options — Common Crawl represents hundreds of trillions of tokens filtered through scientific classifier training and pruning decisions. Language models now require maths and code performance, necessitating dataset remixing. You sample tiny portions from GitHub, Stack Exchange, Reddit, and Wikipedia, train small models on various mixes, measure evaluation performance, and apply linear regression for optimal datasets. Changing evaluations change datasets substantially. OLMo 3 added reasoning sources improving maths and coding. Lex Fridman: Are there surprising high-quality data sources? Nathan Lambert: Reddit proved useful; PDFs definitely did. Sebastian Raschka: Especially arXiv. Nathan Lambert: AI2 long operated Semantic Scholar, competing with Google Scholar. This required locating and scraping openly accessible PDFs beyond closed publisher paywalls. Processing these yields substantial value. Frontier labs pursued this work earlier. Lex Fridman: As LLMs generate more arXiv and GitHub data, how do you handle resulting worlds where much content emerges from LLMs? Nathan Lambert: Infrastructure and systems dominate concerns; from AI perspectives, it's inevitable. Open-source contributors legitimately experience burnout. Some submit numerous pull requests quickly, likely involving LLMs. Sebastian Raschka: I maintain mlxtend, a ten-to-fifteen-year-old repository for frequent data mining algorithms. Recently, several people submitted numerous pull requests within short timeframes, likely involving LLMs. As maintainer, I feel overwhelmed lacking time for thorough reviews. This parallels data labelling — among the most expensive activities for RLHF phases. Higher-quality data emerges through verification phases. Lex Fridman: Raw LLM data differs fundamentally from verified LLM-generated data. Sebastian Raschka: Absolutely. Human verification, however minimal, distinguishes approaches substantially.
Nathan Lambert: Reinforcement learning with verifiable rewards (RLVR) represents a significant shift. Unlike traditional RLHF requiring human preferences, RLVR uses ground-truth answers for maths, code, and logic problems. The model generates solutions, checks against known answers, and updates accordingly — fully automated feedback loops. This enabled the o1 and DeepSeek R1 breakthroughs. Extended chain-of-thought reasoning allows models to explore multiple solution paths before committing to an answer. The "thinking tokens" budget creates a new inference-time scaling axis. Sebastian Raschka: The key insight is that reasoning capability emerges from reinforcement learning on top of pre-trained knowledge. The model already knows mathematics — RL teaches it to apply that knowledge systematically rather than pattern-match to the nearest training example.
Lex Fridman: What's the most interesting development expected in 2026? Nathan Lambert: Agents operating autonomously over extended periods with minimal human oversight. The infrastructure for this — reliable tool use, long-context memory, verifiable output quality — is just becoming robust enough. The economic pressure to deploy these at scale is intense. Sebastian Raschka: Multimodal breakthroughs. Video understanding at the level of text understanding changes everything — research, media production, scientific analysis. We're close to models that genuinely reason over video rather than just describing it. Lex Fridman: Do you think AGI is close?
Nathan Lambert
AGI
is an awkward concept. What we're building is systems that are superhuman at specific narrow tasks. Whether that constitutes AGI depends entirely on your definition. What I'm confident about: systems dramatically more capable than today across most cognitive tasks will exist within a few years. Whether that's "AGI" or just "very powerful AI" is semantic. Sebastian Raschka: I prefer thinking about task completion rates. Today's models complete perhaps 30–40% of complex multi-step tasks reliably. As that approaches 90–95%, the practical difference from "AGI" becomes meaningless for most purposes.
END OF AVAILABLE CONTENT