Key Researchers and Ideas

Affiliations: University of Toronto, Google Brain (2013-2023), now independent. Nobel Prize in Physics (2024) jointly with John Hopfield, for foundational work on artificial neural networks.

Key contributions:

1. Backpropagation (Rumelhart, Hinton, Williams, 1986). The learning algorithm that made multilayer neural networks trainable. The chain rule applied to compute gradients through a computation graph. Still the foundation of all deep learning training.

2. Boltzmann Machines (Ackley, Hinton, Sejnowski, 1985). Stochastic recurrent networks that learn probability distributions. Introduced the idea of learning by adjusting weights to match data statistics. Restricted Boltzmann Machines (RBMs) later enabled deep belief networks.

3. Deep Belief Networks (Hinton, Osindero, Teh, 2006). Showed that deep networks could be pretrained layer-by-layer using unsupervised learning (RBM stacking), then fine-tuned. This broke the "deep networks are untrainable" barrier before ReLU and batch normalization existed.

4. AlexNet (Krizhevsky, Sutskever, Hinton, 2012). Won ImageNet 2012 by a large margin using deep convolutional networks with ReLU, dropout, and GPU training. The result that convinced the field that deep learning works.

5. Dropout (Srivastava, Hinton et al., 2014). Regularization by randomly zeroing activations during training. Still used, though largely superseded by batch normalization and other techniques in modern architectures.

6. Capsule Networks (Sabour, Frosst, Hinton, 2017). An attempt to replace max-pooling with "capsules" that preserve spatial relationships. Theoretically motivated but has not displaced standard architectures.

What still matters in 2026: Backpropagation is universal. Dropout remains in use. Hinton's advocacy for AI safety (post-Google resignation in 2023) has influenced policy discussions. His capsule network agenda has not gained traction.

Affiliations: Meta AI / FAIR (Chief AI Scientist), New York University. Turing Award (2018) jointly with Hinton and Bengio.

Key contributions:

1. Convolutional Neural Networks (LeCun et al., 1989, 1998). Applied backpropagation to networks with shared-weight convolutional filters. LeNet-5 for digit recognition was the first practical deep learning system deployed at scale (check readers at banks). The architecture principle of weight sharing and local connectivity remains fundamental.

2. Energy-Based Models (LeCun et al., 2006). A framework for learning that assigns low energy to correct configurations and high energy to incorrect ones. More general than probabilistic models because it does not require normalized probabilities. Theoretical framework that informs LeCun's current research.

3. JEPA (Joint Embedding Predictive Architecture) (LeCun, 2022 position paper; Assran et al., V-JEPA 2024). LeCun's proposed path to human-level AI. Instead of generating pixels (like autoregressive or diffusion models), JEPA predicts representations in an abstract latent space. The claim: predicting in representation space avoids the computational waste of generating pixel-level detail and better captures abstract structure.

4. Self-supervised learning (DINO, DINOv2, I-JEPA, V-JEPA). Under LeCun's direction, FAIR has produced strong self-supervised vision models that learn useful representations without labeled data.

What still matters in 2026: CNNs remain in production for many vision tasks. The JEPA research program is ongoing but has not yet produced models that outperform autoregressive or diffusion approaches on standard benchmarks. Self-supervised vision (DINO/DINOv2) is widely used.

Affiliations: Mila (Quebec AI Institute, founder and scientific director), Universite de Montreal. Turing Award (2018) jointly with Hinton and LeCun.

Key contributions:

1. Neural probabilistic language model (Bengio et al., 2003). One of the first papers to train neural networks on word sequences with learned embeddings. Introduced the idea that words could be represented as dense vectors and that a neural network could model the joint probability of word sequences. This is the ancestor of all modern language models.

2. Deep learning revolution (Bengio, multiple papers 2006-2012). Systematic work on training deep networks: curriculum learning, denoising autoencoders, understanding gradients in deep networks. The theoretical and empirical groundwork that made deep learning practical.

3. Attention mechanism (Bahdanau, Cho, Bengio, 2014). Introduced attention for neural machine translation: instead of compressing the entire source sentence into a fixed vector, let the decoder attend to different parts of the source at each step. This idea is the direct ancestor of the Transformer's self-attention.

4. GFlowNets (Bengio et al., 2021-present). Generative Flow Networks: a framework for sampling diverse, high-reward objects proportional to a reward function. Applications in drug discovery, combinatorial optimization, and causal discovery. An active research program.

What still matters in 2026: The attention mechanism is the foundation of the Transformer. Word embeddings are universal. GFlowNets are an active research area with practical applications.

Affiliations: OpenAI (co-founder, Chief Scientist, 2015-2024), Safe Superintelligence Inc. (co-founder, 2024-present). PhD student of Hinton.

Key contributions:

1. AlexNet (Krizhevsky, Sutskever, Hinton, 2012). Co-designed the architecture and training procedure that won ImageNet 2012.

2. Sequence-to-sequence learning (Sutskever, Vinyals, Le, 2014). The encoder-decoder architecture for mapping variable-length sequences to variable-length sequences. Used LSTMs to encode a source sequence into a fixed vector, then decode it into a target sequence. Direct predecessor to the Transformer encoder-decoder.

3. GPT scaling (2018-2023). As OpenAI's Chief Scientist, Sutskever oversaw the scaling of GPT models from GPT-2 (1.5B) through GPT-4. The core thesis: scaling model size and data produces qualitatively new capabilities. This bet defined the field's trajectory.

4. Alignment focus (2023-present). Sutskever's departure from OpenAI and founding of SSI (Safe Superintelligence Inc.) signals a belief that safety research must be the primary focus, not an add-on.

What still matters in 2026: Sequence-to-sequence is the architectural paradigm underlying all encoder-decoder models. The scaling hypothesis he championed remains the dominant paradigm, though the emphasis has shifted from pure scale to data quality and reasoning-focused training.

Affiliations: Anthropic (co-founders, CEO and President). Both previously at OpenAI (Dario as VP of Research).

Key contributions:

1. Scaling laws for neural language models (Kaplan, McCandlish, Henighan, ..., Amodei, 2020). While at OpenAI, Dario Amodei was senior author on the paper establishing power-law relationships between model size, data size, compute, and loss. These laws guided compute allocation decisions across the industry.

2. Constitutional AI (Bai et al., 2022). Alignment method where the model critiques and revises its own outputs according to explicit written principles. Reduces dependence on human preference labels and makes alignment criteria transparent.

3. Anthropic's safety-focused research agenda. Under their leadership, Anthropic has published significant work on mechanistic interpretability, honest AI, and responsible scaling policies.

What still matters in 2026: Scaling laws remain the primary planning tool for training runs. Constitutional AI influenced alignment practice across the industry. The organizational model of Anthropic (safety-focused, benefit corporation structure) influenced how other labs think about governance.

Affiliations: Google DeepMind (co-founder, CEO). Nobel Prize in Chemistry (2024) for AlphaFold.

Key contributions:

1. AlphaGo (Silver, Huang, ..., Hassabis, 2016). Defeated the world Go champion using deep RL with Monte Carlo tree search. Demonstrated that deep learning combined with search could solve problems previously considered decades away.

2. AlphaFold (Jumper, Evans, ..., Hassabis, 2021). Solved the protein structure prediction problem. Given an amino acid sequence, AlphaFold predicts the 3D structure with experimental accuracy. AlphaFold 2 was released as open-source and has been used to predict structures for virtually all known proteins. AlphaFold 3 (2024) extended to protein complexes and ligands.

3. Gemini (2023-present). Under Hassabis's leadership of the merged Google DeepMind, the Gemini model family was developed as a natively multimodal system.

What still matters in 2026: AlphaFold transformed structural biology. AlphaGo proved that deep RL plus search works for complex sequential decisions. The DeepMind research philosophy of combining neural networks with search/planning remains influential.

Affiliations: OpenAI (founding member, led GPT-2/3 training), Tesla (Director of AI, Autopilot), independent educator (2023-present). PhD student of Fei-Fei Li at Stanford.

Key contributions:

1. Neural network education. Karpathy's blog posts ("The Unreasonable Effectiveness of Recurrent Neural Networks," 2015), Stanford CS231n lectures, and YouTube tutorials introduced a generation of practitioners to deep learning.

2. nanoGPT (2023). A minimal, readable implementation of GPT training in about 600 lines of PyTorch. Designed for education: you can read the entire codebase and understand how a GPT is trained.

3. Tesla Autopilot. Led the computer vision team for Tesla's self-driving system, applying large-scale neural networks to real-time perception.

4. Eureka Labs / AI education (2024-present). Focus on using AI to improve technical education.

What still matters in 2026: Karpathy's educational materials remain among the best introductions to neural networks. nanoGPT is a standard teaching tool. His influence is primarily pedagogical rather than through novel architectures.