In 2008, neural networks were considered a field past its prime. During my computer science graduate studies at Princeton, the prevailing sentiment was that while neural networks had shown promise in the late 80s and early 90s, progress had plateaued. Sophisticated yet seemingly more fruitful approaches like support vector machines were taking center stage in the AI research community. This perspective was reinforced in courses like COS 402: Artificial Intelligence, where neural networks were presented more as a historical footnote than the future of the field.
Unbeknownst to me, a revolution was brewing right within the same Princeton computer science building. A team led by Professor Fei-Fei Li was embarking on a project that would challenge this very notion and unleash the latent power of neural networks. Their focus wasn’t on refining neural network algorithms themselves, but on something far more fundamental: data.
Their ambitious endeavor was the creation of ImageNet, a groundbreaking image dataset of unprecedented scale. It comprised a staggering 14 million images, meticulously labeled across nearly 22,000 distinct categories. This massive dataset became the fuel that would ignite the deep Learning Boom, transforming artificial intelligence as we know it.
Fei-Fei Li, a pioneer in artificial intelligence and computer vision, speaking at the Clinton Global Initiative, highlighting her contributions to AI data and research.
In her insightful memoir, The Worlds I See, Fei-Fei Li recounts the initial skepticism surrounding ImageNet. Colleagues and mentors questioned the project’s practicality and relevance. One mentor cautioned her in 2007, “I think you’ve taken this idea way too far. The trick is to grow with your field. Not to leap so far ahead of it.”
The sheer scale of ImageNet presented immense logistical hurdles. However, the doubt extended beyond mere logistics. Many experts at the time were unconvinced that machine learning algorithms, constrained by the datasets of the era, could effectively leverage such a vast collection of images. As Li stated in a Computer History Museum interview, “Pre-ImageNet, people did not believe in data. Everyone was working on completely different paradigms in AI with a tiny bit of data.”
Undeterred by the prevailing skepticism, Li dedicated over two years to ImageNet. The project stretched her research funding and tested the endurance of her graduate students. Upon joining Stanford in 2009, she carried the ImageNet project and several dedicated students with her to California.
Initially, ImageNet’s impact was muted. Released in 2009, it garnered little immediate attention. However, 2012 marked a turning point. A team from the University of Toronto harnessed ImageNet to train a deep neural network, achieving unprecedented accuracy in image recognition. This revolutionary AI model, named AlexNet after lead author Alex Krizhevsky, unleashed the deep learning boom that continues to reshape our technological landscape today.
AlexNet’s success was not solely attributable to ImageNet. It also relied on CUDA, a platform developed by Nvidia that enabled the use of Graphics Processing Units (GPUs) for general-purpose computing. CUDA’s introduction in 2006 was met with its own share of doubt.
Thus, the AI learning boom of the past decade was the result of the confluence of three visionary figures who championed unconventional ideas in the face of widespread doubt: Geoffrey Hinton, who tirelessly advocated for neural networks; Jensen Huang, Nvidia’s CEO, who foresaw the broader potential of GPUs beyond graphics; and Fei-Fei Li, the creator of ImageNet, a dataset deemed excessively large by many, yet proved essential for realizing the potential of GPU-accelerated neural networks.
Geoffrey Hinton: The Neural Network Pioneer
A neural network is fundamentally a complex network of interconnected nodes, or neurons, numbering from thousands to billions. Each neuron operates as a mathematical function, generating an output based on a weighted average of its inputs.
Example of a handwritten digit “2” used in neural network training, illustrating the type of image recognition task that fueled early AI development.
Consider building a network to recognize handwritten digits, such as the number “2” shown above. The network would receive pixel intensity values from an image as input and output a probability distribution across the ten digits (0-9).
Training such a network begins with random initial weights assigned to connections between neurons. The network is then fed a series of example images. For each image, the network’s connections are adjusted: strengthened if they contribute to the correct prediction (high probability for “2” when shown “2”), and weakened if they lead to incorrect predictions. Through training on numerous examples, the network learns to accurately identify digits.
Early experiments with neural networks in the late 1950s focused on single-layer networks. However, enthusiasm waned as the limitations of these simple networks for complex tasks became apparent.
Deeper networks, with multiple layers, held greater promise. Yet, in the 1960s, efficient training methods remained elusive. The challenge lay in the complex, unpredictable effects of parameter adjustments within multi-layer networks.
By the 1970s, when Geoffrey Hinton began his career, neural networks had fallen out of favor. Despite this, Hinton remained a steadfast advocate, though initially struggling to find academic institutions supportive of his research. Between 1976 and 1986, he navigated through four research institutions: Sussex University, UCSD, the UK Medical Research Council, and finally, Carnegie Mellon, where he became a professor in 1982.
Geoffrey Hinton, a leading figure in deep learning and neural networks, speaking in Toronto, Canada, discussing advancements in AI technology.
In a seminal 1986 paper, Hinton, along with former UCSD colleagues David Rumelhart and Ronald Williams, introduced backpropagation, a breakthrough algorithm for efficiently training deep neural networks.
Backpropagation operates by starting from the network’s final layer and working backward. For each connection in the final layer, it calculates a gradient, indicating whether strengthening the connection would improve the network’s accuracy. Parameters in the final layer are then adjusted based on these gradients.
Crucially, backpropagation propagates these gradients back to the preceding layer using a formula derived from the chain rule of calculus. This allows for the calculation of gradients in each layer based on gradients in subsequent layers. This process repeats, layer by layer, propagating gradients backward through the network.
While each training step involves small adjustments, repeated iterations over vast datasets—millions or even trillions of examples—gradually enhance the model’s accuracy.
Hinton and his collaborators popularized backpropagation, although they weren’t its original inventors. Its significance lay in making deep network training practical, reigniting interest in neural networks and paving the way for the learning boom.
Hinton joined the University of Toronto in 1987, attracting a new generation of researchers to neural networks. Yann LeCun, a French computer scientist, was among the first, spending a postdoctoral year with Hinton before joining Bell Labs in 1988.
LeCun applied Hinton’s backpropagation to train deep models capable of real-world tasks like handwriting recognition. By the mid-1990s, LeCun’s technology was successfully deployed in banks for check processing, marking an early practical application of neural networks.
However, attempts to scale neural networks to larger, more complex images encountered limitations. Neural networks once again faced setbacks, leading some researchers to shift their focus.
Yet, Hinton remained convinced of the superior potential of neural networks, patiently awaiting the data and computational resources necessary to validate his long-held belief and trigger the deep learning boom.
Jensen Huang: The GPU Visionary
Jensen Huang, CEO of Nvidia, speaking in Denmark, emphasizing the role of GPUs in powering the AI revolution and advancements in computing.
The central processing unit (CPU) serves as the brain of personal computers, executing instructions sequentially. While suitable for general software, demanding tasks like rendering 3D worlds in video games strain CPUs.
Graphics Processing Units (GPUs) emerged to address this, employing parallel processing. GPUs contain numerous execution units, essentially miniature CPUs, operating simultaneously to render different parts of the screen, enhancing graphics performance.
Nvidia pioneered the GPU in 1999 and quickly became the market leader. By the mid-2000s, Nvidia CEO Jensen Huang recognized the potential of GPUs beyond gaming. He envisioned their application in computationally intensive scientific tasks like weather simulation and oil exploration.
In 2006, Nvidia introduced CUDA, a platform enabling programmers to write “kernels,” short programs designed for parallel execution on GPU units. CUDA allowed complex computations to be divided into smaller, parallelizable tasks, significantly accelerating processing compared to CPUs.
However, CUDA’s initial reception was lukewarm. As Steven Witt noted in The New Yorker, Wall Street reacted with dismay. Huang’s vision of democratizing supercomputing was met with skepticism, as the demand for such capabilities outside niche scientific communities was unclear.
Despite initial doubts and a significant drop in Nvidia’s stock price, Huang persisted. He believed the very existence of CUDA would expand the supercomputing landscape. While CUDA downloads peaked in 2009 and subsequently declined, Huang’s conviction remained.
Crucially, Huang didn’t initially envision AI or neural networks as the killer application for CUDA. However, it turned out that Hinton’s backpropagation algorithm was ideally suited for parallel processing on GPUs. Training neural networks became the unexpected yet transformative application that drove the learning boom.
Witt highlights Hinton’s early recognition of CUDA’s potential. In 2009, Hinton’s research group utilized CUDA to train a neural network for speech recognition, achieving surprisingly high accuracy. Hinton then contacted Nvidia, recognizing the synergy between GPUs and neural networks.
Despite initially being denied a free GPU, Hinton and his students, Alex Krizhevsky and Ilya Sutskever, acquired Nvidia GTX 580 GPUs for the AlexNet project. Each GPU, with 512 execution units, enabled training speeds hundreds of times faster than CPUs. This computational leap was essential for tackling the massive ImageNet dataset and unlocking the deep learning boom.
Fei-Fei Li: The Data Visionary
Fei-Fei Li at the SXSW conference, sharing insights on AI, data-centric approaches, and the future of computer vision.
Fei-Fei Li’s journey to ImageNet began with her arrival at Princeton in 2007. Prior to this, during her PhD at Caltech, she had created Caltech 101, an image dataset of 9,000 images across 101 categories.
Caltech 101 revealed the critical role of large, diverse datasets in improving computer vision algorithm performance. It became a benchmark in the field, demonstrating the value of data-driven approaches.
Inspired by this, Li aimed to create a dataset of unprecedented scale at Princeton. She was captivated by vision scientist Irving Biederman’s estimate that humans recognize approximately 30,000 object categories. Li envisioned a comprehensive image dataset encompassing all commonly encountered objects.
A Princeton colleague introduced her to WordNet, a vast database cataloging 140,000 words. Li used WordNet as a foundation for ImageNet’s categories, focusing on tangible nouns and excluding verbs, adjectives, and abstract nouns. This resulted in around 22,000 object categories, from “ambulance” to “zucchini.”
Li initially planned to replicate her Caltech 101 approach: using Google image search to find candidate images and then manually verifying them. While feasible for Caltech 101, the scale of ImageNet demanded a more efficient strategy. She initially considered hiring Princeton undergraduates for image selection and labeling.
However, even with optimized processes, Li and her graduate student Jia Deng calculated that manually labeling millions of images would take over 18 years.
Amazon Mechanical Turk (AMT), a crowdsourcing platform launched by Amazon, provided a solution. AMT offered a global workforce that was not only more cost-effective than Princeton students but also highly scalable and flexible. Li’s team could access on-demand labor and pay only for completed tasks.
AMT drastically reduced ImageNet’s completion time from 18 years to two. Li describes her lab operating “on the knife-edge of our finances” during this period. Ultimately, they secured sufficient funding to have each of the 14 million images reviewed by three individuals, ensuring data quality.
ImageNet was finalized in 2009 and submitted to the Conference on Computer Vision and Pattern Recognition (CVPR). While accepted, it was relegated to a poster session, a disappointing outcome given the years of intensive effort.
To generate broader interest, Li transformed ImageNet into a competition. Recognizing the full dataset’s size, she created a smaller, more manageable version with 1,000 categories and 1.4 million images for the ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
The inaugural ILSVRC in 2010 attracted 11 teams. The winning entry, based on support vector machines, showed only marginal improvement over existing methods. The 2011 competition saw fewer participants, with another support vector machine winning with similarly incremental gains. Li began to question if ImageNet was indeed too challenging for existing algorithms.
However, the 2012 ILSVRC delivered a paradigm shift. Geoff Hinton’s team submitted AlexNet, a deep neural network model. Its top-5 accuracy of 85% was a staggering 10 percentage points higher than the 2011 winner.
Li’s initial reaction was disbelief. Neural networks, once considered outdated, had suddenly achieved a monumental breakthrough, marking the true beginning of the deep learning boom.
“This is proof”: The Dawn of the Learning Boom
Yann LeCun, a pioneer of convolutional neural networks, testifying before a Senate committee, highlighting the impact of deep learning on AI advancements.
The ILSVRC winners were announced at the European Conference on Computer Vision in Florence, Italy. Despite having a newborn at home, Fei-Fei Li knew she had to attend.
In Florence, Alex Krizhevsky presented AlexNet’s results to a packed audience of computer vision researchers, including Fei-Fei Li and Yann LeCun.
Cade Metz recounts that after the presentation, LeCun declared AlexNet “an unequivocal turning point in the history of computer vision. This is proof.”
AlexNet validated Hinton’s long-standing faith in neural networks and was a significant vindication for LeCun as well.
AlexNet was a convolutional neural network (CNN), a type of neural network LeCun had developed two decades prior for handwritten digit recognition. Architecturally, AlexNet shared similarities with LeCun’s earlier networks, but it was vastly larger. LeCun’s 1998 network had seven layers and 60,000 parameters; AlexNet had eight layers but a staggering 60 million parameters.
Training such a massive model was impossible in the early 1990s due to limitations in computing power. Even if sufficient computing existed, the lack of large training datasets would have been a major bottleneck. Collecting such datasets before Google and Amazon Mechanical Turk would have been prohibitively expensive.
This underscores the transformative impact of Fei-Fei Li’s ImageNet. While she didn’t invent CNNs or GPU acceleration, she provided the crucial training data that enabled large neural networks to realize their full potential, triggering the learning boom in AI.
The technology world immediately recognized AlexNet’s significance. Hinton and his students formed a company, promptly acquired by Google for $44 million. Hinton joined Google while retaining his Toronto academic position. Ilya Sutskever later became a co-founder of OpenAI.
AlexNet also cemented Nvidia GPUs as the standard for training neural networks. Nvidia’s market capitalization soared from under $10 billion in 2012 to trillions today, driven by the overwhelming demand for GPUs like the H100 optimized for neural network training.
Sometimes the Conventional Wisdom is Wrong: Lessons from the Learning Boom
“That moment was pretty symbolic to the world of AI because three fundamental elements of modern AI converged for the first time,” Li reflected. “The first element was neural networks. The second element was big data, using ImageNet. And the third element was GPU computing.”
Today, leading AI labs prioritize training massive models on enormous datasets. The demand for computational resources is so intense that tech giants are leasing entire nuclear power plants to power their AI data centers.
While this approach appears to be a direct consequence of AlexNet’s success and the ensuing learning boom, it also prompts a deeper reflection. Perhaps the true lesson of AlexNet is to question conventional wisdom.
“Scaling laws” have been remarkably effective in the 12 years since AlexNet, and further scaling may yield continued advancements.
However, we must avoid dogmatism. There is a possibility that scaling laws may eventually reach their limits. Should that occur, a new generation of nonconformist thinkers will be needed to challenge established paradigms and explore uncharted territories in AI, mirroring the visionary spirit that ignited the initial deep learning boom.
