Machine learning (ML) is revolutionizing numerous fields, acting as a powerful branch of artificial intelligence and computer science. At its core, ML leverages data and algorithms to mimic human brain functions, progressively enhancing accuracy over time through experience.
ML algorithms are essentially statistical tools that identify patterns within vast datasets. This data can encompass a wide range of formats, including numerical values, textual information, images, user clicks, and any other digitally storable information, as highlighted by the MIT Technology Review.
Deep neural networks, inspired by the structure of neurons in the human brain, form the foundation of deep learning algorithms. These networks process information by receiving inputs and generating outputs, acting as sophisticated information messengers.
To enable these networks to learn the necessary parameters for accurate predictions, a process called “training” is employed. This involves feeding the network with data where the outcomes are already known.
A classic example is teaching a computer to differentiate between images of cats and dogs. This is achieved by inputting thousands of images of both animals, varying in breed, size, color, and pose. Through this extensive exposure, the network calibrates its internal parameters to accurately map an input image to the correct classification (cat or dog).
However, achieving high accuracy in complex tasks often demands an enormous volume of training data. This is where the innovative approach of physics-informed machine learning (PIML) becomes crucial.
The relationship between input and output in a machine learning model is termed a “map.” This map allows the model to make predictions in diverse and complex systems, such as weather forecasting. For instance, predicting New York City’s temperature might involve feeding the model historical weather data. Traditional machine learning suggests that more data leads to better predictions.
However, acquiring sufficient data for complex physical systems can be incredibly challenging and resource-intensive. Physics-informed machine learning offers a transformative solution. Instead of relying solely on massive datasets, PIML integrates pre-existing physical models and scientific knowledge into the machine learning process. In the weather prediction example, PIML would incorporate established physical models of temperature distribution patterns in New York City over time.
For centuries, scientists have developed sophisticated models to describe physical systems. Physics-informed machine learning ingeniously leverages this wealth of prior knowledge to guide and constrain the training of neural networks. By incorporating physical laws and equations, PIML achieves greater training efficiency and accuracy, even with smaller datasets.
Crucially, PIML doesn’t just feed data to the neural network; it imposes constraints based on the underlying physical model. This constraint is a defining characteristic of PIML. The input data, instead of being random, becomes part of a known physical process governed by established laws or equations. In the weather example, the model is constrained by known principles of thermodynamics and atmospheric science.
This approach offers significant advantages. Physics-informed machine learning drastically reduces the need for extensive training data, often by orders of magnitude, making it applicable in scenarios where data is scarce or expensive to acquire.
A Historical Perspective on Physics-Informed Machine Learning
Machine learning’s origins trace back several decades, with landmark moments shaping its evolution. A foundational paper in 1943 by logician Walter Pitts and neurophysiologist Warren McCulloch laid early groundwork by attempting to mathematically model thought processes and decision-making in human cognition. They proposed an abstract description of brain functions, demonstrating the potential of simple interconnected elements in neural networks to achieve substantial computational power.
In 1950, Alan Turing, a British mathematician, introduced the groundbreaking “Turing Test,” designed to assess a computer’s capacity for intelligent behavior. Turing argued that the question of whether machines can “think” was too vague. Instead, he proposed a practical test: the “Imitation Game,” where a human interrogator tries to distinguish between a computer and a human respondent based on their answers to questions within a fixed timeframe. Turing suggested that a computer’s “thinking” ability could be judged by its probability of being mistaken for a human.
Just two years later, Arthur Samuel developed the first computer learning program for checkers. This IBM computer program demonstrably improved its game play with experience, showcasing actual learning. Later in the 1950s, in 1957, Frank Rosenblatt created the perceptron, the first neural network for computers, designed to mimic human brain thought processes.
The concept of “Explanation Based Learning” emerged in 1981, introduced by Gerald Dejong. This approach involved computers analyzing training data to derive general rules by discarding irrelevant information. A significant shift occurred in this field around 1991, moving from a knowledge-driven approach to a data-driven paradigm. Scientists began developing programs that enabled computers to analyze large datasets and extract meaningful conclusions.
The term “deep learning” was coined by Geoffrey Hinton in 2006 to describe advanced algorithms that allowed computers to “see” and identify objects and text within images and videos. By 2011, IBM’s Watson demonstrated the power of machine learning by defeating human champions on the quiz show Jeopardy!.
Google Brain, also invented in 2011, marked another milestone. This deep neural network demonstrated the ability to learn and categorize objects autonomously. Shortly after its development, it famously learned to recognize cats by itself after being exposed to videos.
Machine learning is a cornerstone in the advancement of self-driving cars. While widespread adoption of autonomous vehicles is still unfolding, machine learning remains central to their development and continues to be a major focus for automotive manufacturers.
Beyond autonomous vehicles, machine learning is already integrated into everyday life, powering recommendation systems for online platforms like Amazon and Netflix, and playing a crucial role in fraud detection systems.
Physics-informed machine learning emerged in the 1990s, initially appearing in scattered research papers. The resurgence of machine learning around 2010 revitalized PIML, highlighting its potential.
A further advancement in 2014 was Facebook’s DeepFace. This algorithm achieved 97.25% accuracy in determining if faces in unfamiliar photos belonged to the same person, even with variations in lighting and angles. This accuracy rate was remarkably close to the average human accuracy of 97.53%, demonstrating near-human facial recognition capability.
The Significance of Physics-Informed Machine Learning
Physics-informed machine learning unlocks the potential for scientists to tackle previously intractable problems.
PIML can accelerate the development of improved pharmaceuticals by narrowing down the vast design space and guiding the selection of promising experiments.
One of its most significant promises lies in enhancing fluid flow prediction. Fluid dynamics is ubiquitous, relevant to systems ranging from blood flow in vessels to river currents. It is also critically important in the oil and gas industry. Physics-informed machine learning offers powerful tools to improve the simulation of fluid flows in complex geometries, an area that has traditionally been computationally expensive.
Advantages and Limitations of Physics-Informed Machine Learning
A key advantage of physics-informed machine learning is its speed in generating results – often in fractions of a second. The neural network’s architecture allows for efficient processing of new samples, leading to rapid output. When training is successful, PIML achieves impressive prediction accuracy.
However, PIML is not without limitations. Insufficient training data hinders its performance. In many scientific and engineering contexts, acquiring large, representative datasets can be prohibitively costly or time-consuming.
While massive image datasets of cats and dogs are readily available, data for training models in areas like material science or specific chemical reactions may be scarce.
Even with abundant data, challenges remain, especially in high-stakes applications like self-driving vehicles. Despite being trained on billions of data points, self-driving systems cannot guarantee complete safety, as unforeseen scenarios can still arise.
Achieving truly safe autonomous driving requires an immense amount of training data, ideally encompassing billions of hours of real-world driving footage. However, capturing all possible driving scenarios, especially rare but critical events like accidents or encountering unusual road debris, is exceptionally difficult and expensive.
This data scarcity means self-driving cars might encounter situations not fully represented in their training data. Automakers are acutely aware of this limitation and are actively working to mitigate it. Despite these efforts, earlier predictions of widespread self-driving car adoption by 2020 have not materialized, as noted by industry analysts.
Furthermore, physics-informed machine learning, being trained on specific data distributions, may struggle to generalize to conditions significantly different from its training data. Neural networks excel at interpolation—making predictions within the scope of their training data—but are less effective at extrapolation—predicting outcomes outside of that scope.
For instance, a neural network trained to recognize cats and dogs using images taken under specific lighting conditions might perform poorly when presented with images taken under drastically different lighting. This is because neural networks are excellent function approximators within their trained domain, but their accuracy can decline when applied to data outside of that domain.
Consider training a neural network solely on images of common drug molecules to predict drug efficacy. While it might accurately predict the effectiveness of known drugs, its predictions for entirely novel drug compounds, with different structures and properties, would be significantly less reliable.
Future Horizons for Physics-Informed Machine Learning
Beyond materials discovery and compound design, physics-informed machine learning holds immense promise for personalized medicine.
Imagine a tool capable of designing customized drug therapies tailored to an individual patient’s unique genetic makeup, medical history, and specific needs.
Personalized medicine, a rapidly advancing field, utilizes an individual’s genetic profile to guide decisions related to disease prevention, diagnosis, and treatment.
Scientific research indicates that a substantial portion of the variability in drug response is genetically determined. Factors such as age, nutrition, health status, environmental exposures, and concurrent therapies also influence drug effectiveness.
Physics-informed machine learning can potentially account for complex interactions among these factors that might be beyond the capacity of individual clinicians or even medical teams to discern. Well-trained neural networks can identify subtle patterns and correlations in patient data that are invisible to human observation.
This capability can be further enhanced by incorporating counterfactual reasoning into neural network training, improving the robustness and interpretability of these models.
PIML’s role in optimizing experimental design is also highly promising. Researchers can leverage PIML to identify the most informative experiments to conduct to achieve specific scientific goals. Machines can often devise experimental strategies that are unconventional and beyond human intuition, potentially leading to more efficient and impactful research outcomes.
Physics-Informed Machine Learning Innovations at Pacific Northwest National Laboratory
The Pacific Northwest National Laboratory (PNNL) has been at the forefront of physics-informed machine learning research for many years. Their ongoing work has yielded significant insights with implications for areas ranging from drug development to industrial control systems. In May 2021, PNNL researchers announced findings from a study exploring graph generative models for designing novel drug candidates targeting SARS-CoV-2 viral proteins, demonstrating the potential of PIML to accelerate pandemic response efforts.
PNNL’s research in this area is timely and crucial for accelerating drug discovery in future pandemics and beyond.
In April 2020, PNNL scientists integrated prior knowledge of molecular structure into a neural network to improve the identification of molecular water configurations with desirable properties. This approach enhances the interpretability of deep neural networks in chemical applications and promotes their wider adoption within the scientific community.
In the same month, PNNL announced progress in applying PIML to differential equations, which are fundamental tools in engineering domains such as industrial system modeling and control, where safety and performance are paramount.
PNNL has also made significant strides in subsurface modeling. In July 2020, they presented a physics-informed neural network approach that requires fewer measurements to accurately estimate both the state and parameters of subsurface fluid flow. This research has applications in environmental remediation efforts, such as cleanup operations at the Hanford Site, a decommissioned nuclear facility in Washington state.
Around the same time, PNNL unveiled a novel method for incorporating noisy measurements into the training of physics-informed neural networks. Using a probabilistic framework, they extended PIML to quantify the impact of measurement noise on state estimation and system identification across various problems. This advancement is particularly valuable for utilizing real-world measurements, which invariably contain some degree of noise, to train more robust and reliable PIML models.
