Machine learning models are growing increasingly powerful in their abilities, whether that might be in processing natural language, tackling the intricacies of computer vision, or any other number of exciting applications that are emerging.
But we are finding out that as these models grow larger and larger, so do their corresponding carbon footprints, especially when it comes to creating and training these complex models.
This trend has led to some experts developing tools capable of monitoring the carbon impact behind machine learning models, as well as laying down the groundwork for mitigation approaches like carbon-aware computing.
Yet, most of these tools measuring AI’s carbon footprint are still in relatively early stages of development, and as some recent studies show, their mileage can vary depending on the context.
Now, an interdisciplinary team from the Allen Institute for AI, Microsoft, Hebrew University, Carnegie Mellon University, and AI startup Hugging Face is presenting a more sophisticated approach that uses location-based and time-specific data to measure more accurately the operational carbon emissions of popular AI models.
The new work, which was recently presented at the ACM Conference on Fairness, Accountability, and Transparency (FAccT), differs from previous methods as it documents the energy usage of server GPUs as a series of measurements, instead of oversimplifying it as a sum tallied 0ver the entire period of model training.
In addition, the team’s approach also tracks these fluctuating measurements alongside a series of data points that indicate the local emissions per kilowatt-hour (kWh) of energy used.
These calculations can ultimately help users understand the hidden carbon costs of their cloud-based workloads, or the carbon intensity of how much carbon emissions are produced per kilowatt-hour of electricity consumed wrote the team: “Arguably the single most important contribution of this paper is the simplest: a presentation of the software carbon intensity (SCI) as a proxy for carbon emissions for a given cloud instance as it is running.”
The team’s work involved training 11 machine learning models of varying sizes and applications, with training periods ranging from one hour on one GPU, to eight days on 256 GPUs.
Energy use was documented at regular intervals, in addition to tracking the different carbon intensities during different times of the day, in 16 geographical locations at 5-minute intervals. This approach allowed the researchers to compare models’ energy consumption as it related to geography, time and type of energy generation.
“The carbon intensity of a grid is sensitive to small changes in carbon-intensive generation, and can vary by both location and time,” explained study co-author Jesse Dodge in a blog post.
“Each region incorporates a different mix of energy sources, so can vary widely. Carbon intensity varies by hour, day, and season due to changes in electricity demand, low carbon generation (wind, solar, hydro, nuclear, biomass), and conventional hydrocarbon generation. As a result, there are many opportunities to shift computing resources to capitalize on these variations: this is known as carbon-aware computing.”
During their experiments, the team came up with some eye-opening findings. For instance, they found that the carbon emissions generated in training some of the lighter AI models were equivalent to that of charging a cellphone, whereas one of the larger models tested — such as a 6.1-billion parameter “transformer” model — was trained to only 13% completion, and yet produced a “staggering” amount of carbon equal to powering a house for a year in the United States. One can only imagine the emissions involved in training even larger models like OpenAI’s groundbreaking GPT-3 NLP model, which consists of 100 billion parameters.
The team found that the biggest factor in reducing emissions was selecting the best geographical location. For example, in training the commonly used NLP model BERT, the team found that emissions could be halved by training it using renewable energy sources located in countries like Norway or France.
The researchers found that even the time of day when the training occurred had an impact; training a model in Washington state during the night resulted in lower emissions, as that is when electricity is produced by hydroelectric power alone, rather than being mixed with energy from gas-fired power stations during the day.
In addition to these findings, the team also evaluated two optimization algorithms for reducing emissions of AI workloads on the Microsoft Azure cloud compute platform.
The first of these was “Flexible Start”, which delayed training for 24 hours and benefited smaller models the most. In contrast, the team’s “Pause and Resume” approach would halt training during hours of peak emissions but would favor larger models as long as it didn’t double training time.
While the team’s work focuses solely on the operational carbon emissions of training AI models and doesn’t take into account the carbon emissions associated with building the hardware, cooling data centers and so on, the team nevertheless pointed out that more comprehensive carbon-aware approaches will become vital in ensuring the future sustainability of machine learning models.
“Carbon accounting and reporting is becoming increasingly common in machine learning, with conferences such as NeurIPS requesting that submissions report their emissions and recent work reporting the emissions incurred,” noted the team. “However, it has yet to become the norm in our field, and we are still lacking systematic information regarding the environmental footprint of training machine learning models and how we can reduce it. In this paper, we argue that if members of the machine learning community had access to information about the carbon emissions of their actions, they could adapt their decisions to reduce these emissions while still meeting the computational needs for their end tasks.”
Read more in the team’s paper.
Images: Luis Ruiz via Pexels; Allen Institute for AI.