Training§

Training costs are proportional to the amount of data the model is trained on. The more data points there are, the longer the training takes. This means that it’s always possible to make training more expensive by throwing more data in. There’s no strict upper bound on how expensive training can get, but we can still estimate a range based on how people are actually doing training in practice.

In 2019, the cost of training a language model on standard hardware ranged from 17 kWh to 174 kWh³:

Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP

In 2022, training costs for the BLOOMz model family of small LLMs were benchmarked, and the range for total training costs was 11,000 kWh to 51,586 kWh:

Luccioni, A. S., Jernite, Y., & Strubell, E. (2023). Power Hungry Processing: Watts Driving the Cost of AI Deployment? (Truncated) Table 5: The BLOOMz models from our study with their training energy cost (from [Luccioni]), finetuning energy cost (from [Muennighoff]) …

	BLOOMz-7B	BLOOMz-3B	BLOOMz-1B	BLOOMz-560M
Training energy (kWh)	51,686	25,634	17,052	10,505
Finetuning energy (kWh)	7,571	3,242	1,081	543

But it is possible to pour massive amounts of energy into training language models. The more data you use as input, the more work is required to train a satisfactory model. In more recent history, Meta’s Llama 3 large language model’s small and large sizes were trained in 2024 for 1.3 and 6.4 million 700-watt GPU hours, respectively, with a maximum theoretical power usage of 910,000 kWh and 4,480,000 kWh.

Generative models go through the same process of training. The Stable Diffusion v2 model was trained on A100 PCIe 40 GB cards running for a combined 200,000 hours, which is a specialized AI GPU that can pull a maximum of 300 W. 300 W for 200,000 hours gives a total energy consumption of 60,000 kWh. This is a high bound that assumes full usage of every chip for the entire period; SD2’s own carbon emission report indicates it likely used significantly less power than this, and other research has shown it can be done for less.

So the energy consumption of model training is somewhere in the range of 11,000 kWh to 52,000 kWh for moderate-sized LLMs, 40,000 kWh to 60,000 kWh for more expensive image generators, and up to 5,000,000 kWh for global-scale LLMs like Llama and ChatGPT.

Inference§

An inference operation (using an AI model that’s already been trained) takes energy proportional to the model’s complexity. Again, this is intuitive: if the dictionary is larger, it takes more time to search. A response from a model trained on vast amounts of data takes more processing time than a simple one trained on a minimal set. A chatbot that returns answers in complex language takes more energy than a tool limited to yes/no answers. A large image takes more energy to generate than a small one. In other words, we expect the computation power required will be proportional to both the domain and range of the function.

Luccioni, A. S., Jernite, Y., & Strubell, E. (2023). Power Hungry Processing: Watts Driving the Cost of AI Deployment?

The trend is that categorization² and recognition operations are less expensive than generation operations (0.002 vs. 0.007 Wh).

LLM-based text processing is slightly more expensive. Other research finds that a ChatGPT text generation operation takes 2.9Wh to 4.0 Wh, and 1000 messages would take 2.9 kWh to 4 kWh.

Google’s published estimates indicate that 3 Wh is also the energy cost of each interaction with their LLMs.

But the costs of image generation are higher. Luccioni et al. (in the table above) found that among generation operations, text generation is less expensive than image generation (0.047 vs. 2.9 Wh). The most popular generative image model that can be run on consumer GPUs is Stable Diffusion. Benchmarking with codecarbon, with 20 samples, Stable Diffusion 1.5 takes ~0.09 Wh per image. Images from diffusion models can reach a cost of 3 Wh/image as a high upper-bound, but reaching this requires a larger model, and 100+ iteration steps as opposed to the standard 20—30.

Power Hungry Processing: Watts Driving the Cost of AI Deployment? Figure 1: The tasks examined in our study and the average quantity of carbon emissions they produced (in g of CO₂eq) for 1,000 queries. N.B. The y axis is in logarithmic scale.

We can also estimate the energy costs of modern black-box services that don’t release this data by using what we do know. Midjourney runs in real time, and takes 30–60 seconds to generate a batch of four images. As a high bound, if that process is running on a top-of-the-line GPU running at a full 450W for the full 60 seconds, that represents power consumption of 7.5 Wh per image set, and 1.88 Wh per subimage.

So the energy consumption of an inference operation is somewhere in the range of 1.0 Wh to 7.0 Wh for standard operations, and to 4 Wh to 7.5 Wh for more expensive jobs like image or video generation.

7.5 Wh per shot is expensive for a single computer operation. That takes us out of the realm of normal high-performance processing like arithmetic and into the realm of things like visual effects or 3D rendering. But that makes sense when you remember that the work being done is actually visual effects and 3D rendering. Relatively complex work, relatively high cost.

Data centers§

Most AI processing, whether training or inference, is done at scale in dedicated data centers. This uses a considerable amount of energy: data centers are already significant energy consumers, and increased load from AI work only exacerbates that.

If you’re not familiar, data centers are dedicated facilities for running servers. Data centers are “the cloud”: instead of running your own servers, you can rent computer power from experts who are very good at keeping computers from turning off. For the last decade, it’s been true that dedicated service — whether it’s a cloud service, chat program, or just any website — is probably running in a data center.

Data centers are what make all the good stuff go.

While different kinds of energy use aren’t individually tracked, for the purposes of analytics, “data center” is a discretely tracked category of power consumer. And data center energy use (combined) has been continually rising. From Bloomberg:

AI Is Already Wreaking Havoc On Global Power Systems | Bloomberg Altogether, data centers use more electricity than most countries
Only 16 nations, including the US and China, consume more

Sources: Bloomberg analysis of BloombergNEF and DC Byte data

Note: Data center energy consumption through Q1 2024. National energy consumption levels are actual through 2022 and projected for 2023 and 2024.

Data centers used 200 to 500 TWh in 2018, or 0.55 billion kWh per day. By 2024, that number increased to 350 TWh, or 0.95 billion kWh/day. As their graph points out, both of those numbers are higher than the total use of some countries.

The utility value of AI is highly contested§

The problem we immediately run into if we try to think about the proportional cost of AI is that there is no consensus on whether it’s ultimately useful. This matters because if it’s not useful, all energy spent on it is automatically wasted, so that’s our answer right there.

This is a deep topic, and one I plan to talk about in depth in another article (which I have a lot of notes for already). But, if you’re not familiar, here is a quick summary of the discourse.

I think it’s self-evident that AI services can be useful. If an AI service can answer a question correctly, that provides value. If one can generate usable text, or a usable graphic, that provides value. And, usually, this can all be done by AI much faster than doing it with human labor. By virtue of being a tool that sometimes works, the utility provided has to be something above zero.

I am not the first person to land on this position. Quoting from a few thoughtful people:

Kyle Orland, “Taking a closer look at AI’s supposed energy apocalypse” | Ars Technica If you cherry-pick the worst examples of AI screwups, it’s easy to see the entire sector as a misguided use of limited money and energy resources. But if you’re a programmer who’s getting twice as much done with AI coding tools or a customer service manager seeing productivity gains from employees who can consult with AI, that may seem like electricity well spent.

Nicholas Carlini, “How I use “AI”“ …the reason I think that the recent advances we’ve made aren’t just hype is that, over the past year, I have spent at least a few hours every week interacting with various large language models, and have been consistently impressed by their ability to solve increasingly difficult tasks I give them. And as a result of this, I would say I’m at least 50% faster at writing code for both my research projects and my side projects as a result of these models.
…
You might think—why would someone write an entire article justifying that language models are useful??! Isn’t that obvious?!? But there seem to be a (large?) contingent of people out there—in the academic literature, in the software engineering space, and also in the media sphere—who proclaim widely that LLMs contribute nothing, are just another hype cycle, and in a few years will die having had no impact on the world. I will be arguing these people are wrong because current LLMs are already useful.

So that’s the position to beat. But there are a number of arguments to do this:

There are moral arguments that AI is inherently unethical, not because of its efficiency, but due to problems with training methods, data sources, and intellectual property violations.

There are utilitarian arguments that AI-generated artifacts are generally harmful to public life, because they pollute the internet with well-formatted-but-wrong documents, making it harder for everyone to learn true things across the board.

There are labor arguments that AI services will allow companies to replace whole categories of human jobs with Capital they can own outright, allowing the owners of the new means of production to take a much larger share of revenue as pure profit, enriching themselves while workers go unpaid. What makes this particularly dangerous is that, from the prospective of the corporations, this might be the most efficient thing to do even if the resulting work is worse than peoples’.

For the sake of the energy question, let’s put a pin in the ethical question, since that’s really a separate topic. “Is using AI an unethical waste of electricity” is the only place these topics overlap, and whether it’s a waste of electricity or not is already the question we’re trying to answer. Otherwise, the ethical question is on a totally separate dimension.

If AI usage is inherently evil or necessary to oppose as part of a labor struggle, we don’t really care about the energy question because the answer is AI mustn’t be used no matter what. If it’s unethical it doesn’t matter how efficient it is, it shouldn’t be used at all. Energy isn’t really a factor in that case.

So for this article about energy, let’s make the middle-of-the-road assumption that AI services are useful and valuable, but not magically so. Let’s just conceptualize AI as a new tool that can be used productively, even if we’re not convinced the energy cost is low enough to make it worth using. And, at least in some cases, AI generative software can produce outputs comparable to human work in a practical sense. (Obviously this has a massive social and economic impact, but for now let’s just use it to do energy comparisons.)

So if AI is at least somewhat useful, and we know the rough ranges for the energy costs, how does that proportional cost compare to other things?

Cryptocurrency§

As I mentioned earlier, the AI boom feels a lot like the blockchain cryptocurrency push of a few years ago. Like cryptocurrency, AI is a tech fad, it requires data centers, it consumes more energy than a webserver… the comparison is extremely natural.

In fact, naturally seguing from cryptocurrency to AI is what Alex de Vries (the expert I’ve already cited) did:

Alex de Vries is a PhD candidate at the VU Amsterdam School of Business and Economics and the founder of Digiconomist, a research company dedicated to exposing the unintended consequences of digital trends. His research focuses on the environmental impact of emerging technologies and has played a major role in the global discussion regarding the sustainability of blockchain technology.

So people are coming off cryptocurrency and applying that same energy to AI, and I think it makes complete sense to do that. But, while the two trends look similar from a distance, they’re really very different, for two major reasons.

Image generation§

A much closer race is AI image generation. I think this is a much better exercise, since image generators were the most expensive models to train and use.

Is using ~3 Wh to 7.5 Wh to AI-generate an image a waste of energy compared to just doing the real work yourself?

Research§

We can apply this same logic to any task. First, consider the separate preparation/usage phases, then identify a comparable unit of work for each, then see how the energy usage requirements compare.

A LLM response takes 10-15x more energy than a Google search, sure. But does using a tool like ChatGPT as a research assistant consume more energy than searching for as long as it takes to get the same information? An LLM response just has to save you 10 clicks to break even, so as long as you’re really using it and not just asking something you could easily search yourself, it’s worth it.

This is something I’ve found makes ChatGPT genuinely useful: its ability to be a general conceptual index. If there’s a topic I’m not familiar with, I can describe the kind of thing I’m talking about in general terms, and ChatGPT can give me an outline of the topic to use as a launching point. To do that research with Google, I’d at least need to know what keywords to start with, which you often don’t.

For instance, here’s a question I asked back in 2023 (I don’t use ChatGPT much):

Some chocolate from a candy bar melts, but then rehardens in the refrigerator. Part of the chocolate separates into a tasteless white substance. What is the white substance?

Here I’m actually able to describe a phenomenon and work backwards to the terminology needed to do more robust research on the topic. (It was fat bloom.) With ChatGPT, I can ask a real question, get an approximate answer back that gets me on the right track, and refine from there with external research.

This ask-check-iterate loop seems extremely valuable when compared with traditional research. In order to get this information you’d normally need to be a subject-matter expert yourself or have access to one. Alternatively, with something like Wikipedia, you could find a general topic and try to browse through lists of subtopics; if what you were looking for was named clearly, you might be able to find it from there. But LLMs are able to replace that task of searching through the whole space with something much closer to an automatic, indexed lookup.

Turning a task from a search a human has to manually perform into a one-shot lookup is an incredible advancement. This enables people to do work that would otherwise take many man-hours of research and consultation to accomplish without the LLM tools. As for energy-efficiency, training and inference costs for LLMs are both less than the energy costs of image generators, and we’ve already seen how that went.

Entertainment§

Taking a step back from industrial comparisons, AI use even seems fine when compared to entertainment. Thinking about how whether a use “pays for itself” is good, but using it just for fun is fine too.

As I mentioned earlier, a computer can only draw so much power.

Watching one hour of Netflix takes around 0.08 kWh to 0.8 kWh. Let’s take the conservative assumption that in your scenario, watching an hour of Netflix costs 0.08 kWh. If you instead used that hour talking with an LLM like ChatGPT for no purpose other than entertainment, you could generate a reply every two minutes and still use less power than streaming. (At 0.8 kWh, you can message every 14 seconds!)

Sitting down and doing nothing but generate Stable Diffusion nonstop for an hour with a 200 W GPU takes the same 200 Wh that playing any video game on high settings would.

Any normal consumer AI use is going to be equivalent to these medium-energy entertainment tasks that we basically consider to be free. I’ve turned on a streaming service just to have something playing in the background, and probably so have you. AI just doesn’t use enough power to be more wasteful than just normal tech use.

Software§

(Nicholas Carlini, “How I use “AI”“ covers this topic in great depth, and I recommend it if this train of thought interests you.)

I was planning on writing about how the preparation/usage model was structurally similar to software development, with a clear software development stage followed by multiple usages, all of which consume energy. But that’s a challenging comparison to make, since there’s so much variance in the domain of “what software can do”. Worse, I’m not aware of any real data around either power requirements of enterprise software development, nor is it clear to me how much real work these “AI assist” tools do. In other words, quoting Wikimedia’s Chris Albon,

Chris Albon@chrisalbon

I have a hard time describing the real value of consumer AI because it’s less some grand thing around AI agents or anything and more AI saving humans a hour of work on some random task, millions of times a day.

Sat Jul 20 15:00:12 +0000 2024

I just experienced this the other day. I needed to quickly automate some clicking but I didn’t remember the exact javascript semantics and didn’t want to context-switch to reading the documentation, so I just asked

In the javascript development console, I have a NodeList returned from a query selector. Write a console snippet that calls .click() on each item in the list, but with a one-second delay between each call.

and got back exactly what I needed:

You can achieve this by using setTimeout within a loop to introduce a delay between each .click() call. Here’s a snippet that does that:

1 2 3 4 5 6 7

const nodeList = document.querySelectorAll('your-selector'); // Replace with your actual selector nodeList.forEach((node, index) => { setTimeout(() => { node.click(); }, index * 1000); // 1000ms = 1 second });

This code will call .click() on each item in the NodeList with a 1-second delay between each call. The index * 1000 ensures that the delay increases for each subsequent item.

This saved me a few minutes and eliminated the overhead involved in thinking through a new development problem. And yes, it probably replaced 10-page clicks too.

But, for something larger and less incidental, I was reminded of a conversation I saw. The AI service Claude is, apparently, able to build simple software tools and user interfaces just from conversational description.

That’s really cool. But that’s also a much better basis for a cost/benefit comparison.

None of the tools in that conversation thread are particularly impressive by themselves. A good text editor already has a feature to diff text, good composition software already tells you your document’s character/word/paragraph count, and a good image editor has a color-picker as a basic feature. Still, it’s useful to be able to do any of these things quickly from a browser if you don’t have the relevant program open, as evidenced by the fact that many mini websites like this exist.

Let’s take the color picker example. With access to the internet, I could probably put together something serviceable in an hour. If Claude can spend power to do that work for me, based on what I’ve seen so far, I would expect whatever additional energy cost using the AI tool involved (if any!) to be made up for in the time and energy it saved me.

But the value of Claude here isn’t that it would save me an hour, it’s that it would enable someone without as much development experience to make a tool that might otherwise take them days.

That’s a significant factor, but not one that slots cleanly in to our proportional cost calculation. That’s a benefit that’s hiding under “utility”.

Unfortunately, in the same way, there are also hidden costs. We also need to consider the impacts and implications this energy use implies, and there are significant problems under that rug. Because there are energy problems with AI; they’re just not that every time you use chatgpt they have to burn a barrel of coal. That, we can rule out.

That leads us into Part 2.