Does AI Actually Save You Time? What Four Studies Measured, and How to Check Your Own

“It saves me time” is the most common thing people say about AI tools, and it is a claim almost nobody checks. That matters more than it sounds, because the one study that compared what people believed with what a stopwatch recorded found the two pointing in opposite directions. This piece lays out what four controlled studies actually measured, what they have in common, and a logging method small enough that you can test your own use instead of taking anyone’s word for it, including the tool vendor’s.

What the controlled studies found

These are the four studies most often cited on the question. All four randomly assigned people (or tasks) to work with or without AI, which is what separates them from surveys where users report how productive they feel.

Study	Who and what	Measured result
Noy & Zhang, MIT (2023)	444 college-educated professionals, occupation-specific writing tasks, half given ChatGPT	Time taken fell by 0.8 standard deviations; graded quality rose by 0.4. Weaker writers gained most. Work shifted away from rough drafting toward idea generation and editing.
Brynjolfsson, Li & Raymond, NBER (2023)	5,179 customer-support agents given an AI assistant	Issues resolved per hour up 14% on average, up 34% for novice and lower-skilled agents, minimal change for the most experienced.
Dell’Acqua et al., Harvard/BCG (2023)	758 Boston Consulting Group consultants on realistic consulting tasks	On tasks inside the model’s ability: 12.2% more tasks finished, 25.1% faster, quality more than 40% higher. On one task chosen to be outside it: AI users were 19 percentage points less likely to get the right answer.
METR (2025)	16 experienced open-source developers, 246 real issues in their own repositories	With AI allowed, tasks took 19% longer. Beforehand the developers forecast a 24% speed-up; afterwards they still believed they had been 20% faster.

Read as a group, these do not contradict each other. They describe the same thing from different sides.

The pattern: checkable work and unfamiliar work

The gains cluster where the output is cheap to check. A customer-support reply, a press release, a product-idea memo: the person reading the draft can tell in seconds whether it is usable. The writing study is explicit that the tool replaced rough drafting and left people editing, and editing a draft you can judge at a glance is fast.

The gains go to people for whom the task is hard. The 34% figure belongs to novices; veterans in the same call centre barely moved. The METR developers were the opposite case in every respect: experts, working in codebases they had maintained for years, on tasks where they already knew roughly what the answer looked like. There was little for the tool to add and a real cost to prompting, waiting and reviewing.

The losses appear where a wrong answer looks like a right one. The consulting study’s authors call this a jagged frontier: some tasks that seem similar in difficulty fall on opposite sides of what the model can do, and nothing on the screen tells you which side you are on. The consultants who did worse were not careless. They were handed a fluent, confident, incorrect analysis and had no quick way to see that it was incorrect.

Feeling faster is not evidence. The METR result is the uncomfortable one. A 39-point gap between measured and perceived speed means self-report is close to useless here. Generating text in thirty seconds feels efficient even when the following four minutes of fixing it bring the total to what writing it cold would have taken. The authors are careful about scope, and so should anyone quoting them be: sixteen developers, early-2025 tools, no claim that the result holds for most developers or for newer models. The perception gap is the part that travels.

One question before you reach for it

All of that compresses into a single test: can I check this answer faster than I could produce it?

If yes, the tool is probably a win. That covers transformation where you hold the input (a messy pasted table into clean rows, a list of dates into another format, a long paragraph into a short one) and it covers first drafts of things you have been avoiding, where the value is not the text but that reacting is easier than starting.

If no, you are not saving time. You are moving the work to a place where the errors are harder to see. The typical cases are specific facts you cannot verify at a glance: the details of a pricing structure, how a particular feature behaves, whether an integration exists, what a contract clause means. A fluent wrong answer about any of these costs nothing until it reaches a client, and then it costs an awkward correction. The same logic applies to tools that summarise things you did not witness; it is why an AI meeting notetaker is only as useful as your willingness to spot-check it against the recording.

A ten-second log to test your own use

You do not need a study. You need a month of honest notes, and each entry takes about ten seconds.

1. Log every use, not the memorable ones. One line: what you were trying to do.
2. Time the whole exchange. Prompting, waiting, reading, fixing, verifying. Not the generation.
3. Write your without-AI estimate before you start, not after. Afterwards the estimate drifts toward whatever flatters the choice you made.
4. Mark each entry afterwards: saved time, cost time, or a wash.
5. Note whether you could verify the output at a glance. This is the column that will explain the other four.

Then sort by that last column. If the studies above describe you, the wins will sit almost entirely under “verifiable at a glance” and “I was stuck”, the losses under “specific fact I could not check”, and a large middle will be washes that felt productive.

Two cautions when reading your own log. It flatters the tool, because you only logged the occasions you chose to use it, and the choosing already reflects an expectation of success. And savings that are real are usually not concentrated: a handful of three- and five-minute wins a week is worth having, but it does not add up to a free afternoon, and it should not be the basis for a staffing decision.

What this means for what you pay for

The practical use of the log is a spending decision. If your wins are all transformations and first drafts, a single general-purpose assistant covers them, and the specialised add-ons stacked on top of it deserve the scrutiny you would give any other recurring charge. Whether an AI subscription is just a wrapper around a model you already pay for is a question your own log can answer better than a feature page. And when an AI feature is the stated reason a tool made the list you think you cannot cut, the log is the evidence for or against.

None of this is an argument for using these tools less. It is an argument that “it saves time” is a measurable claim, that the best available measurements say the answer depends on the task and on you, and that your own feeling about it is the least reliable instrument in the room.

Study figures checked September 18, 2026 against the METR write-up, the NBER abstract, the MIT working paper and the published abstract of the Harvard/BCG study. The Noy & Zhang paper later appeared in Science (2023) and the Dell’Acqua et al. paper in Organization Science*; the figures above are from the versions linked.*