Your Email Open Rate Is Mostly Robots: Why Open-Rate A/B Testing Is Broken

Somewhere in your last campaign report is a number that says roughly forty percent of your guests opened the email. Most of them did not. A good share of that figure is Apple’s mail servers opening it on their behalf, before the guest looked at their phone, and in many cases whether or not the guest ever did.

Key Takeaways: Apple Mail Privacy Protection pre-fetches the tracking pixel in email sent to its users, which registers as an open with no human involved. Litmus, measuring more than 1.5 billion opens, attributes 76.56 percent of all tracked opens to Apple clients and states plainly that privacy-protected opens “are not considered reliable opens”. The consequence is not that your open rate is a bit high. It is that open-rate A/B testing is measuring a quantity that cannot respond to the thing you are testing, which dilutes every result and raises the list size you need to reach significance by roughly five times. Clicks are not the fix either, because corporate link scanners fetch every URL and register as clicks. Revenue attribution, through both hard and soft association, is the only signal a machine has no reason to fake.

---

What MPP Actually Does

Open tracking has always been a trick. There is no “open” event in email. What exists is a one-pixel image with a unique URL, and when a mail client loads that image, the sender records a request and calls it an open.

Apple Mail Privacy Protection, introduced with iOS 15 on 20 September 2021, routes that request through Apple’s own proxy and fires it on delivery rather than on reading. The pixel loads. The sender records an open. The guest may be asleep.

This is not a bug or an oversight. It is the stated purpose of the feature, and Apple was entirely open about it in the launch announcement:

Mail Privacy Protection prevents senders from learning whether an email has been opened, and hides IP addresses so senders can’t learn a user’s location or use it to build a profile on them.

Read that again with your campaign report in mind. The company that makes the mail client used by most of your guests announced, back in 2021, that it had stopped you knowing whether your email was opened. The open-rate column did not disappear. It just stopped meaning what it says.

The Scale of It

The number that matters is how much of your reported engagement this accounts for, and it is larger than most operators assume.

Litmus, calculating from over 1.5 billion opens in September 2026, puts Apple clients at 76.56 percent of all tracked opens. That figure is the combined Apple category, covering Apple Mail, iPhone, iPad and Apple’s Mail Privacy, plus some third-party apps running on Apple devices, so it is not a clean measure of MPP alone. What Litmus does say about those opens is unambiguous:

opens from users opted into Apple’s Mail Privacy Protection do not pass along specifics of the email client version and platform. As such, they are not considered reliable opens

Litmus separately states that MPP affects roughly 55 to 60 percent of all email opens, though it does not name the study behind that range, so treat it as an industry estimate rather than a measurement.

For a hospitality list the real figure is likely at the high end rather than the low one. Leisure travellers skew toward consumer mail on consumer devices, which is exactly the population Apple’s default reaches.

Why This Breaks A/B Testing Specifically

Inflation alone would be survivable. If every number is forty percent too high, you adjust your expectations and carry on. The problem with A/B testing is different and worse, and it is worth walking through slowly.

A subject line test asks one question: did variant A persuade more humans to open than variant B? Machine opens cannot answer it. The pre-fetch happens when the mail is delivered, before any human has read any subject line, so it fires at exactly the same rate for both variants. It is not responding to your test. It cannot.

Watch it happen with real numbers

Say you send a campaign to 10,000 guests, split evenly, 5,000 per variant. Suppose 55 percent of your list is privacy-protected, so 2,750 of each 5,000 will register an automatic open no matter what you write in the subject line.

Now suppose subject line B is genuinely better. Among real humans it gets 24 percent to open, against 20 percent for subject line A. That is a solid four point win, the kind of result you would happily build a playbook on.

Here is what lands in your report.

Variant A: 2,750 machine opens, plus 20 percent of the 2,250 humans, which is 450 real opens. Total 3,200 opens out of 5,000, a reported open rate of 64.0 percent.

Variant B: 2,750 machine opens, plus 24 percent of the 2,250 humans, which is 540 real opens. Total 3,290 opens out of 5,000, a reported open rate of 65.8 percent.

Your genuine four point win shows up as 1.8 points.

Nothing was lost or hidden. The 2,750 machine opens sat in both columns and swamped the difference. In general, with m as the machine-open share:

what you observe = (1 - m) x what is really there

At 55 percent machine opens you see 45 percent of any real effect. The signal did not vanish, it got quieter.

Why a quieter signal costs so much more than you would guess

This is the part that surprises people, and it is worth saying in plain English before the formula.

Every measurement has noise in it. With a bigger sample, the noise shrinks, but it shrinks slowly: to halve your margin of error you need four times the data, not twice, because random error falls with the square root of the sample size.

Now run that backwards. MPP has halved the size of the difference you are hunting for. To resolve a difference half as large, you need your noise to be half as large too. And halving the noise costs four times the sample.

So the cost is not proportional. It is squared:

list size needed = 1 / (1 - m)^2

Machine-open share	You observe	List size needed
0%	100% of the real effect	1x
30%	70%	2.0x
50%	50%	4.0x
55%	45%	4.9x
70%	30%	11.1x

Back to the worked example. To call that 1.8 point gap with the confidence you would once have had on a 4 point gap, you need roughly five times the list: not 10,000 guests, but something near 50,000.

Most hospitality senders do not have 50,000 guests to test on. What they have instead is a run of A/B tests that declared winners on differences well inside the margin of error, and a subject-line playbook quietly built on coin flips.

That is the real damage. Not an inflated number on a dashboard, but a sequence of confident decisions made from noise.

Deriving Your Real Open Rate

The data is not lost. It is contaminated, and contamination can be separated if you know where it sits. There are two ways to do it, and they are worth understanding in that order.

The reliable way: measure the clean half

Every serious ESP reports an email-client breakdown. Use it to split your audience in two:

1. Recipients whose opens are privacy-protected.
2. Everyone else.

Throw away the open data from group one. It tells you nothing about human behaviour. Then compute the open rate on group two alone, using only those recipients in both the numerator and the denominator. That number is a measured human open rate, not an estimate, and you can quote it.

Stay with the worked example. Of 10,000 guests, 4,500 are not privacy-protected. If 990 of them opened, your human open rate is 990 / 4,500, or 22 percent. That is a real figure about real people. The 64 percent on your dashboard is not.

One honest limitation. The unprotected group is not a random sample of your list: it skews toward Android, toward desktop, toward corporate mail, and in hospitality that often means it skews away from exactly the leisure guests you care about. You are extrapolating from a biased sample. You are also doing enormously better than reading the blended number.

The quick way: back out the robots

If you want a correction without segmenting, you can reason it out from the blended figure. Your reported rate is a weighted average of two groups: the protected ones, who open at nearly 100 percent because a machine does it for them, and everyone else, who open at some real human rate.

Write R for your reported rate, m for the protected share of your list, and h for the human rate you are trying to find:

R  =  m x (nearly 1)  +  (1 - m) x h

Rearranged, that gives you:

h  =  (R - m) / (1 - m)

Run the example: R is 0.64, m is 0.55.

h = (0.64 - 0.55) / (1 - 0.55)
  = 0.09 / 0.45
  = 0.20, or 20 percent

Which is right, because 20 percent is what we built the example from.

When the formula returns nonsense, listen to it

Try it on a more typical reported figure. Suppose your dashboard says 42 percent and you believe 55 percent of your list is protected:

h = (0.42 - 0.55) / 0.45  =  a negative number

A negative open rate is impossible, so one of your inputs is wrong, and the formula has just told you something useful. The usual culprit is the assumption that pre-fetch fires on essentially every delivery. It does not: devices are off, mail goes unsynced, some recipients never trigger it. So either your true protected share is lower than you assumed, or your human open rate is in the low single digits.

Either way you have learned the important thing, which is that the blended number was never describing what you thought it described.

Use the segmented measurement for the baseline you quote internally. Use the formula as a sanity check. Then stop reporting a single blended open rate as though it meant something, and move your decisions onto metrics that do not depend on a pixel at all.

Clicks Are Not The Answer Either

The usual advice at this point is to switch to clicks, and it is half right. A click is harder to fake than an open. It is not clean.

Corporate mail security opens your links before any human sees them. Link-protection and sandboxing services, the Safe Links and Proofpoint and Mimecast layer that sits in front of most business mailboxes, fetch every URL in every message to check where it goes. That fetch is recorded by your ESP as a click. Nobody clicked anything.

So clicks carry their own machine contamination, and it is worse than a flat tax, because it is uneven:

• It concentrates on corporate and institutional domains, which is where your group, corporate and event enquiries live.
• It fires on every link in the email, so the more links you include, the more phantom clicks you earn.
• It often fires in a burst within seconds of delivery, which quietly poisons any send-time analysis.
• It varies by recipient domain, so comparing two segments can be comparing two different scanner populations.

Clicks per delivered is still a better test metric than open rate, and the denominator still matters: use clicks per delivered, never click-to-open, because click-to-open puts the contaminated open count underneath and you have simply swapped one distorted number for another. But treat a click as evidence, not proof.

Revenue Attribution Is The Only Clean Signal

There is exactly one event in this chain that a machine has no reason to fake, and it is the one you actually care about. A security scanner does not book a room. A privacy proxy does not buy an upsell. A bot has never paid a deposit.

Which is why the only unambiguous answer to “did this campaign work” is the revenue that followed it. Two kinds of link get you there, and you want both.

Hard association. The guest clicked through from the campaign and booked in that same path, so the booking is tied to the send deterministically. It is the cleanest possible evidence and it is also the rarest, because real guests leave the email, think about it, come back through a search result or a bookmark two days later, and book without ever retracing the link.

Soft association. The guest received the campaign and then booked, with no tracked click in between, and the platform matches the reservation back to the send. This is the one that recovers the majority of genuine influence, because that is how people actually behave. SendSquared does this against live PMS data, so the campaign revenue report shows the reservation number, the campaign, arrival and departure dates and the attributed revenue, with no UTM wrangling and no post-hoc stitching in a spreadsheet.

Hard association alone understates your email programme badly, because it only counts the guests who behaved like a funnel diagram. Soft association alone is looser evidence on any single booking. Together they give you the honest number, and crucially they give you a number that machine traffic cannot inflate in either direction.

This is also what makes the A/B problem tractable again. You cannot run a meaningful subject-line test on a metric that robots contribute to. You can run one on bookings, because the robots are not booking.

What to put in front of an owner

Stop sending blended open rate to owners and executives. It is the number most likely to be wrong by a factor of three, and it describes nothing they can act on. Send these instead:

• Revenue attributed to the campaign, hard and soft association together
• Reservations influenced, with arrival dates
• Clicks per delivered, labelled as directional rather than exact
• Human open rate from the unprotected cohort, if anyone insists on an open figure at all

What To Do On Monday

1. Pull your email-client breakdown and find your privacy-protected share. That one number tells you how much of your reporting is fiction.
2. Recompute last quarter’s open rate on the unprotected cohort only. Expect the real figure to be roughly half what you have been quoting.
3. Switch every automated A/B test off an open-rate winner. Revenue is the metric to settle on; clicks per delivered will do as an interim. If your platform only offers open rate, that is worth knowing about your platform.
4. Stop reporting blended open rate to owners and executives. Replace it with attributed revenue, hard and soft association together.
5. Re-examine any subject-line convention you adopted from an open-rate test run after 2021. There is a decent chance it was never significant.

Open rate had a long run as the headline email metric, and it was always a proxy. The proxy broke. The uncomfortable part is not that the number is wrong, it is that an entire layer of marketing practice, including most of the A/B testing built into email platforms, still optimises against it as though nothing happened.

If you want to see what your real engagement looks like when it is measured on actions rather than pixels, book a demo and we will go through your own numbers.