If you're planning on using a wrist-based monitor to track your heart rate while walking, running, or cycling, a group of scientists at Stanford (in partnership with the Swedish School of Sport and Health Services in Stockholm) claim the Apple Watch is the monitor to get, with the smallest margin of error (2%) out of seven tested devices.
The experiment also looked at each device's caloric estimations (or "EE", for energy expenditure). Though the Apple Watch doesn't do poorly in this arena, that doesn't mean much: the lowest margin of error across the pack was 27.4% average, with a whopping 92.6% average error for the Fitbit Surge. In short: Still a long way to go when calculating calories burned effectively on a wrist-worn device.
How was this experiment conducted?
In studies like this, scientists are primarily looking at margins of error when determining what device works "best": In other words, you want a device that regularly reports within a certain margin of error as compared to the control heart rate, or "gold standard."
For this experiment, Stanford used the following for its gold standard:
Because so little testing has been done on wrist-worn devices, there is no "official" standard for such experiments:
As such, the scientists have also proposed a public repository of validated heart monitor data.
To do this first experiment, the scientists identified 45 potential manufacturers, then limited it to eight based on the following criteria:
After excluding the ePulse2, the experiment was left with seven devices.
It is interesting to note that neither Garmin nor Polar's sport-specific wrist trackers were included in this study — we don't know if they were originally considered and then discarded, but it's worth noting given both manufacturers' prior expertise in sport-specific heart tracking.
So what do the heart rate (HR) results mean?
Essentially, after all these tests, the scientists determined that the Apple Watch has the lowest margin of error when it comes to calculating heart rate while walking, running, or biking.
Most of the devices tested came within a median 5% margin of error throughout the tests, with only the Samsung Gear S2 falling outside the range on all activities (5.1% on cycling; a range of 6.5-8.8% on walking; and 6.8% total average).
So the Apple Watch is the best at heart rate for wrist-worn devices, right? According to this study, yes, but its competition is nipping at its heels — a less than 5% margin of error is still quite good when it comes to overall monitoring, so there's no need to throw out your Fitbit Surge if you're otherwise happy with it.
It's also worth noting that this experiment only tested wrist-worn devices in common exercise situations like biking, running, and walking — yoga, weight-lifting, and other wrist-bending activities were excluded, all of which have been known to negatively effect the accuracy of wrist-worn heart monitoring.
What about the caloric (EE) results?
"Calories burned" has always been a bit of a mysterious stat on wrist-worn devices, in part because the calculations behind energy expenditure (or EE) are obscured on a per-device basis. From the study:
As noted above, because there are a lot of variables involved in the calculation of EE — some that require user input, like height, weight, and activity type — it's much harder for any device to give you an accurate estimate. And the study proved it accordingly:
In other words: The Apple Watch may have had the fewest variations in energy expenditure when compared to the other devices in the study, but it still isn't anywhere near the level of accuracy provided by the study's gold standard.
What does this mean for wrist monitors going forward?
For health tech junkies, Stanford's study is actually an incredibly important step forward in getting more reliable data from our devices. Stanford's proposal for a "wearable sensor evaluation framework" alone is a pretty exciting development — if scientists standardize a baseline testing framework and data repository, it allows experiments to be done all over the world with large testing groups, getting us comprehensive data.
Essentially, the more scientific experiments done on wrist-worn devices, the better: More data leads to competition from manufacturers to better their sensors, which gives us (the end-users) even better devices down the line.
And Apple Watch users? For now, you can rest smugly knowing that you'll get a pretty accurate heart rate for most walking, running, and biking activities. (And hope that Apple works on a better system for measuring energy expenditure in the future.)
Serenity was formerly the Managing Editor at iMore, and now works for Apple. She's been talking, writing about, and tinkering with Apple products since she was old enough to double-click. In her spare time, she sketches, sings, and in her secret superhero life, plays roller derby. Follow her on Twitter @settern.
Since the beginning of the Apple Charitable Matching program, to Stanford University Hospitals, Cook himself has donated $50 million. Take from that what you will.
Given that the experiment made its entire repository and testing data available to other scientists to replicate and retest as they see fit, I'll take from it that Cook's $50 million has been well-spent on some great servers for the scientific community.
Makes me question my Fitbit Charge HR. I like the Apple Watch but dont care for the smartphone features, just want a device that performs as a fitness band. I wish Apple offered a budget Watch that removed all the smartphone features and sold at a fraction of the cost.
Hi ohezm, don't throw out your Charge HR just yet. The second paragraph in the article incorrectly quotes the study saying the Surge has an error rate of 92.6%; that actually belongs to the PulseOn. Reading through the rest of the study, the EE error rate with the Surge was actually 27.4%, which might be the best of the bunch and NOT the Apple Watch. So Apple was only ahead with HR accuracy. Also, the Charge 2 has an improved HR tracking, which is the technology behind the guided breathing sessions and advanced sleep tracking. I look forward to see the Charge 2 tested against the rest of the devices.
Your have to understand, I don't I've seen anything do better than apple on this site.
They didn't bother to test a single Android Wear Watch? Ok...
Android watches are irrelevant, insignificant, not to be considered in such studies. They simply don’t have the processing power to do anything other than tell time. You know, what you people say about Apple products. Deal with it.
First of all, let me give you some unsolicited life advice and suggest that in the future you refrain from saying "you people", as if you have someone completely figured out by a couple of sentences in a post. Case in point, you're implying that I'm an Android/Google fanboy and I can assure that is most definitely not the case. My current primary phone is an iPhone 7+ and I use my iPad Air 2 a lot during the week. I also have a Nexus 6P, a Nexus 9 tablet and my laptop is a Surface Pro 4. I'm very brand agnostic and a fan of technology in general, which is precisely why I found it very strange that they basically ignored one of the major smartwatch/wearable platforms on the market. Nothing more, nothing less.
It says Samsung G2
The Gear S2 runs Tizen, not Android Wear. Posted via the iMore App for Android
It may be the best but it is still not that accurate. Don't even get me started on the calorie counting.
OK I think there's something wrong with this data because THE ONLY COMPANY THAT COMES CLOSE TO CHEST WORN HEART RATE MONITOR is MIO Especially for GYM Exercises. I'm saying this with personal experience and you can even see that in many reviews. every other hand based heart rate monitor fails at tracking gym/hiit/cross fit exercises including MIO but the closest is the MIO.
As a serious road cyclist I would never consider a phone-based HRM accurate, instead we use a chest strap HRM that communicates by ANT+ or BlueTooth LE to a Garmin bike computer, problem solved.
Yeah I know that but the closest one is MIO not the Apple Watch.
Apparently the Apple Watch matched the Mio: http://i.imgur.com/fsloX6z.png
I do not see Kardia AliveCor in test ? Why a serious study would not take into consideration an existing wearable for AppleWatch (medical grade device by the way) ?
This study would have carried more eight had they included a broader range of devices, particularly high end models from Garmin and Polar, and also chest-band models for reference, since those are considered the gold standard. It almost seems as if they cherry-picked lower end products to make the Apple Watch appear more favorable .
Thank you for signing up to iMore. You will receive a verification email shortly.
There was a problem. Please refresh the page and try again.