Activity/Fitness Monitor Showdown
OK, I admit it. I’m a data junkie. I just totally believe that you can’t improve what you can’t measure. So, I want to measure everything. Although, it’s even better when someone or something does the measuring for me. With this in mind, I recently decided to figure out what was the best way to measure my physical activity – movement, calories, weight lifted, stairs climbed, etc. Unfortunately, some of this data remains hard to come by. While machines in the gym – treadmill, elliptical, stationary cycle, etc. all generate some types of data. This data isn’t normalized across machines and, generally, cannot be exported for tracking.
The answer seemed to lie in the new crop of activity monitors available. The small devices worn on the wrist or kept in the pocket to track the steps one has taken, the calories burned, elevation climbed and so forth. Unfortunately, the current crop of activity monitors don’t really even try to cover some of the data I was looking for. Even worse, I had heard that many of the facets of activity they claim to cover are not all that accurate. Since my interest in gathering activity data was bordering on a need rather than desire, I had to find out what my options were. So I tried several trackers out.
My pseudo-scientific test included most of today’s popular devices plus one monitor that runs on my Android phone (there are other Android apps and, of course, iOS apps as well, especially for Apple phones with the M7 processor).
- Withings Pulse
- Fitbit Flex (thanks Brad Feld)
- Garmin Vivofit
- Jawbone UP (thanks Shawn Broderick)
- Moves App
There are several more, of course, but my arms are short. I wore these devices daily for about a month. It was still cold outside so I could hide this embarrassing electronic armband with the sleeves of a shirt. There are lots of features of these devices that I don’t cover here. If you’re going to make a purchase, you should hit the companies’ web sites for complete information.
I need to note that I didn’t test all types of activities that these monitors track. Since these devices are for the arm or pocket, cycling isn’t accurately tracked (nor is it claimed to be) and I am not a runner, so I didn’t even test for running. I also didn’t test the sleep functions of any of the trackers. I move around a reasonable amount each day, I frequently spend time on an elliptical trainer and in the gym lifting weights or doing body-weight exercises. I am a reasonably fit and active person. The question is, would the data reflect this.
Let me get right to the bottom line: These devices measure some activities moderately well and others either poorly or not at all. They all greatly depend on a certain type of body movement that their accelerometers (motion detectors) can pick up. A person who has a hard-pounding walking style will register more activity than one who floats over pavement, for example. If your primary activities are non-aerobic (weight lifting), these devices are useless. And even if they are aerobic, but only involve smooth movements (e.g. cycling), very little data is acquired. They are, basically, walking and running monitors. Additionally, the data gathered is best used as a comparison of the user’s activity over time because the absolute accuracy of these devices is questionable.
Here are a couple of examples that bear this out . . .
The reports below are generated by each of the devices. The first three – by the Garmin, Withings (side-by-side) and Fitbit (below), respectively – are clips from their web pages. Unfortunately, the next two – from Moves and Jawbone – don’t provide a graphical web interface. The data is only available on a computer by downloading it into a spreadsheet via the web. All the devices have phone apps that display the data graphically. While it’s nice to be able to access the data on a phone, I much prefer being able to see and manipulate the data with a computer on the web. Personal preference.
This is the data from May 10 – a randomly chosen day (I did the test on the elliptical 14 times during the month). All devices were worn throughout the day. You can see that the data on number of steps varies wildly.
All reported correctly that the primary movement started at about 7:30 pm and lasted for about an hour. During that time I was on an elliptical machine which can very accurately track the number of steps taken. The elliptical reported, roughly, 6,500 steps were taken, making the reported values of less than 5,000 for the whole day a bit suspect.
The distance covered also varies a lot between devices. It ranges from 2.09 mi (Fitbit) to 4.19 miles (Withings). Regardless of what distance I actually traveled during the day, the 2x difference in range makes me question all of the data. FWIW, the elliptical claims I ran/climbed more than 6 miles during that session.
On another day, I did a similar test on the elliptical with all devices in my pocket instead of on my wrist. The results were different – all values were higher, but the variance was just as high.
The data from April 21 is below in the same order as previously reported. Unfortunately, there is no Jawbone data for this day. This was a moderately active day with a concentrated weight lifting session from 2:30-3:30 pm. Note that none of the apps register much activity during this time. For most of the day, I was just moving around, doing whatever I needed and wanted to do without actually “exercising.” For this, most of the trackers were more aligned, however, there is still almost a 2x difference between the lowest (Moves) and highest (Garmin) in step count.
Clearly, the lack of activity reported during a very active weight lifting session shows that these trackers are not a reliable way of tracking this type of activity data.
Some thoughts about each of the trackers . . .
- Device fits in pocket nicely or on the wrist with a watch-like band
- Great display of all data, scrollable to see results from other days
- Touch sensitive screen for scrolling through data
- Micro-USB port on device for charging (this is a pro because as an Android user, I always have a micro-USB charger with me)
- Hard to read display in sunlight
- Not water resistant
- Replaceable battery lasts for a year (others need recharging after 10-14 days)
- Bright display easily readable in sunlight
- Red reminder indicator to get your ass off the couch
- Water resistant
- Data is combined with that from other Garmin devices to give a bigger activity picture
- Bluetooth syncing failed frequently
- Web site is difficult to negotiate
- Small, easily moves from wrist band to pocket
- Water resistant
- No data display – just some LEDs showing progress towards the day’s goals
- I found the wrist band hard to put on even after a couple of month’s usage
- Requires USB dongle for recharging
- Runs on phone so has optional access to GPS data – knows how fast your moving and where you are
- Runs on phone so it’s almost always with you
- Runs on phone so it consumes battery power
- No graphical data available on the web
- Looks the most like jewelry
- Water resistant
- No data display – just a colored light to tell you when you’ve achieved your goal
- No graphical data on the web
- No wireless communication – must connect to computer to download data (newer version of hardware apparently has Bluetooth)
- Requires USB dongle for recharging
Where do I go from here . . .
None of these devices are perfect or, for that matter, even very good in an absolute sense. As I said earlier, they do a decent job indicating your relative activity from day to day and in that way, they can disclose and track some valuable metrics. Many of these devices have other features that may increase their value to the user as well. The Withings Pulse can also track your pulse and blood oxygen levels and the Garmin Vivofit always shows the current time, for example.
For me, I think I’m going to move to a combination of devices. Perhaps Moves on my phone because it’s so transparent for daily activity (I’m carrying it anyway) and either the Garmin Vivofit or Withings Pulse for when I’m purposefully exercising. I’m then going to use the HumanAPI to combine the data so that I can track my overall activity in one place. Or, perhaps I’ll get some help with my OCD-ish need for collecting data and drop the whole thing altogether.
Thank you for doing so much work and the informative data shared.
Nice comparison! I went through a few Jawbone UPs and have a Fitbit Flex now. I’m only using it for two features really – have I been a deskbound shut in today: Yes/No, and the silent alarms are a killer feature for not waking up the rest of the family in the morning.
Thanks for the comment, John. I forgot to mention the silent alarm thing. I agree, that’s a killer app all by itself.
Awesome work, thanks for sharing. Perhaps one day the accuracy of activity trackers will get good enough that their reported values will converge on some actual true values, but until then we can still get plenty of use out of relative tracking to see day to day progress.
Personally, I’m betting that the next ‘wave’ of wearable activity trackers are going to be tackling specific individual user needs beyond just activity tracking. At Backtrack we’re currently working on a wearable tool to help people with back pain monitor the relationship between their pain and their movement, for instance, but there are plenty of great opportunities for new tools to do specific jobs past the basic ‘how much did I move’. Maybe they’ll all converge into HealthKit someday, who knows.
Hey Alex, Backtrack sounds cool. I think it will be one of several, perhaps specific devices, that really help get to a truly quantified self. Thanks for the comment.