Contact Time accuracy when compared with other systems and high speed video?


#1

When I compare RunScribe (RS) data with Garmin’s Running Dynamics (GRD), Milestone Pod (MP), Sensoria (S), Wahoo Tickr (WT), and High Speed Video (HSV), I’m finding that the RS is generally reading rather too long.

  • GRD and MP seem to read close to the HSV.
  • S is wildly too long (one of many problems)
  • WT is reading a little short, but only slightly
  • RS is long by more than 10%

Any suggestions?


#2

Hi Jonathan, sounds like you’re collecting a lot of data!
Before I started working for Runscribe, I was involved in sports research and product testing. There’s a lot of variability in the devices available on the market, so to validate and fine tune the Runscribe foot strike and toe off detection algorithms we used the gold standard of high speed video.

Could you tell me a bit more about the set up you used? I’d like to try and figure out why the Runscribe measures are different from high speed video.


#3

Sure; I used a standard treadmill with a high speed video camera positioned just above the platform to give a good view of the landing surface. I tested with the RS on both the heal (solidly glued in) and the laces with similar results. I used frame advance to determine the first and last contact. I took a sample of several strides from different sections of the run, and they all came out fairly similar. What else would you like to know?


#4

Thanks. A few quick questions - what frame rate did you use for your high speed video and how did you define initial contact and toe off? From my experience even using manual digitization of ‘gold standard’ high speed video has a margin for error.

Just to give you some context - before I joined RunScribe, I was on a research team at a university that RunScribe hired to do an external validation study to test and fine tune the foot strike and toe off detection algorithms. The study used 16 participants, at 3 speeds each, and the participants had a variety of foot strike types, different levels of running experience, and all wore their own choice in shoes, to ensure we were testing the robustness of the algorithm in as many conditions as possible. To allow for ms accuracy, running was filmed at 1000 fps, with a resolution focused entirely on a tiny area right at the treadmill belt. The RunScribe data and high speed video were synchronized, so that we were comparing each step from the high speed video to the same exact step detected by RunScribe. Ten consecutive steps from steady state running were digitized for each condition, and both left and right were measured simultaneously (4 RunScribes in total).

Even using top of the line high speed video equipment, there were often 3-5 frames for foot strike and 2-3 frames for toe off that were incredibly hard to distinguish between and pinpoint exactly when contact starts and ends. Is it when first deformation of the shoe is visible? Is it when there appears to be no space between the shoe and the treadmill? To circumvent the potential source of human error, shoes were marked on the heel and toe, and digital lines imposed on the video taken to ensure each footstep for a participant was digitized at the exact same point, so while there may be some error in choosing the correct frame, the difference should be consistent for each participant, for example participant A may have a human error of 5 ms, and participant B might have an error of -5 ms, but the error is consistent for each participant.

Sorry this is getting long winded! It was a painstaking and time consuming process that I wouldn’t wish on anyone. From the results, it was found that RunScribe was within the gold standard high speed video data by an average of 3 ms. This includes running at 12 minute per mile pace, which is more difficult to detect foot strike and toe off because steps are less impulsive, but we wanted to see how the algorithm performed in all running conditions. I’ve attached one of the regression plots from this study below.

We’ve had a number of questions regarding accuracy & validation, and we want to be transparent in our approach, especially given the range of accuracy you noted in the devices in market. I’ve been working on a summary of research to share with our community, and will include the link on this thread when we publish.

We are confident in our algorithms and data, but know that many of our users are doing their own analysis - which we fully support! I wanted to share how difficult the process can be and how much there is to consider when interpreting the results. I’d be happy to chat with you more about your results or any future validations you have planned, if you’d like to email me at contact@runscribe.com we could discuss your work directly.


#5

My frame rate is 240 FPS, which gives a ~4ms resolution. I measured the GCT as both point of initial contact (disappearance of the air gap) and first compression of the midsole. I’m seeing RS measure about 15ms longer than the longer time. I’m averaging 4 frames across a test, and I’ve repeated the test 6 times.


#6

@fellrnr thanks so much for making this post and starting the discussion about the “How” for runScribe metrics. I have made comparisons of runScribe, Milestone Pod, and raw data from a 3D accelerometer demo board sold by Analog Devices (ADXL345). It has puzzled me that while rS always reports a solid 12 forefoot strike, Milestone always reports ‘heel strike’ for same shoe, lace mounting. This will occur even when I go out for a 1 miler and deliberately and come down flat foot mid strike. I just wrote it off to different teams developing different algorithms for slightly different purposes, and the fact the measurements that went into the development would probably never be revealed.

@Eileen, and thank you for that detailed description of the experimental footstrike determination (and that fantastic graph!). It inspires confidence in runScribes effort and product. Footstrike-to-toe-off time is easy to judge on video and certainly appears in a lot of studies, however I’m not sure how useful it is and here is why:

Without going into the calculations here, the gist is that I studied the signature of footstrike and toe-off on the ADXL345 graph, using the time between them and the running pace to figure out flight time. Flight time could then be used to determine vertical oscillation distance. (At the time, I was wanting to see what my ‘take off angle’ was, and how it compared to 45 degrees, the perfect ballistic angle). The vertical height reached always came out ridiculously low (like 2 cm), compared to other accerlometer based measuremts I took at chest level, and which other people reported (more like 5 to 10 cm). Taking a cue from correspondence with John L. however, I re-examined the raw acceleration data for a dead flat spot indicating ‘foot solidly planted in stance, vertical acceleration = zero’, and found that ballistic calculation of launch height based on the duration of that signal jibed very nicely with other measurements of vertical oscillation.

Conclusion: at least for me, while the foot is in contact with the ground much longer than the flat portion of the stance, it is not doing much pushing as the heel is coming up off the ground, foot rolling up to toe-off. The body mass is essentially launched at the point the flat portion of the acceleration trace ends, and the foot and leg are along for the ride.

I’m absolutely delighted with the related ‘per cent flight time’ addition to the metrics and have been playing around with that. Haven’t checked how it relates to vertical oscillation yet, but it’s a very useful metric regardless.


#7

Hi guys, just wanted to let you know the report from the validation study I mentioned has been posted on Running Unraveled, and can be found at this link:

http://www.runningunraveled.com/2016/08/02/runscribe-validation-research-study/

We’ve also started a new thread called RunScribe Data Validation where we’ll be sharing any new research that is published.

Thanks!