When you’re trying to catch the bus, there are few things as important as an accurate countdown—and few things as frustrating as an off-base ETA.

Sometimes you’re left waiting at the stop for a bus that is supposed to be coming “in two minutes” for 10 minutes straight. Other times the prediction is too early, and you’re left stranded because the bus passed before you could even get to the stop.

When a prediction hits the mark, though, it feels like magic when the bus pulls up right as the countdown strikes zero. And accurate ETAs are essential to making public transit a reliable way to get around.

In Transit’s quarterly survey of app users, improved accuracy of real-time information is regularly the number one thing riders say would get them to take public transit more. And research shows that accurate real-time ETAs do, in fact, increase transit usage and boost customer satisfaction.

Given all that, shouldn’t there be a consistent yardstick to measure the accuracy of real-time ETAs? Some kind of metric that gives the entire industry a shared way to know how real-time countdowns perform? (After all, it’s hard to improve if you don’t even know where you stand!) Wouldn’t it be great if this benchmark covered every single real-time ETA prediction: for every trip, every stop, every route, and every transit system?

Turns out, it now exists. It’s called the ETA Accuracy Benchmark.

What transit agencies are saying

RTD-Denver (Denver, CO)

“RTD is proud to have been involved in the 2016 effort with Arcadis IBI Group to develop a framework for measuring prediction accuracy, and we have been following the work to update it. We support the effort to develop and align on this new ETA Accuracy Benchmark and encourage the industry to join together around it.”

—Will Adams, Senior Manager, Customer Care at RTD-Denver

Miami-Dade Transit (Miami, FL)

"Climate change is no longer a distant threat; it's a daily reality. In Miami, we face five months of extreme heat each year, with temperatures exceeding 90 degrees and over 50 days above 100 degrees Fahrenheit. To protect riders from these harsh conditions, we need to deliver accurate real-time arrival information so that they can easily catch their bus or train and minimize waiting time outside. I'm thrilled to support this new benchmark to enhance passenger information quality not just in Miami, but across the entire transit industry."

—Carlos Cruz-Casas, Chief Innovation Officer at Miami-Dade County Department of Transportation and Public Works

CapMetro (Austin, TX)

"We are highly focused on providing the best rider experience possible, and we know that starts with having accurate real-time passenger information. Bad data is worse than no data in today’s connected world. CapMetro fully supports the ETA Accuracy Benchmark to give the industry a common definition to align our efforts around. We’re excited to see how we compare to other transit agencies benchmark ETAs and discover opportunities to further improve our rider experience and collaborate with our peers.”

—Daryl Weinberg, Transit Systems Architect at CapMetro

The past and present of measuring ETA accuracy

While there’s a common definition now, that wasn’t always the case. Over the years, multiple companies and transit agencies have measured real-time accuracy differently.

In 2015, IBI Group worked with RTD-Denver to develop the first modern accuracy yardstick. Their fundamental insight was to ask a basic question:

1. At 8:30 AM, what was the predicted ETA for bus X to arrive at stop Y—and when did that bus actually arrive at the stop?
2. ☝️… At 8:31 AM, the same question
3. ☝️… At 8:32 AM, the same question
4.  Times infinity, across every bus and every stop.

To assess what a “good” prediction was, they set up some benchmarks: if you looked up a transit departure far into the future, it was okay if the ETA prediction was off by a few minutes, give-or-take.

But they were much stricter as the bus got closer to your stop, especially when it came to early predictions—after all, you can still catch a bus that’s a little late, but it’s a lot harder to hop aboard if it’s already passed you by!

Put simply, IBI and RTD-Denver's methodology is a measure of whether a real-time prediction is accurate relative to when the bus actually arrives. It’s like aiming at a target that’s moving towards you, with a bullseye that gets smaller and smaller the closer it gets.

Over the years, this framework was used by different companies and transit agencies to measure real-time accuracy, in part because it’s easy to explain and easy to calculate. In Boston, the MBTA used it to select the vendor that performed best to improve accuracy for its real-time bus information system. Swiftly used it to measure the accuracy of predictions it was generating. Transit used it to measure how its machine-learning prediction algorithm compared to data provided by transit agencies.

So far, so good. Except there’s one big problem: each company and each transit agency used slightly different numbers to define what counted as “accurate.”

In one case, ETA predictions between 0 and 2 minutes before arrival would get grouped together for analysis. In another, between 0 and 3 minutes. One might consider a prediction of 18 minutes accurate when the bus arrived 12 minutes later. Others would label that same 18-minute prediction as inaccurate.

There’s nothing inherently wrong about having different definitions. After all, each one was crafted by transit professionals doing their best to paint an accurate picture of what riders were experiencing. And each organization could use its own definition to make comparisons and measure improvement over time.

But when it came time to compare notes, they were speaking totally different languages.

A shared definition

It’s important to have a consistent definition across the industry. That way, a real-time accuracy rating is the same no matter which organization produces it. Having consistent definitions puts us all on the same page about what’s accurate and what’s not.

It helps transit agency board members have confidence that staff are reporting numbers based on an accepted, industry-wide metric rather than a one-off definition. It makes it easier to perform analyses since the work of defining thresholds is already done. It allows different organizations to analyze different real-time feeds and come to common conclusions.

Which is why a coalition of companies and transit agencies support the ETA Accuracy Benchmark, a metric that is open to everyone across the industry. These thresholds are designed to be easy to understand, consistent, and to reflect the expectations that actual riders have when they use public transit.

This makes it easy to measure real-time feeds against each other, and for anyone to produce an analysis that gives an apples-to-apples comparison over time.

What’s next?

We’re starting today with ETA accuracy, but there are many more things to look out for when measuring the quality of real-time information. Things like:

• Accuracy (covered by the ETA Accuracy Benchmark)
• Reliability (what is the real-time feed’s uptime rate?)
• Percentage of trips with real-time (are there a lot of missing ETAs?)
• Jitter (are predictions yo-yoing up and down?)
• Ghost buses (are there ETAs but no bus is actually coming?)
• Surprise—aka zombie or bonus—buses (does a bus show up when there’s no ETA?)
• And more!

Additional organizations, like Cal-ITP, are emphasizing the importance of high-caliber real-time information, too. Together, we’re all rowing in the same direction as the industry determines industry-wide metrics.

MobilityData, the non-profit that governs GTFS and GTFS-realtime, is coordinating these efforts and working over the long run to provide more guidance to the entire industry about how to measure for high-quality data. 

Reliable real-time information is essential to transit service that riders can count on. Agencies need to know whether they’re delivering high-quality information to riders, and our industry needs common benchmarks to measure the quality of that information.

The ETA Accuracy Benchmark is a huge first step to getting everyone onboard.

What industry leaders are saying

Swiftly

“In 2015, I had the privilege of collaborating with RTD-Denver and my former colleagues at Arcadis IBI Group to develop the original method for measuring ETA accuracy. We started with trying to answer the question: 'How do we know if the ETAs that we are giving riders are improving their transit experience?' We were thrilled to see this method not only work but also be adopted and adapted widely—by transit agencies and companies. So, it is an incredibly proud moment for me to be part of the effort to update this method and develop the ETA Accuracy Benchmark.”

—Ritesh Warade, General Manager, Transit at Swiftly

Transit

“The ETA Accuracy Benchmark solves an all-too-common problem in public transit: knowing with certainty if our real-time countdowns help riders stay on track, or if we miss the mark and leave them stranded. At Transit, we’ve embraced the ETA Accuracy Benchmark to evaluate our own machine-learning algorithm, as well as predictions directly from transit agencies. This is particularly exciting for me, as I worked with former colleagues at the MBTA on an early version of this framework almost seven years ago to evaluate potential real-time vendors. With the ETA Accuracy Benchmark, the entire industry can now speak a common language that helps us deliver accurate information to riders.”

—David Block-Schachter, Chief Business Officer at Transit

Ito World

“Ito World proudly supports the ETA Accuracy Benchmark. It assures riders that their time is valued, agencies that their data is reliable, and the industry that we're all committed to excellence. By supporting this benchmark, we're not just endorsing a metric but fostering a culture of improvement and reliability in public transportation."

—Joseph Holmes, Vice President of Business Development (North America) at Ito World

Ualabee

"The Ualabee platform provides real-time data from various providers for many cities in South America. We understand the challenge of measuring the accuracy of real-time ETA predictions and how users experience this issue daily. That's why we support the ETA Accuracy Benchmark as a common methodology for measuring prediction accuracy."

—Luis Lenta, Backend & DevOps leader at Ualabee

Learn more

Check out the technical documentation for the ETA Accuracy Benchmark.

Have any questions? Want your organization to sign on as a supporter of the ETA Accuracy Benchmark? Wondering how to get started on analyzing the accuracy of your ETAs? Get in touch!