Maps are Great Mmkay. Now Let’s Get to Testing Them.
Now that we’ve got choices (plural!) for maps on CarPlay, the testing and evaluation floodgates have opened and we at CarPlay World couldn’t be happier.
We love this [CarPlay] stuff.
When Google Maps and then Waze became available with iOS 12 in September, we settled into a fairly constant mindset of switching between them every few days to pick the best for our specific manner of driving and locating destinations. See our Apple Maps vs. Waze comparison to get an idea of how we approach maps/mapping/getting to where you’re going in CarPlay.
How do you test the three mapping apps for accuracy? Why, you say a destination and watch them work. The goal is to have as few distractions and as little further human input as possible — after all, you are driving.
It’s not the Journey, it’s uh… The Destination
Colorado State Capitol
200 E Colfax Ave, Denver, CO 80203
This test is so easy. For humans. For computers however, phones included, “state capitol” — our search phrase for this test — is hard… it’s not a “proper noun” business, like for instance “Starbucks”. My phone and these mapping apps have to do some extra work. In this case, they have to know which state I’m in before they can do the rest of the mapping calculations. And they have to know I’m looking for the state capitol… that I’m looking for a single location.
Apple Maps: Pass
Understood the request? Yes
Remembered the request? Yes
Result: Passed
After the dictation, I was given a choice screen to pick the correct destination. Subsequent “state capitol” voice commands didn’t elicit this choice — it just became my destination. Nice.
By remembering my request, we avoid — excuse the pun — an extra “round trip”. Next time, Apple Maps won’t offer me a choice, and I don’t need to involve myself in choosing. Remember, I’m driving.
Waze: Fail
Understood the request? No
Remembered the request? N/A
Result: Failed
Waze just didn’t get it. Literally, I heard “I didn’t get that” or some variation repeatedly while saying “state capitol”. It was all for naught. To a passerby I probably looked like a lunatic: enunciating “state capitol” loudly and clearly, over and over, to my car’s interior.
I tried speaking normally at the first go. “I’m not sure I understand”. STATE CAPITOL. “I didn’t get that.” STATE. CAPITOL. “I’m not sure I understand.” Etc.
I tried about ten times, then gave up and used “Colorado state capitol”. That worked. I used the same test environment as the Apple Maps test: driving around 25 MPH, no music playing, normal road/traffic noise, windows up. I even tried when stopped at a light to eliminate the already very low ambient road noise.
Google Maps: Fail
Understood the request? No
Remembered the request? N/A
Result: Failed
Google Maps acted a lot like Waze in its failure. It’s probably no coincidence, because Waze became a Google company in 2013. Five years is enough time to cross-pollinate them so they use the same engine, or at the very least, use many of the same methods to do their work. Which is why we see them fail similarly.
Both apps got in trouble not in hearing or parsing my phrase, but by failing to understand that I was looking for, simply, the Colorado State Capitol building. This isn’t a display, UI or mapping problem, it’s a “deep” AI problem.
