Putting Apps to the Test

Given the level of competition in the mobile application space, developers are under a lot of pressure to bring a bug-free product to market as efficiently as possible.

Open platform communities like Symbian and Android, and in some instances, even carriers, are all working on ways for developers to test their wares before pricing and posting them in front of consumers. Some developers go it alone and hope for the best; others use third-party testing services. Still others submit their apps to small test groups that offer feedback.

King: Releasing untested software will kill revenue growth.

Stephen King, CEO of Mob4Hire, a provider of “crowd source” testing solutions for developers, said that making sure an application is ready for market is key to survival in the dog-eat-dog world of mobile applications.

“It is all about word of mouth. The problem developers face is, ‘How do I get discovered?’ It only takes two or three bad user reviews before you say, ‘I’m not interested in downloading it.’ This whole notion of releasing software untested out to the real market basically is going to kill your revenue growth,” King says.

Faraz Syed, CEO of DeviceAnywhere, notes two reasons testing is such a big deal.  “Generally, in the software business, catching a bug pre-production is significantly cheaper than catching a bug in production,” Syed says. But he also points out that operators usually get stuck with the high cost of support calls and reimbursement costs when an application doesn’t work right, which doesn’t make them happy. “As a result, the operator may choose to shut that developer down,” he adds.

THIRD-PARTY OPTIONS
Syed and King are well versed in what it takes to thoroughly test an application before it goes to market. Their respective companies represent two different camps within the third-party testing market. Where Mob4Hire’s solution is a relatively new one, DeviceAnywhere finds its roots in a long line of software testing solutions.

DeviceAnywhere offers developers remote, over-the-Internet access to more than 2,000 handsets that are located in seven countries around the world.

“Our customers subscribe to our service, log into their account and have immediate access to all of our available phones,” Syed says. “They can interact with the phone live online. When a button is pressed on thescreen, the actual button is electrically pressed on the phone that’s remotely sitting in a data center. You can do anything you can do with a phone in your hand. These are not emulators – they’re live, physical phones.”

The ROI benefits to services like these are fairly obvious. “You no longer have to buy all these phones, apply for the service plans … or travel,” Faraz explains, noting that in the past, developers working in one market but developing for another incurred major travel expenses.

Mob4Hire’s solution, which is called crowd source testing, leverages a community of 17,000 testers, in 105 countries on 325 operators, as well as 20 professional testing houses. It boasts a growing community of about 600 developers.

King says his company’s solution solves the developer’s problem in a unique way. It goes something like this: A developer posts an application and specifies what they need done. Then they select the handsets and testers in the countries and operators they want. An e-mail goes out to those testers and then the testers bid on that project in a kind of eBay-like bidding process.

“When the testers are accepted, money from the developers goes into escrow, and then the application goes out to the testers with a testing plan. When the testing is finished, the testers get rated and the money gets released from escrow,” King explains.

Both King and Syed could undoubtedly bicker for hours about the merits of their chosen method. Nevertheless, they both emerge with the same purpose: helping developers test their products in an increasingly fragmented market.

CARRIERS GETTING INTO THE ACT
Carriers realize that a poor app experience ultimately has an effect on their customers’ overall experience. A buggy app, no matter where it comes from, reflects poorly on their brand. That’s part of the reason why some of the bigger carriers want to get more involved in the applications market. Of course, their involvement also puts them at the center of a potentially monumental revenue stream.

AT&T developed its Apps Beta program as an avenue for developers to generate feedback from consumers before their apps go to market. Emily Soelberg, director of product marketing management for AT&T, says the Apps Beta program generates important user feedback that improves application quality and lets developers know what customers care most about.

The speed of the program is one of its great strengths. “I think what the program really does is it makes it easier for developers to get a direct line to the customer,” Soelberg says.

Marc Jacobstein, CEO of iSkoot, whose applications enable Skype to work on feature phones from Hutchinson, has put his company’s applications through testing with AT&T’s App Beta program. “The Apps Beta program was terrific because it’s organized, they’ve already got all the customers there for you and the feedback mechanisms are very quick,” Jacobstein says.

Soelberg says that participation from both the consumer and developer sides has exceeded expectations, noting some happy reactions from developers. “I’ve talked to developers who say it’s so great and gratifying when you get a customer raving about something, but they’re also happy when a customer identifies a problem.”

The Apps Beta program is relevant for those working on a number of platforms, excluding, of course, Apple. Soelberg says that AT&T identifies a number of different devices and platforms for developers to work on, including GPS devices. “We’ve tried to cover all the major ground,” she says.

But she also admits that the program has given AT&T a better idea of what exactly customers are looking for, while simultaneously helping foster a higher quality product.

Not to be outdone, Verizon Wireless recently offered a hand to developers. At its first annual developer conference, the carrier opened key network APIs to developers. It also discussed its involvement in the Joint Innovation Lab (JIL), along with China Mobile and Softbank.

Big entities like JIL give the appearance of a reduction in fragmentation, but few believe it can make a dent in a current multi-platform system that offers carriers and OEMs differentiation as well as a highly profitable form of competition.

EASING PAIN OF FRAGMENTATION
Regardless of what platform a developer is working on, if they’re running a successful app on one platform, chances are they’re seeing dollar signs as they look across the border at the potentially profitable customer base of another. The problem is that writing that second or third version can be like building an application from the ground up. It’s costly and involves a whole new round of testing.

“Fragmentation is a huge problem. Even when I think about the smartphones, it seems like there’s only about seven or eight operating systems that you can develop for, but  even inside those operating systems they get fragmented. Motorola’s coming out with some Android handsets ... if somebody wants to write an application that runs on Motorola’s Android handsets, they’re going to have to test that application on a network that supports those handsets,” King says.

“I downloaded an app for my Blackberry Bold yesterday, and even though it ran on my Bold, it was obvious that it wasn’t developed for my BlackBerry, because it was a movement game and it didn’t make use of the trackball.”

Jacobstein agrees, saying fragmentation tops his list of barriers to market as well, adding that carrier distribution contracts have to be in place before iSkoot goes ahead with developing for additional platforms. He points out that partnerships with carriers are not a luxury that a lot of smaller developers have, thus throwing them to the fates of the App Store lottery. “It’s next to impossible to get a carrier deal, and without carrier distribution, it’s very hard to be seen,” Jacobstein says.

Syed is skeptical there’s a solution to fragmentation. “It’s pretty apparent to us that consolidation is far from a reality, because there are so many vested interests. No one wants to end up in a PC kind of world where Microsoft rules.”

“Even though larger entities might be saying, ‘hey, we’re supporting this open platform, or this consortium,’ that’s just one part of what they’re doing. Every major player, while they’re supporting this or that platform, has all the other platforms being developed.”

Yet Syed tries to maintain perspective, choosing to view fragmentation as just an organic part of the industry. “How would they differentiate without fragmentation? This market hasn’t reached the kind of maturity that a PC market has. The OEMs and carriers are still struggling to grab what land they can. Granted, 10 or 15 years from now, there may be some consolidation, but as it stands right now, I can’t see it being consolidated to the level that the PC market has been.”