Mobile Testing Coverage Strategy: OS Families vs. Minor OS Versions?

As the author of the Factors coverage magazine, I am often asked about the operating system (OS) testing strategy.

Some teams would state that testing on the Latest, Latest – 1, and Latest -2 for Android is sufficient, and for iOS testing on the Latest and Latest -1 would satisfy their coverage risks. I actually say that it is not about covering the specific minor OS version, but covering sufficiently the representative OS families. In this post, i will explain my recommended strategy.

Mobile OS Versions Market Share

Late September 2017, Android platform is led by top 5 OS families (4.x, 5.x, 6.x, 7.x and latest 8.x).

In fact, Android 4.x includes 2 families – JellyBeans and KitKat, with the latest being more widely used and popular.

OEM’s are very late in updating their smartphones to the latest OS version, and when they do, they only update the most recent smartphone versions and not the legacy ones that in many cases, cover a large user base.

For Android, as we can see above, any app developer that supports the global market, cannot disregard a market share of 15.1% like the KitKat holds – KitKat is by far not a Latest -2 or even – 4 OS platform. Testing, therefore, an Android App needs to go quite far in the history of the Android OS versions to provide sufficient and risk-free testing.

iOS isn’t that simple either. This platform, especially after the recent iOS11 release, is now split into 3 major OS families that include iOS9.3.5, iOS 10.3.3 and iOS11.0. Each iOS family supports quite important devices that were left behind and did not receive the most up to date iOS update. As an example, iPhone 4S, iPad 2, iPad Mini 2 are stuck on iOS9.3.5, and iPhone 5C, iPhone 5 and iPad 4th Generation are stuck on iOS 10.3.3.

 

While the above market share still doesn’t reflect the iOS11 new market share, there is still 9% of iOS9.x users out there and there will be at least a similar number for iOS10.x once most users shift to iOS11. As in Android, here also we see 3 major OS families that need to be included in the testing strategies.

Beta Testing

Many organizations are being naive to the market innovation, and are not taking advantage of the public beta releases and developer previews coming from both Apple and Google. Android Oreo (8.0) and iOS11 were available for testing prior to their general availability release, however, many teams didn’t leverage this and are finding late in the game the defects, and in many cases, are hearing about them from their customers.

Above is the iOS11 GA bug that was reported, while below is an Android Oreo OS version specific defect that impacted many end-users.

 

Recommendations

Each customer has its unique user base, target geographies and in many cases also access to the user’s analytics.

Customers should follow the following guidelines

  • Monitor your user base and build a test lab that addresses the top user’s devices/OS permutations (use your own analytics)
  • Learn and monitor the market trends like the above so they do cover in addition to their analytics the major OS families (5-10% and above market share should be highly considered)
  • Testing on public OS beta’s is a must. Google Pixels, iOS Simulators as well as desktop web beta versions are always available for testing from day one
  • Mix device manufactures with the selected OS families to maximize the coverage (e.g. Samsung XX/Android 6.x, LG XX/Android 5.1.x, etc.)
  • Cover real environment testing to identify real-life glitches like mentioned above and as we see often in the market.

 

Happy Testing!

 

Advertisements

HTML5 Makes Mobile Testing More Encouraging For Gaming Apps

The need for mobile testing of gaming applications has long been apparent. With app stores for Android and iOS devices more or less overflowing with games of all different kinds, it stands to reason that a lot of developers are rushing the process. That is to say, there are a lot of cheaper or less refined games reaching the market in this category, and testing will reveal as much. As one article put it in an overarching write-up on the very concept of mobile testing, app store halls are now littered with thousands of one-star reviews that tell a tragic tale.

This is largely due to frantic developers hoping to be first to market with a given idea, or simply hoping to flood the market with multiple ideas, without worrying much about quality. However, it’s also because the coding languages that serve as the foundation of mobile games are varied and can be inconsistent. Certainly, a capable and creative developer can always find a way to design a beautiful and intuitive experience. But as technology improves and evolves, newer design methods will improve the overall quality of the mobile gaming market to a degree.

One thing to watch in this regard is HTML5. Hailed as a borderline revolutionary cross-platform design option for games, it has been somewhat slow to emerge in the mobile market in a big way. Casino games, in particular, are beginning to show its potential. As a popular gaming platform puts it, there are plenty of developers out there who are more than happy to indulge gamers’ craving for new experiences. In doing so, one method they’ve embraced is the use of HTML5 to put together slot games that exist at online casinos but can easily be downloaded as high-quality apps also. It’s a subtle but smoother version of a game being adapted from scratch to suit mobile devices.

We’ve seen this same process occur with some one-off arcade games as well. Bejeweled, for instance, made such a seamless transition from browser mobile devices because it was actually one of the earlier major HTML5 games. In fact, the same can be said of Angry Birds. Those are probably the two most high profile examples, though there are other mobile games that are recognized for high quality and built on HTML5 as well.

This should gradually be leading to a better selection of high quality, high-performance mobile games, and while it’s difficult to make any kind of overarching statement about mobile game testing, one article, in particular, indicated that HTML5 has had an encouraging effect. Discussing a focus on mobile, the writers declared that they weren’t even bothering to test desktops because performance had been so consistently strong. They referred to this as a positive sign for the maturity of HTML5.

By extension, that means good things for the mobile gaming market.

Complementing Cross-Browser Testing with Headless Unit Testing Solutions

Nothing new in the land of cross-browser testing. Selenium as the underlying API layer serves leading frameworks including WebDriverIO, Protractor (Angular based testing), NightWatchJS, RobotJS and many others.

For web application developers that require fast feedback capabilities post their code commit or bug resolution, there are various testing options. Some would quickly test manually on a set of local or cloud based VM’s, some will develop unit tests (qUnit etc.), but there are also very mature cross browser testing solutions that add more layers of coverage and insights in an automated and easy way.

In a recent eBook that I developed, I’m covering the 10 emerging cross-browser testing tools with a set of considerations around how to choose the right one or the right mix of them.

As can be seen in the 10 tools shown above, there is a mix of a unit as well as E2E functional testing tools mostly javascript based.

Developers who would like to include as part of their quick sanity post commit a validation of the load time it takes the site to load, can easily add this PhantomJS based test into their CI post build acceptance testing and get such visibility after each successful build – that, match the result with a benchmark and take decisions.

In a quick test that I ran on the NFL.com website, I was able to not only detect a slow load of 10sec. but I also identified a long set of errors while the page is loaded.

Another powerful capability tools like PhantomJS can offer is the ability to both capture a specific rendering of a web page by a pre-defined viewport, as well as the ability to generate a page HAR file for network traffic analysis (I am aware that it is not the newest tool, and that Goole already provides a newer version, but still this is a valuable open-source free tool that can help add coverage capabilities to any web development team).

So if as an example, the load time with errors above turns on a red light regarding that site, with 2 simple tests that BTW PhantomJS provides as their starting kit in GIT, the developer can address the above 2 use cases of HAR file generation as well as page rendering screenshot.

The result of the above snippet is the screenshot below:

The HAR file creation that is based on the following GIT code sample will result in the following (I am using the google add-on HTTP Archive Viewer for Chrome, it can be done simply with other HAR viewers as well):

Bottom line

You can download my latest eBook and learn more, but in general – leverage both unit testing powerful tools, as well as traditional E2E tests, hence they do complement each other and add their unique value – And it’s Free!

Happy Testing!