What You Need To Know When Planning Your Test Lab in 2017

As we kick-off 2017, I am thrilled to release the most updated 6th edition of the Digital Test Coverage Index report, a guide to help you decide how to build your test lab. 2016 was an exciting year in the Digital space, and as usual, Q4 market movement is sure to impact 2017 development and testing plans. And it doesn’t appear that the market is slowing down, with continued innovation expected this year. In this post, I will summarize the key insights we saw last quarter, as well as few important things that are projected for 2017 that should be applied when building your test lab.

dtci

Key Takeaways

  • Beta OS versions remain an important aspect of your test coverage strategy. With Apple releasing 5 different minor versions of iOS 10 since it’s release in September 2016, iPhone/iOS 10 beta are a “must-include in your test lab” device/OS combination. On the browser side, Chrome and Firefox beta versions are also critical test targets for sustaining the quality of your mobile web/responsive websites.
  • The Android fragmentation trend is changing, with Google putting pressure on device manufacturers to keep pace with the latest OS versions. As evidence, we already see that Android 6.x has the greatest market share as of Q42016, with roughly 27%, followed by Android Lollipop. With Google releasing its first Android Pixel devices, the market is already starting to see a boost in Android 7 Nougat adoption which is expected to grow within Q12017 to between 2-5% market share.
  • Galaxy S7 and S7 Edge were a turning point for Samsung: Over the last year, Samsung has seen a revenue slowdown due, in part, to competition from both Apple and emerging Android manufacturers OnePlus, Xiaomi, and Huawei. With the launch of Samsung S7 & S7 Edge, the company is regaining its position. We can see in this edition of the Index (and the previous one,) that Samsung is the leading brand in many countries, which should impact the test coverage plans in Brazil, India, Netherlands, UK, Germany and U.S.
  • The mobile app engagement methods are evolving, with various enterprises counting on the mobile platform to drive more revenues and attract more users. We are seeing greater adoption of external application integration either through dedicated OS-level applications like the iOS iMessage or through other solutions like the Google app shortcuts that were recently introduced as part of Android 7.1. These changes represent a challenge from a testing perspective, since there is now additional outside-of-app dependencies that the Dev and QA teams need to manage.
  • Test Lab size is expected to slightly grow YoY as the market matures:   Looking at the annual growth projection below, we see a slight growth in the need for a 10, 25 and 32 device lab, based on new the devices that are being introduced into the market faster than old devices are retired. What we see is an annual introduction of around 15 leading devices per year with an average retirement of 5-7 per year (due to decreased usage, terminated support by vendor etc.). Integrating these numbers into the 30%-80% model would bring the annual growth as demonstrated in the following graph.

annual_growth

 

2017 Trends

As this is the first Index for 2017, here are the most important market events that will impact both Dev and QA teams in the digital space, in the categories of Mobile, Web or both.

New Players

The most significant player to joins the mobile space in 2017 is Nokia. After struggling for many years to become a relevant vendor, and being unsuccessful under the Windows Phone brand, Nokia is now back in the game with a new series of Android-based devices that are supposed to be introduced during MWC 2017. A second player that is going to penetrate the mobile market is Microsoft who is supposed to introduce the first Microsoft Surface Phone during H1 2017.

Innovative Technologies

During 2017 we will definitely continue to see more IoT devices, smartwatches, and additional features coming from both Google and Apple, in the mobile, automotive and smart home markets. In addition, we might see the first foldable touch smartphone released to the market by Samsung under the name “Samsung X”. In addition, we should see a growing trend of external App interfaces in various forms such as bots, iMessages, App Shortcuts and Voice based features. The market refers to these trends as result of “App Fatigue” which is causing organizations to innovate and change the way their end-users are interacting with the apps and consuming data. From a testing perspective, this is obviously a change from existing methods and will require re-thinking and new development of test cases. In a recent blog, I addressed the above – feel free to read more about it here.

Key Device Launches to Consider for an Updated Test Lab

Most of the below can be seen in the market calendar for 2017, but the highlights are listed here as well:

  • Samsung S8/S8 Edge flagship devices from Samsung are due by February 2017 and should be the successors of the highly successful S7/S7 Edge devices
  • iPhone 8/iPhone 8 Plus together with iOS 11 launch in MID-September 2017 will mark the 10th anniversary for the Apple iPhone series. This launch is expected to be a groundbreaking one for iOS users.
  • Huawei Mate 9/Mate 9 Pro, and in general, the Huawei smartphone portfolio is continuing its global growth. 2017 should continue the growth trend both in China and India, but also as seen in this Index report in many European countries where we are already seeing devices like Huawei P8, P9, and others in use.

From a web perspective, we are not going to see any major surprises from the leading browsers like Chrome, FireFox, and Safari. However, from Microsoft Edge browser, we expect a significant market share uptick as more and more users adopt Windows 10 and abandon legacy Windows OS machines.

cal2017

 

In the Index report, you may find all the information necessary to better plan for 2017, as well as market calendars for both mobile and the web, plus a rich collection of insights and takeaways. DOWNLOAD HERE.

Happy Testing in 2017!

Joe Colantonio’s Test Talk: Mobile Testing Coverage Optimization

How does a company nowadays put together a comprehensive test strategy for delivering high-quality experiences for their applications on any device? I think this is the question I get asked most frequently and it is the biggest challenge in today’s market, how to tackle mobile testing and responsive web testing. The solution can be the difference between an app rated 1 star or an app rated 5 stars.

Play Podcast

I had a lot of fun talking to Joe Colantonio from Test Talks about how to create a successful app starting with my Digital Test Coverage Optimizer. Listen to the full talk to hear my ideas on moving from manual testing to automation, tracking the mobile market, the difference between testing in simulators and emulators versus real devices and more.

https://joecolantonio.com/testtalks/110-mobile-testing-coverage-optimization-eran-kinsbruner/

 

JC

Responsive Web: The Importance of Getting Test Coverage Right

When building your test lab as part of a RWD site test plan, it is important to strategically define the right mobile devices and desktop browsers which will be your target for your manual and automated testing.

For mobile device testing you can leverage your own analytics together with market data to complement your coverage and be future ready, or leverage reports such the Digital Test Coverage Index Report.

For web testing you should also look into your web traffic analytics or based on your target markets understand which are the top desktop browsers and OS versions on which you should test against – alternatively, you can also use the digital test coverage index report referenced above.

Related Post: Set Your Digital Test Lab with Mobile and Web Calendars

Coverage is a cross organizational priority where both business, IT, Dev and QA ought to be consistently aligned. You can see a recommended web lab configuration for Q1 2016 below which is taken from the above mentioned Index – Note the inclusion of Beta browser versions in the recommended mix due to the nature silent updates of these versions deployment on end-user browsers.

WCReport
For ongoing RWD projects  – once defining the mobile and web test coverage using the above guidelines, the next steps are of course to try and achieve parallel side by side testing for high efficiency, as well as keep the lab up to date by revising the coverage once a quarter and assure that both the analytics as well as the market trends still matches your existing configuration.

As a best practice and recommendation, please review the below mobile device coverage model which is built out of the 3 layers of Essential, Enhanced and Extended where each of these layers includes a mix of device types such as legacy, new, market leaders and reference devices (like Nexus devices).

MobileCoverageLayers

To learn more, check out our new Responsive Web Testing Guide.

responsive web testing strategy

Tests to Include Within Automation Suite

When developing a mobile or desktop test automation plan organization often struggle with the right scope and coverage for the project.

In previous post, i covered the test coverage recommendations in a mobile project and now, i would like to also expand on the topic of which tests to automate.

Achieving release agility with high quality is fully dependent today more than ever on continuous testing which is gained through proper test automation, however automating every test scenario is not feasible and not necessary to meet this goal.

In the below table  we can see some very practical examples of test cases with various parameters with a Y/N recommendation whether to automate or no.

As shown below, and as a rule for both Mobile, Web and other projects the key tests by definition which should be added to an automation suite (from ROI perspective and TTM) are the ones who are:

  • Required to be executed against various data sets
  • Tests which ought to run against multiple environments (Devices, Browsers, Locations)
  • Complex test scenario’s (these are time consuming and error prone when done manually)
  • Tedious and repetitive test cases are a must to automate
  • Tests which are dependent on various aspects (can be other tests, other environments etc.)

Picture1

Bottom line: Automation is key in today’s digital world, but doing it right and wisely can shorten time to market, redundant resources and a lot of wasted R&D time chasing unimportant defects coming from irrelevant tests

Happy Testing!

 

 

Planning Mobile Test Coverage

In any conversation i participate the topic of test coverage comes up – and it is indeed a great challenge for business, practitioners whether they are developers or testers (Agile, DevOps, Waterfall etc.)

Before we understand the how, let’s understand the objectives and coverage definition.

Coverage Aspects:

  1. Device coverage
  2. Market coverage
  3. Test case/use case coverage
  4. Environment conditions coverage

When we mention device coverage, we should try and include some relevant factor, not just the DUT (Device under test), because it is simply not enough.

Device Coverage

Proper device coverage shall include few important properties and the more permutations you’re going to include in your test lab the higher coverage you will reach. Some of the MUST properties which i would recommend to have as part of the mix are:

  • Screen size & Resolution
  • PPI (Pixel per inch)
  • OSV (OS Version)

To that mix you need to relate to leading market devices and also to legacy devices which are still popular by many users in various geo’s (e.g. Samsung Galaxy S3, iPad 2) in order to obtain both legacy OS and new OS coverage + the above device characteristics.

Market Coverage

Let’s understand Market coverage – This term relates to a combination of data sources to which some teams may have access to, and some won’t. Such coverage term will typically be a combination of leading market statistics and organizational web traffic or monitoring reports which would highlight information around most usage coming from which platform, browsers etc. When combining both Market and Org. data teams can best match their target audience and test against what’s right for their customers from current top usage perspective and in addition get market coverage around new and emerging devices/OSV to allow them to stay on top of market trends.

Another important aspect around coverage is of course the test cases themselves.

Test Case Coverage

Determining the right test cases to execute against each platform and in each test iteration throughout the SDLC (software development life cycle) is a crucial AGILE enabler and an efficiency driver. When there is a robust automation foundation within the organization teams can take advantage of this system and sometime “fail” by overloading it with either redundant test cases or inefficient test cases which does not add the right value. The key to increase test case coverage is to combine Manual & Automation testing (automate of course as much as possible) but only include the cross platform robust test automation and unit tests which are repeatable, valuable for a quick feedback loop between Dev and QA and leave the platform specific tests, corner cases, and such either to be done manually or as a separate JOB/cycle to assure flawless CI/Automation process.

Even with the above in mind, keep in mind that automation without ongoing maintenance, review of the test code will eventually fail especially around mobile due to constant platform specif changes, new features added or new unexpected popups which may block automation tests from running end to end.

Test Environment

Last, for a digital test coverage the user experience and the environment in which the user operates in is everything. Not covering the right environment would eventually waste testing and dev time since these efforts will be done against the wrong or “happy path only” environment. A real mobile environment takes into account the following:

  • Network conditions (2G, 3G, Wifi)
  • Background applications running as a “noise background” – consuming resources, taking over GPS resources or camera
  • incoming calls/popups
  • different screen orientation changes while app is in the foreground
  • Location of the app 
  • Locale & language

When taking all of the above under consideration, organizations can really build a test lab which provides sufficient coverage for their product and can easily adjust the lab based on market and product dynamics.

Happy Testing!

Blog Series (2) Digital Test Coverage Assuring the Right Mobile/Web Testing Mix

It’s an exciting time to be a digital company. Your customers are engaging with your products on various screens, moving between desktop web browsers to apps on mobile devices. But in the effort to guarantee quality web and mobile experiences, organizations are struggling to find the right testing mix.

It’s true that mobile is far more complex and fragmented than the web, but with so many web browser/OS permutations out there (i.e. Chrome OS 47 running on Windows 7, 8.1, 10, Mac Yosemite, etc.) precise testing becomes a challenge.

To help DevTest teams test more precisely, Perfecto recently published the “Digital Test Coverage Index – Edition 3”, a quarterly report that provides a prescriptive way to build a digital test lab that covers 30%, 50%, or 80% of mobile device and web browser markets in various geographies. The report — intended for organizations just starting their digital journey or trying to move to the next stage — is based on market share data and analysis of enterprise customer usage in Perfecto’s cloud testing lab.

2015Q3Index_Shared_Image_4

Using the above 30%-50%-80% coverage model featured in the Index report, teams can more accurately define their desired testing parameters and allocate the recommended devices and virtual machines running the relevant desktop browsers. Teams that are developing a responsive web application (RWD) can refer to the Index and then test the app in their lab on the recommended smartphones and tablets alongside the recommended desktop browser/OS permutations.

On the subject of web browser/OS mixes: According to our latest Index, of the top 30% of desktop browsers in the U.S. market, Chrome OS 46 (version 47 was just released and is well-adopted already) is by far the leading browser on Windows 7, followed by FireFox OS version 42 on Windows 7, and Safari OS 9 on Mac OS El Capitan. The Index report includes the complete 30%-50%-80% matrix for web/OS and mobile device/OS combinations.

411d4a3d-b435-46d4-89b6-f783935a82e3-original

It’s also worth noting that in the browser testing landscape, the Windows 10 platform is gaining momentum and will soon become the second most popular desktop OS in most of the geographies, according to market share numbers.

It will take more than looking at a list of smartphones and web browsers to ensure full digital test coverage for native and hybrid mobile apps, mobile web browsers and RWD. So organizations need to combine their existing customer analytics with a regularly updated test coverage index that reflects market adoption rates in various geographies. Another important metric to monitor is the status of legacy platforms that are still relevant enough to test against. For example, the Samsung Galaxy S3 is a leading legacy smartphone in most markets in the same way that we still see many Windows 7 machines even though Windows 8 and Windows 10 are widely available.

For more details on how to test for the full digital experience, download the free Digital Test Coverage Index.