The Keys To Mobile Test Automation That Works!

In the world of mobile app testing, a developer’s claim “Well it works on my device” is not acceptable. Mobile is by far more fragmented than Desktop/Web platforms due to the variety of device types, OS versions, and the environment on which it operates (network impact, roaming, battery stretch etc.).

It is not sufficient to test on one or even a few devices, but instead test apps on several devices that adequately cover your end-users’ device pool. How can you keep up with testing apps on so many devices, you ask? Easy. Build a working mobile test automation strategy that runs the same script across all of your devices, making your Dev/QA teams confident that they are delivering the best app to end-users.

From the experience Perfecto Mobile gathers everyday, we came up with the must-haves for building a continuous, unattended mobile test automation strategy that provides immediate feedback to its Dev/QA teams.

Relevant Device Selection

It is one of the toughest dilemmas in today’s market to identify the right device X OS mix on which teams should develop and test their mobile app against. Base your selection on the right criterias such as:

    1. Target market and competition/End-Users –> On which devices and OS versions your end-users actually use the most (based on organizational analytic data)
    2. Generic market statistics –> Top sold devices (smartphones and Tablets) in the relevant regions. Such data can be retrieved from analyst research such as IDC, Flurry, ComScore and others.
    3. Change rate and market dynamics –> As we all see, the mobile market is in constant change and your teams need to “listen” to the trends and changes- adopt them and change the strategy accordingly

Unattended mobile app test automation which supports continuous integration and fast feedback:

    1. Mandates mobile OS system level control – If the script cannot initiate events outside of the AUT (App under test) it cannot really test the app as the end-user will use it
    2. Incoming alerts and error handling are also mandating system level control out side of the AUT
    3. Hybrid objects support – automation framework should support both OS level objects (DOM for web and hybrid apps) and Visual objects for full use case support
    4. Device logs, rich media reports (Video and Visual) and vitals (Memory status, Battery etc.) are critical evidence to Dev/DevOps teams as feedback to act upon

Device Elasticity

As your mobile app project grows/shrinks based on your SDLC phase, your automation solution needs to be flexible enough to accommodate necessary changes and provide the needed support to your team (even more critical in cases of distributed teams).

Integration is Key for Cross Team Collaboration

Whether your Agile/Dev/QA teams are using ALM tools (HP, Microsoft, etc.) or OSS (Selenium, Robotium, JUnit) tools, your automation framework solution should be able to connect to these tools and bridge any gaps in the daily continuous work flow. Use what your team knows – this is the best way to release mobile apps faster without compromising quality.

Real Devices

It is imperative to understand that testing on something other than real devices in your critical automation scripts and as your mobile app evolves is a huge risk. Only real devices running formal OEM OS flavors against real networks reveal an app’s true functionality in the same way your end-user will experience it.

To conclude, see below (Fig 1), a check list of mobile testing pillars which need to be part of your Mobile App Testing Plan.

Fig 1: The pillars of mobile app testing

Good Luck!

Advertisements

Leveraging Real Network Conditions As Part Of The Continuous Quality Processes

As the digital experience becomes key for organizations to offer end-users and a high user experience based on mobile moments, it is important to understand the environment in which mobile apps are “living” in.

 

We have taken to calling a mobile app’s environment as a wind tunnel – many different conditions and “states” occur, causing mobile apps (Web and Native) running on various screens (Smartphones, Tablets, Wearables) to behave differently with functionality and performance POV.

 

In order to extend a continuous quality process and enrich its coverage, mobile app quality ought to include real network conditions as part of the test matrix as early as possible in the mobile app development lifecycle.

 

To achieve coverage of real network conditions developers and network engineers should add the following scenarios into their test plan:

  • Network virtualization – What happens to your mobile app in various network conditions, through the use of various profiles which can simulate poor, average, good 2G/3G/4G/Wifi network connections in various geographical locations (see e.g. Fig1 below). Understanding the behavior of your mobile apps in degraded network conditions early is a key to drive your mobile app release velocity as well as increase your app quality.

Screen Shot 2015-01-22 at 20.21.07

  • Degraded application performance due to other processes, like apps running in the background, can be achieved through mobile device vitals collection, logs analysis and other examinations (e.g. many apps today leverage 3rd party apps like social media apps etc. which can impact the app under test performance)
  • Degraded server performance due to load (light load, heavy load) – how do your mobile app performs from response time and availability in peak times, in various network conditions?
  • Interoperability and events impact on mobile app functionality – how is your mobile app going to behave and respond when an incoming call, pop-ups, and other events which comes in while your mobile app is running?
    • To measure this impact mobile app vendors need to have full access to the mobile system (e.g. iPhone SpringBoard) so they can trigger the events, control them and debug them properly.

Screen Shot 2015-01-27 at 14.02.24

Figure 2: Perfecto Mobile Wind Tunnel various capabilities 

Some examples for more mobile app testing interruptions

  • Incoming Call
  • Text message
  • Other app notifications
  • Storage/RAM low
  • Battery low
  • Battery dead
  • No storage
  • Airplane mode
  • Intermittent connectivity
  • Home screen jump
  • Sleep mode

device-2012-07-25-133714

 

To read more about the continuous quality approach and the continuous quality lab which can enable such capabilities, visit the Perfecto Mobile web site: http://www.perfectomobile.com