Model-Based Testing and Test Impact Analysis

In my previous blogs and over the years, I already stated how complicated, demanding and challenging is the mobile space, therefore it seems obvious that there needs to be a structured method of building test automation and meeting test coverage goals for mobile apps.

While there are various tools and techniques, in this blog I would like to focus on a methodology that has been around for a while but was never adopted in a serious and scalable way by organizations due to the fact that it is extremely hard to accomplish, there are no sufficient tools out there that support it when it comes to non-proprietary open-source tools and more.

First things first, let’s define what is a Model-Based testing

Model-based testing (MBT) is an application for designing and optionally executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment.

mbt

Fig 1: MBT workflow example. Source: Wikipedia

In the context of mobile, if we think about an end-user workflow with an application it will usually start with a Login to the app, performing an action, going back, performing a secondary action and often even a 3rd action based on the previous output of the 2nd. Complex Ha

The importance of modeling a mobile application serves few business goals:

  1. App release velocity
  2. App testing coverage (use cases)
  3. App test automation coverage (%)
  4. Overall app quality
  5. Cross-team synchronization (Dev, Test, Business)

 

As already covered in an old blog  I wrote, mobile apps would behave differently, and support different functionality based on the platform they are running (E.g., not every iOS device support both Touch ID and/or 3D Touch gesture). Therefore, being able to not only model the app and generate the right test cases but also to match these tests across the different platforms can be a key to achieving many of the 1-5 goals above.

In the market, today there are various commercial tools that can assist in MBT like CA Agile Requirement Designer, Tricentis Tosca, and others.

Looking at an example provided by one of the commercial vendors in the market (Tricentis), it can show a common workflow around MBT. A team aims to test an application; therefore, they would scan it using the MBT tool to “learn” its use cases, capabilities, and other artifacts so they can stack these into a common repository that can serve the team to build test automation.

tosca

Fig 2: Tricentis Tosca MBTA tool

In Fig 2., Tricentis examines a web page to learn its entire options, objects, and other data related items. Once the app is scanned, it can be easily converted into a flow diagram that can serve as the basis for test automation scenarios.

With the above goals in mind, it is important to understand that having an MBT tool that serves the automation team is a good thing, but only if it increases the team efficiency, its release velocity, and the overall test automation coverage. If the output of such tool would be a long list of test cases that either does not cover the most important user flows, or it includes many duplicates than this wouldn’t serve the purpose of MBT but rather will delay existing processes, and add unnecessary work to teams that are already under pressure.

In addition to the above commercial tools, there is an older but free tool that allows Android MBT with robotium called MobiGuitar. This tool not just offers MBT capabilities but also code coverage driven by the generated test scripts.

A best practice in that regards would be to probably use an MBT tool that can generate all the application related artifacts that include the app object repository, the full set of use cases, and allow all of that to be exported to leading open-source test automation frameworks like Selenium, Appium, and others.

Mobile Specific MBT – Best Practices and Examples

Drilling down into a workflow that CA would recommend around MBT would look as follows – In reality, the below is easier said than done for a Mobile App compared to Web and Desktop:

  1. The business analysts will create the story using tools like CA Agile Requirement Designer or such (see below more examples)
  2. The story is then passed to an ALM tool (e.g.: CA Agile Central [formerly Rally], Jira, etc.) for project tracking
  3. Teams use the MBT tools to collaborate
    1. The automation engineer adds the automation code snippets to the nodes where needed or adds additional nodes for automation.
    2. The programmer updates the model for technical specs or more technical details.
    3. The Test Data engineer assigns test data to the model
  4. Changes to the story are synchronized with the ALM Tool
  5. Test cases are synchronized with the ALM Tool
  6. The programmer completes coding
  7. The code is promoted from Dev to QA
  8. Testing begins
    1. The tester uses the test cases with test data from MBT tools for manual test case execution
    2. The automation scripts with test data are executed for functional and regression testing

To learn more about efficient MBT solutions, practices please refer to these sources:

Advertisements

Porting mobile apps – few guidelines and insights

Hi

When we talk about mobile world complexities, it is important to understand the meaning of Porting.

We know that the mobile platform is dynamic, often changes and being influenced by many aspects (Calls, SMS’s and other interrupts, but also from diversity of mobile OS’s, new devices entering the market and more)

Porting of mobile apps, means – To take an existing working mobile application (Game, utility, web service or other) which was proved to work fine on few “lead”/”gold” devices which were (hopefully) well-defined by the project manager, and make a port of the application to many other handsets, tablets and so on.

You can imagine, that any generic core bug in one of the “lead” devices will obviously drill down to the ported handsets, so at first it is important to pick the right “lead” device which is new enough in the market, solid, and represent a “family” of existing devices (e.g. Samsung Galaxy SII for the Android OS).

Second thing to keep in mind, is that the time to develop and release a product (Mobile related) is ~3-6 months so the device needs to still be “relevant” in the market and be considered a “lead” in the market at the time of release.

Once we are certain that we picked the right lead devices and we have a Beta version with all features implemented and functioning, we than enter the porting phase in which we start to introduce new devices (hopefully from the same families of the lead devices which we ported) and install the application on them to perform sanity.

If the application was developed and designed  “for portability” – we should be able to produce various version per new handsets which are built from the same core base with the distinguish of icons, resources and minor differences – in that case fixing bugs should also be easy.

If the design was not right than we can enter a “nightmare” of fixing bugs backwards in the core source, and maintaining many old and new devices for a long time.

The right model is to have a solid and generic core source base from which the release manager generates using build targets the right handsets by using property files which matches the right device “family” and profile.

Porting bugs can vary between handsets and can be caused by many reasons which are not always related to our application (e.g – an application which uses GPS or Camera and these functions are buggy on the device may impact the application stability – this is something which porting should handle, issues related to screen resolution, rotating from landscape to portrait and more).

In our days when we have the Android OS, iOS and Windows Phone and we wish to develop similar application to these 3 or more platforms, porting becomes even more complex – For that we can consider the above insights, and also integrate tools such as Code Name One, PhoneGap or other tools which allows to use one code base and deploy it to various mobile platforms and languages.

In future blogs i will try and refer to the method of developing a master test plan for the lead devices and a subset/sanity test plan for the ported devices which are derived from the lead ones.

For questions and more details, you may contact me as usual: eran.kinsbrunner@gmail.com

Regards,

Eran Kinsbruner

Windows Phone platform – Useful information

All,

Many people were surprised a while back when Microsoft and Nokia joined forces to build the future platform for Microsoft targeted for Windows-based smart phone and Tablets.

Time passed, and it seems like a lot is going on in the Windows Phone platform.
Visual Studio Express with Windows Phone Tools was built, works great on Windows 7 OS.
Express Blend, Silver Light and many other tools became available for Windows phone developers, as well as XNA framework integration to the mobile platform.

In the latest MWC 2012 in Barcelona, the new Nokia Lumia devices were introduced (Lumia 900, 800, etc.) and few weeks later started to be sold.

Current OS version which runs on Windows phones is windows phone 7.5, however there are already news about the launch of the poweful Windows Phone 8 OS – which will be named Apollo, and will be released this coming Fall

http://www.engadget.com/2012/06/20/microsoft-introduces-windows-phone-8/

The bad news are that thew new OS will not be backward compatible with the current existing WP7.x devices which is disappointing and surprising (If this indeed will be the case).

For more reading on the windows phone 8 features and more – visit the Microsoft blog:

http://windowsteamblog.com/windows_phone/b/windowsphone/archive/2012/06/20/announcing-windows-phone-8.aspx

Basically, the additional enhancement for windows phone will be these:

  • Multi-core processor support
  • Bigger, sharper screens: Windows Phone 8 supports two new screen resolutions—1280×768 and 1280×720, opening the door to amazing new handsets with high-definition 720p displays.
  • More flexible storage
  • NFC wireless sharing
  • Internet Explorer 10: The next version of Windows Phone comes with the same web browsing engine that’s headed for Window 8 PCs and tablets.
  • Wallet: Windows Phone 8’s new digital Wallet feature does two great things. It can keep debit and credit cards, coupons, boarding passes, and other important info right at your fingertips.
  • Better maps and directions
  • Cooler apps and games: Basing Windows Phone 8 on the Windows core will unleash a new wave of amazing apps and especially games, for reasons I’ll touch on in a moment.

Saying all of that, we start to see more mobile OEM’s such as HTC, Huawei and Samsung which also declare about their plans to release WP based devices.

In a poll which asked questions about the new WP8 release, and its compare to Android and iOS, people answered as follows:

Are you impressed with the new Windows Phone 8?

  • Yes, it brings some new cool features (35%, 166 Votes)
     
  • It’s OK, but nothing to write home about (23%, 110 Votes)
     
  • It’s not even close to Android ICS (20%, 95 Votes)
     
  • Yes, it leaves Android in the dust (15%, 71 Votes)
     
  • Underwhelming (7%, 39 Votes)
     

Total Voters: 481

This is how the WP8 start screen shall look like (More or less):

HTC press release from yesterday about their plans to release 3 new HTC devices running WP8 (“Rio”, “Zenith” and “Accord”):

http://www.wired.com/gadgetlab/2012/06/report-three-htc-windows-phone-8-devices-in-the-works/

As mentioned above, Huawei also announced their plan to release a new WP8 device by the end of this year:

http://www.theverge.com/2012/6/20/3104859/huawei-ascend-windows-phone-8-announcement-wp8?utm_source=dlvr.it&utm_medium=twitter

In case you wish to see live videos about the WP platform, features and more, visit the Youtube channel for WP:

http://www.youtube.com/playlist?list=PL0C88F4DC0C435D09

Since this is a Testing Blog, i would like to share an important link which holds information about tools which Microsoft provides for the testers (Certification tools, and more, called MPR – Microsoft Platform Ready tools – soon to be more relevant for the mobile platforms):

http://www.microsoftplatformready.com/dashboard.aspx

http://www.microsoft.com/en-us/download/confirmation.aspx?id=29945

Stay tuned for further news and updates.

Regards,

Eran Kinsbruner