Introducing Reporting Test Driven Development (RTDD)

In the era of “[.. ] Driven Development” trends like BDD, TDD, and ATDD it is also important to realize the end goal of testing, and that’s the quality analysis phase.

In many of my engagements with customers, and also from my personal practitioner experience I constantly hear the following pains:

  1. Test executions are not contextually broken, therefore are too long to analyze and triage
  2. Planning test executions based on trends, experience, and insights is a challenge – e.g. which tests are finding more bugs than the other?
  3. Dealing with flaky tests is an ongoing pain especially around mobile apps and platforms
  4. On-Demand quality dashboards that reflect the app quality per CI Job, Per app build, Per functionality tested area etc.

 

Introducing Reporting Test Driven Development (RTDD)

As an aim to address the above pains, that I’m sure are not the only related ones, I came to an understanding, that if Agile/DevOps teams start thinking about their test authoring and implementation with the end-in-mind (that is the Test Reports) they can collect the value at the end of each test cycle as well as prior during the test planning phase.

When teams can leverage a test design pattern that assigns their tests with custom Contextual Tags that wrap an entire test execution or a single test scenario with annotations like “Regression“, “Login“, “Search” and so forth – suddenly the test suites are better structured, easily maintained and can be either included/excluded and filtered through at the end of execution.

In addition, when the entire suite is customized by tags and annotations, management teams can easily retrieve on-demand quality dashboard and be up to date with any given software iteration.

Finally, developers that get the defect reports post executions, can easily filter and drill down into the root cause in an easier and more efficient manner.

If you think about the above, the use of annotations as a method to manage test execution and filter them is not a new concept.

TestNG Annotations with Selenium Example (source: Guru99)

As seen above, there are supported ways to tag specific tests by their priority, it is just a matter of thinking about such tags from the beginning.

Doing reverse engineering to a large test suite is painful, hard to justify and most often too late since the product by then is already out there and the teams are left to struggle with the 4 mentioned consequences from above.

RTDD is all about putting structure, governance, and advanced capabilities into your test automation factory.

If we examine the following table that divides various tags by 3 levels, it can serve as 1 reference that can be immediately used either through built-in tagging and annotation coming from TestNG or other reporting solutions.

As can be seen in the above table, think about an existing test suite that you recently developed. Now, think about the exact test suite that is tag-based according to the above 3 categories:

  1. Execution level tags
    1. This tag can encapsulate the entire build or CI JOB-related testing activities, or it can differentiate the tests by the test framework in which you developed the scripts. That’s the highest classification level of tags that you would use.
  2. Test suite level tags
    1. This is where you start breaking your test factory according to more specific identifiers like your mobile environment, the high-level functionality under test etc.
  3. Logical test level tags
    1. These are the most granular test tags identifiers that you would want to define per each of your test logical steps to make it easy to filter upon, triage failures, and plan ongoing regressions based on code changes.

As a reference implementation for an RTDD solution in addition to the basic TestNG implementation that can be very powerful if being used correctly with its listeners, pre-defined tags and more,  I would like to refer you to an open-source reporting SDK that enables you to do exactly what is mentioned in the above post.

When using such SDK with your mobile or responsive web test suites, you achieve both, the dashboards as seen below as well as a fast defect resolution that drills down by both Test case and Platform under test

Code Sample: Using Geico RWD Site with Reporting TDD SDK [Source: My Personal GIT)

 

Digital Dashboard Example With Predefined ContextTags (source: Perfecto)

 

Bottom Line

What I have documented above, should allow both managers, test automation engineers, and developers of UI/Unit and other CI related tests to extend either a legacy test report, a testNG report or other – to a more customizable test report that, as I’ve demonstrated above, can allow them to achieve the following outcomes:

  • Better structured test scenarios and test suites
  • Use tagging from early test authoring as a method for faster triaging and prioritizing fixes
  • Shift tag based tests into planned test activities (CI, Regression, Specific functional area testing, etc.)
  • Easily filter big test data and drill down into specific failures per test, per platform, per test result or through groups.
  • Eliminate flaky tests through high-quality visibility into failures

The result of the above is a facilitation of a methodological-based RTDD workflow that can be maintained much easier than before.

Happy Testing (as always)!

Advertisements

Model-Based Testing and Test Impact Analysis

In my previous blogs and over the years, I already stated how complicated, demanding and challenging is the mobile space, therefore it seems obvious that there needs to be a structured method of building test automation and meeting test coverage goals for mobile apps.

While there are various tools and techniques, in this blog I would like to focus on a methodology that has been around for a while but was never adopted in a serious and scalable way by organizations due to the fact that it is extremely hard to accomplish, there are no sufficient tools out there that support it when it comes to non-proprietary open-source tools and more.

First things first, let’s define what is a Model-Based testing

Model-based testing (MBT) is an application for designing and optionally executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a System Under Test (SUT), or to represent testing strategies and a test environment.

mbt

Fig 1: MBT workflow example. Source: Wikipedia

In the context of mobile, if we think about an end-user workflow with an application it will usually start with a Login to the app, performing an action, going back, performing a secondary action and often even a 3rd action based on the previous output of the 2nd. Complex Ha

The importance of modeling a mobile application serves few business goals:

  1. App release velocity
  2. App testing coverage (use cases)
  3. App test automation coverage (%)
  4. Overall app quality
  5. Cross-team synchronization (Dev, Test, Business)

 

As already covered in an old blog  I wrote, mobile apps would behave differently, and support different functionality based on the platform they are running (E.g., not every iOS device support both Touch ID and/or 3D Touch gesture). Therefore, being able to not only model the app and generate the right test cases but also to match these tests across the different platforms can be a key to achieving many of the 1-5 goals above.

In the market, today there are various commercial tools that can assist in MBT like CA Agile Requirement Designer, Tricentis Tosca, and others.

Looking at an example provided by one of the commercial vendors in the market (Tricentis), it can show a common workflow around MBT. A team aims to test an application; therefore, they would scan it using the MBT tool to “learn” its use cases, capabilities, and other artifacts so they can stack these into a common repository that can serve the team to build test automation.

tosca

Fig 2: Tricentis Tosca MBTA tool

In Fig 2., Tricentis examines a web page to learn its entire options, objects, and other data related items. Once the app is scanned, it can be easily converted into a flow diagram that can serve as the basis for test automation scenarios.

With the above goals in mind, it is important to understand that having an MBT tool that serves the automation team is a good thing, but only if it increases the team efficiency, its release velocity, and the overall test automation coverage. If the output of such tool would be a long list of test cases that either does not cover the most important user flows, or it includes many duplicates than this wouldn’t serve the purpose of MBT but rather will delay existing processes, and add unnecessary work to teams that are already under pressure.

In addition to the above commercial tools, there is an older but free tool that allows Android MBT with robotium called MobiGuitar. This tool not just offers MBT capabilities but also code coverage driven by the generated test scripts.

A best practice in that regards would be to probably use an MBT tool that can generate all the application related artifacts that include the app object repository, the full set of use cases, and allow all of that to be exported to leading open-source test automation frameworks like Selenium, Appium, and others.

Mobile Specific MBT – Best Practices and Examples

Drilling down into a workflow that CA would recommend around MBT would look as follows – In reality, the below is easier said than done for a Mobile App compared to Web and Desktop:

  1. The business analysts will create the story using tools like CA Agile Requirement Designer or such (see below more examples)
  2. The story is then passed to an ALM tool (e.g.: CA Agile Central [formerly Rally], Jira, etc.) for project tracking
  3. Teams use the MBT tools to collaborate
    1. The automation engineer adds the automation code snippets to the nodes where needed or adds additional nodes for automation.
    2. The programmer updates the model for technical specs or more technical details.
    3. The Test Data engineer assigns test data to the model
  4. Changes to the story are synchronized with the ALM Tool
  5. Test cases are synchronized with the ALM Tool
  6. The programmer completes coding
  7. The code is promoted from Dev to QA
  8. Testing begins
    1. The tester uses the test cases with test data from MBT tools for manual test case execution
    2. The automation scripts with test data are executed for functional and regression testing

To learn more about efficient MBT solutions, practices please refer to these sources:

Latest news from the mobile world

Hi,

In the last couple of days and with the new iPhone5 announcement i wanted to gather the main highlights in one place 🙂

  • iPhone 5 (http://www.apple.com/iphone/) which was announced shall be available next week together with a possible launch of the new iOS6!
  • Details around the app store applications –> Total of 700,000 apps available in AppStore, while ~250,000 also ported to iPad
  • battle in the mobile world continues, and few great smartphones put themselves as leaders and competitors to the new iPhone 5:

** HTC One X – http://www.htc.com/www/smartphones/htc-one-x/ (4.7” Screen)

** Samsung Galaxy SIII – http://www.samsung.com/global/galaxys3/ (4.8” Screen)

** LG Optimus G – http://reviews.cnet.com/smartphones/lg-optimus-g-unlocked/4505-6452_7-35427825.html (4.7” Screen)

** Motorola Razr Maxx HD – http://www.engadget.com/2012/09/05/motorola-droid-razr-hd-maxx-hands-on-bigger-battery-beautiful/ (4.7” Screen)

** Sony Xperia S – http://www.gsmarena.com/sony_xperia_s-4369.php (4.3” Screen)

** Nokia Lumia 920 – http://www.nokia.com/global/products/phone/lumia920/ (4.5” Screen)

Eran Kinsbruner

Reference for Google Jelly Beans 4.1 Google now

Hi,

In this quick post, i just want to atract your attention to one of the coolest features which is coming to the new introduced version of Google – JeakkyBeans 4.1 and called ‘Google Now’ – Voice command controller and more (Provides guide based on your location, and additional information based on your social activity)

At first, check out this short video for overview of the feature:

You may also read more about the first JellyBeans tablet which was also introduced lately and will be the first to come with this new OS – Nexus 7 by ASUS

http://www.wired.com/gadgetlab/2012/06/hands-on-google-asus-nexus-7-tablet-impresses/

Regards,
Eran Kinsbruner

Porting mobile apps – few guidelines and insights

Hi

When we talk about mobile world complexities, it is important to understand the meaning of Porting.

We know that the mobile platform is dynamic, often changes and being influenced by many aspects (Calls, SMS’s and other interrupts, but also from diversity of mobile OS’s, new devices entering the market and more)

Porting of mobile apps, means – To take an existing working mobile application (Game, utility, web service or other) which was proved to work fine on few “lead”/”gold” devices which were (hopefully) well-defined by the project manager, and make a port of the application to many other handsets, tablets and so on.

You can imagine, that any generic core bug in one of the “lead” devices will obviously drill down to the ported handsets, so at first it is important to pick the right “lead” device which is new enough in the market, solid, and represent a “family” of existing devices (e.g. Samsung Galaxy SII for the Android OS).

Second thing to keep in mind, is that the time to develop and release a product (Mobile related) is ~3-6 months so the device needs to still be “relevant” in the market and be considered a “lead” in the market at the time of release.

Once we are certain that we picked the right lead devices and we have a Beta version with all features implemented and functioning, we than enter the porting phase in which we start to introduce new devices (hopefully from the same families of the lead devices which we ported) and install the application on them to perform sanity.

If the application was developed and designed  “for portability” – we should be able to produce various version per new handsets which are built from the same core base with the distinguish of icons, resources and minor differences – in that case fixing bugs should also be easy.

If the design was not right than we can enter a “nightmare” of fixing bugs backwards in the core source, and maintaining many old and new devices for a long time.

The right model is to have a solid and generic core source base from which the release manager generates using build targets the right handsets by using property files which matches the right device “family” and profile.

Porting bugs can vary between handsets and can be caused by many reasons which are not always related to our application (e.g – an application which uses GPS or Camera and these functions are buggy on the device may impact the application stability – this is something which porting should handle, issues related to screen resolution, rotating from landscape to portrait and more).

In our days when we have the Android OS, iOS and Windows Phone and we wish to develop similar application to these 3 or more platforms, porting becomes even more complex – For that we can consider the above insights, and also integrate tools such as Code Name One, PhoneGap or other tools which allows to use one code base and deploy it to various mobile platforms and languages.

In future blogs i will try and refer to the method of developing a master test plan for the lead devices and a subset/sanity test plan for the ported devices which are derived from the lead ones.

For questions and more details, you may contact me as usual: eran.kinsbrunner@gmail.com

Regards,

Eran Kinsbruner

Windows Phone platform – Useful information

All,

Many people were surprised a while back when Microsoft and Nokia joined forces to build the future platform for Microsoft targeted for Windows-based smart phone and Tablets.

Time passed, and it seems like a lot is going on in the Windows Phone platform.
Visual Studio Express with Windows Phone Tools was built, works great on Windows 7 OS.
Express Blend, Silver Light and many other tools became available for Windows phone developers, as well as XNA framework integration to the mobile platform.

In the latest MWC 2012 in Barcelona, the new Nokia Lumia devices were introduced (Lumia 900, 800, etc.) and few weeks later started to be sold.

Current OS version which runs on Windows phones is windows phone 7.5, however there are already news about the launch of the poweful Windows Phone 8 OS – which will be named Apollo, and will be released this coming Fall

http://www.engadget.com/2012/06/20/microsoft-introduces-windows-phone-8/

The bad news are that thew new OS will not be backward compatible with the current existing WP7.x devices which is disappointing and surprising (If this indeed will be the case).

For more reading on the windows phone 8 features and more – visit the Microsoft blog:

http://windowsteamblog.com/windows_phone/b/windowsphone/archive/2012/06/20/announcing-windows-phone-8.aspx

Basically, the additional enhancement for windows phone will be these:

  • Multi-core processor support
  • Bigger, sharper screens: Windows Phone 8 supports two new screen resolutions—1280×768 and 1280×720, opening the door to amazing new handsets with high-definition 720p displays.
  • More flexible storage
  • NFC wireless sharing
  • Internet Explorer 10: The next version of Windows Phone comes with the same web browsing engine that’s headed for Window 8 PCs and tablets.
  • Wallet: Windows Phone 8’s new digital Wallet feature does two great things. It can keep debit and credit cards, coupons, boarding passes, and other important info right at your fingertips.
  • Better maps and directions
  • Cooler apps and games: Basing Windows Phone 8 on the Windows core will unleash a new wave of amazing apps and especially games, for reasons I’ll touch on in a moment.

Saying all of that, we start to see more mobile OEM’s such as HTC, Huawei and Samsung which also declare about their plans to release WP based devices.

In a poll which asked questions about the new WP8 release, and its compare to Android and iOS, people answered as follows:

Are you impressed with the new Windows Phone 8?

  • Yes, it brings some new cool features (35%, 166 Votes)
     
  • It’s OK, but nothing to write home about (23%, 110 Votes)
     
  • It’s not even close to Android ICS (20%, 95 Votes)
     
  • Yes, it leaves Android in the dust (15%, 71 Votes)
     
  • Underwhelming (7%, 39 Votes)
     

Total Voters: 481

This is how the WP8 start screen shall look like (More or less):

HTC press release from yesterday about their plans to release 3 new HTC devices running WP8 (“Rio”, “Zenith” and “Accord”):

http://www.wired.com/gadgetlab/2012/06/report-three-htc-windows-phone-8-devices-in-the-works/

As mentioned above, Huawei also announced their plan to release a new WP8 device by the end of this year:

http://www.theverge.com/2012/6/20/3104859/huawei-ascend-windows-phone-8-announcement-wp8?utm_source=dlvr.it&utm_medium=twitter

In case you wish to see live videos about the WP platform, features and more, visit the Youtube channel for WP:

http://www.youtube.com/playlist?list=PL0C88F4DC0C435D09

Since this is a Testing Blog, i would like to share an important link which holds information about tools which Microsoft provides for the testers (Certification tools, and more, called MPR – Microsoft Platform Ready tools – soon to be more relevant for the mobile platforms):

http://www.microsoftplatformready.com/dashboard.aspx

http://www.microsoft.com/en-us/download/confirmation.aspx?id=29945

Stay tuned for further news and updates.

Regards,

Eran Kinsbruner