Optimizing Mobile Test Automation Across The Pipeline

With the massive innovation that drives the digital market these days, organizations are continuing to develop features, as well as new test code to cover these features.

What I’ve learned is that often, the test code developers would not always stop and look back into their existing test suites and validate whether the new tests that are being developed are somehow a superset to existing ones. In addition, legacy tests are a continuous load and overhead on your SDLC cycles length if they are not being maintained over time.

Oil Transport

Many Owners To The Same Problem

Since we live in an agile/DevQAOps world, test code development is not a QA only problem, but rather everyone3s. Tests are being executed throughout the pipeline from Dev to integration and pre/post production testing.

Use of smart tagging mechanism for your test scenarios (login), suites (App A) and types unit, regression) can be a good step towards gaining control over your tests.

Without some context, discipline, and continuous structured validation of the tests, it will become harder as you progress your SDLC to debug, analyze and solve defects (would be like finding the key in the below visual mess)

Find the Key in the Picture

Recommended Practices

  • Develop the tests with context, tags and proper annotations that would make sense to you and your team even 12 months from the development day. Make sure that in your execution reports you then have a way to filter using these annotations to only get the view of a given functional area, platform etc.
  • Match your device under tests capabilities to the test code and application under test. Make sure that you focus e.g. your fingerprint based tests only on the devices that support it (API XX and above).
  • Perform test code review every agreed upon time – in such review, group your feature specific test suites and try to optimize, merge, eliminate flakiness, identify missing coverage areas etc. It is harder to do it as the time progresses, so depending on your release cadence and test development maturity, set the right goals – more reviews would be better than less – it will also be shorter and more efficient that way since the delta between such review will be smaller.
  • Drive joint Dev, Test, Product, Marketing decisions based on data – When you have the ability to get quality analysis from your entire test suites, it is recommended to gather all counter parts and brainstorm on the findings. Which tests are most effective, can we shrink based on the data the release cycles, are we missing tests for specific areas, are there platforms that are more buggies than others, which tests takes longer than others to finish etc.
  • Optimize your CI and build-acceptance testing – based on the above intelligence, teams can reach data driven decision about what to include in their CI as well. Testing in the build cycle via CI should be fast, reliable with zero false positives. With quality insights on your tests, you can decide and certify the most valuable and fastest tests to get into this CI testing, and by that to shrink the overall process without risking coverage aspect.

CI_Dash1.png

Bottom Line

A test is code, and like you refactor, maintain, retire and improve your code, you should do the same to your tests. Make sure to always be in control over your tests, and by that, gain control over your quality of your app in a continuous manner.

Happy Testing!

Advertisements

How the “Digital Quality Handbook” Was Born

Travel back with me… to late September 2016. It’s the Jewish New Year, and I am in Boston, MA. As I celebrate the passing of another great year, I think to myself, “After being in the software quality space for nearly 20 years, isn’t it about time that I reach out to the community of thought leaders and influencers and create an asset that can fill a gap in the market that we can give back to the world?” A book. A practical book. A “how-to” for DevOps practitioners, designed to make them better, faster, and more… perfect(o).

You see, when it comes to assuring the quality of web, mobile and IoT apps, the market is still struggling with key questions around test coverage, automation best practices, optimization of test automation suites, accomplishing more tests within the pipeline of software build cycle, the practice of shifting left and much, much more.

So, while my wife and children continued celebrating in the next room, I immediately (right then and there) started writing the intro to a book that would, eventually, bring together actionable ideas and practices from many of the world’s most recognized experts, thought leaders and influencers in the area of software quality.

To make it easier to both develop and consume the content, the book is set out in four logical sections:

  1. Introduction to continuous quality and the digital space
  2. Advanced test automation practices
  3. Achieving DevOps maturity in the digital era
  4. Expanding quality coverage with UX and non-functional testing

If you’re reading this article close to its post date, I’m currently down in Orlando, participating in a book signing at the StarEast testing conference. Danny McKeown from Paychex, one of the technical reviewers of the book, is with me, both participating in the signing and speaking at the event.

To name the market leaders who took part and contributed to this book:

  1. Microsoft (Donovan Brown)
  2. Applitools (Adam Carmi)
  3. TestFairy (Yair Bar-On)
  4. Applause (Doron Reuveni)
  5. CA & BlazeMeter (Jonathon Wright, Noga Cohen, Jacob Sharir)
  6. InfoStretch (Manish Mathuria)
  7. Rabobank (Wim Selles)
  8. Utopia Solutions (Lee Barnes)
  9. Angie Jones
  10. Jean Ann Harrison
  11. Lior Kinsbruner

And from Perfecto:

  1. Amir Rozenberg
  2. Roy Nuriel
  3. Paul Bruce
  4. Chris Willis
  5. Uzi Eilon
  6. Yoram Mizrachi
  7. Roi Carmel

Without this crew of contributors, the book wouldn’t be what it is today. Some of the contributed content includes:

  • The best way to include visual analysis testing as part of your test code, using any available open-source framework
  • How to develop API tests that complement your mobile UI test automation
  • How to include non-functional performance testing and UX as part of your overall test strategy
  • How to extend open-source tools like Protractor to better test your hybrid app
  • The bible of UX testing
  • What a valid and high-ranked XPATH should look like (with link to an online free tool that provides that rank to you)
  • How to include chatbots testing into your existing mobile testing plans
  • Where in the overall SDLC strategy does crowdsource testing and beta testing fit

Fun fact: We launched the book on Amazon on March 3rd. On March 5th, at approximately 2:51pm Eastern Time, the book had been added to the Hot New Releases in the Software Testing sidebar, and made it to the #1 Bestseller slot in that same category. We took a screen shot. It really happened!

To get your own copy of the book, please refer to this URL – and if you find it valuable, feel free to share your feedback with me.

Happy Reading!

Google Mobile Friendly With Perfecto and Quantum

Guest Blog Post by Amir Rozenberg, Senior Director of Product Management, Perfecto

resize

Google recently announced “Mobile First Indexing”, from Google:

To make our results more useful, we’ve begun experiments to make our index mobile-first. Although our search index will continue to be a single index of websites and apps, our algorithms will eventually primarily use the mobile version of a site’s content to rank pages from that site, to understand structured data, and to show snippets from those pages in our results (Source).

screen-shot-2017-02-13-at-5-33-26-pm

More recently they made the Google Mobile-Friendly tool and guidelines available. A very nice interactive version is available here, and images at the bottom of the thread, while there’s also an API (which, thanks to Google, can allow users to exercise first before they code). Google also offers code snippets in several languages.

Notes:

  • Google takes a URL and renders it. If you run multiple executions in parallel there’s no point in sending the same URL from every execution because the result would be the same
  • Google returns basically “MOBILE_FRIENDLY” or not. Suggest to set the assert on that
  • The current API differs from the UI such that it only provides the results for Mobile friendly (and the UI gives also mobile and web page speed). Hopefully, Google adds that to the response 😉
  • This will probably not work for internal pages as Google probably doesn’t have a site-to-site secure connection with your network.

 

For developers and testers who do not have time, testing mobile friendliness repeatedly probably will simply not happen. That’s why I integrated Google Mobile-Friendly API into Quantum:

  • Added 2 Gherkin commands
// If you navigate directly to this page
Then I check mobileFriendly URL "http://www.nfl.com"
// If you got to this page through clicks
Then I check mobileFriendly current URL
  • Added the Gherkin command support (GoogleMobileFriendlyStepsDefs.java)
  • And the script example is pretty simple:
@Web
Feature: NFL validate

  @SimpleValidation
  Scenario: Validate NFL
    Given I open browser to webpage "http://www.nfl.com"
    Then I check mobileFriendly current URL
    Then I check mobileFriendly URL "http://www.nfl.com"
    Then I wait "5" seconds to see the text "video"

 

That’s it. Next steps:

 

Ideas for future improvement:

  • You can automate the validation such that every click would trigger a check with Google behind the scenes.

Just for fun, some more screenshots for detailed analysis for NFL.com:

 

screen-shot-2017-02-13-at-5-33-48-pm

 

screen-shot-2017-02-13-at-5-34-09-pm

screen-shot-2017-02-13-at-5-34-23-pm

 

 

3 Motivations That Made Me Switch From iOS to Android

As a mobile evangelist at Perfecto, i foresee the entire mobile and web space for the past 10+ years, following major trends both in the device/hardware front as well as the platform/OS (operating System) front.

I was an Apple user for the past 2 years, using an iPhone 6 Plus device both for my personal as well as my work daily activities. Last month i decided it’s time for a change and i replaced my iPhone with a Google Nexus 6P phablet.

Let me explain some of my reasons to that switch:

  1. Quality and Innovation
  2. Platform Restrictions
  3. Future Looking

vpcqffbw

Quality and Innovation

In the front of quality and/vs. innovation i found out that as a 2 year trend, Apple’s iOS was constantly straggling with quality that mostly came on top of innovative features and end user -experience. For the past 2 years Apple released 10 versions of iOS 8 stopping at a stable GA of iOS 8.4.1, while for iOS 9 Apple released 10+ versions stopping at a recent 9.3.5 GA release that addresses security issues. To compare this trend to Android platform – Android 5.0 Lollipop released in November 2014 and was enhanced till latest version of 5.1.1 (~5 versions in 2 years). Android Marshmallow 6.0 was released in October 2015 and since than only had an additional version of 6.0.1 release. Last month (August 22nd) Google released its new Nougat 7.0 release that is available to users (like me) that hold a Nexus device. iOS 10 is just around the corner with the iPhone 7 devices, but based on the current trend and enormous public Beta versions, it seems like no major changes are expected in the quality/release cadence.

In the Android history we see some major enhancements around sensor based capabilities for payment, logging in as well as UX (user experience) features such as multi window support (see below image), android Doze (battery saving capability). In iOS we also see enhancements around sensors like force-touch, apple pay however these features IMO come in short compared to the platform stability over the past 24 months and the platform constrains which i’ll highlight in the next section.

20160823_142250 Screenshot_20160823-141941

Platform Restrictions

From an ens user perspective, some of the important platform features involves the ability to customize his UX and look and feel of his personal device. Also having the ability to easily manage his media files such as photos and music with a reasonable storage availability. Apple flagship device with massive market share across regions is the iPhone 6/6S with a default storage (un-expandable) of 16GB – I hardly know a person who has this device/storage size that is happy with that, and does not need to constantly delete files, cancel auto savings of WhatsApp media files and alike.  In addition, continuously working with iTunes software as a dependency to media/songs sync is a pain and often i found myself losing my favorite music files or getting them duplicated by simply having to switch from 1 PC to another (people do that, and there are procedures that might have prevented this outcome but still). Compared to the above, most Android devices that are not coming with an external storage option are by default coming with a 64 GB internal memory, and in addition working with music file system is a simple and straight forward task to do.

Switching from my iPhone and iTunes to a Nexus device while having my Gmail account was a very simple thing to do, my music, photos and apps easily “followed” me to the Android device that is already running Android 7 in a stable way.

iOS is not all bad, don’t get me wrong – from an adoption perspective, and device/OS fragmentation this is by far a much better managed platform compared to Android that rolls out its latest GA version in a 4-6 months delay to a non-Nexus device (example: Samsung). In addition the iOS tablets are still a leader in that front with 4-6 years old tablets like iPad Air, iPad 2 that are the most commonly used tablets in the market that can still run iOS 9 OS versions. It is not the case when it comes to Android tablets that tend to be replaced by their end-users in a shorter period of time that iPads.

 

Market_Cal

Future Looking

From a future looking perspective, my opinion is that Google is still going to have a global market share advantage over Apple and will continue to innovate with less frequent releases due to quality than Apple. 2017 is going to show us a continuous battle between Android 7 and iOS 10 in a market that becomes more and more digital and mobile dependent, and with this in mind – the challenge of quality, innovation and less restrictions will be even more critical to independent users as well as large enterprises who are already today fully digital.

As an end-user, i would look at both Google and Apple and examine how their overall digital strategy will transform and enable easier connectivity with smart devices like watches etc., as well as less limited storage and device/OS customization. From a Dev and Test perspective i would assume we will continue to see growing adoption of open-source tools such as Espresso, XCTest UI, Appium etc. as a method of keeping up with the OS platform vendors – Only such open-source frameworks can easily and dynamically grow and support new features and functionalities compared to legacy/commercial tools which are slower to introduce new API’s and new capabilities  into their solutions.

Responsive Web: The Importance of Getting Test Coverage Right

When building your test lab as part of a RWD site test plan, it is important to strategically define the right mobile devices and desktop browsers which will be your target for your manual and automated testing.

For mobile device testing you can leverage your own analytics together with market data to complement your coverage and be future ready, or leverage reports such the Digital Test Coverage Index Report.

For web testing you should also look into your web traffic analytics or based on your target markets understand which are the top desktop browsers and OS versions on which you should test against – alternatively, you can also use the digital test coverage index report referenced above.

Related Post: Set Your Digital Test Lab with Mobile and Web Calendars

Coverage is a cross organizational priority where both business, IT, Dev and QA ought to be consistently aligned. You can see a recommended web lab configuration for Q1 2016 below which is taken from the above mentioned Index – Note the inclusion of Beta browser versions in the recommended mix due to the nature silent updates of these versions deployment on end-user browsers.

WCReport
For ongoing RWD projects  – once defining the mobile and web test coverage using the above guidelines, the next steps are of course to try and achieve parallel side by side testing for high efficiency, as well as keep the lab up to date by revising the coverage once a quarter and assure that both the analytics as well as the market trends still matches your existing configuration.

As a best practice and recommendation, please review the below mobile device coverage model which is built out of the 3 layers of Essential, Enhanced and Extended where each of these layers includes a mix of device types such as legacy, new, market leaders and reference devices (like Nexus devices).

MobileCoverageLayers

To learn more, check out our new Responsive Web Testing Guide.

responsive web testing strategy

Responsive Web: Test for the Real User Experience

One of the great benefits of building a responsive web site (RWD) is it can give the user a consistent web experience across any digital device, in any location.

Related Post: Responsive Web and Adaptive Web: Pros and Cons

When it comes to RWD testing, it’s important to test the navigation and functionality on desktop web browsers and mobile devices, but that alone is not enough to guarantee a consistent user experience at all times. The end user is constantly moving between environments throughout the day, and these environments have various attributes, including:

  1. Network conditions (Poor, good, no network)
  2. Locations
  3. App context based on platform and location
  4. Background activities (apps running and consuming resources)
  5. Ads and other popups that block your site content (see image below)

IMG_8543

With so many real user environments to consider for both mobile and the desktop web, testing teams should include user conditions in their RWD test plan on top of the traditional testing for UI, navigation, functionality and client-side performance. It will give your DevTest team peace of mind and reduce quality risks significantly.

To learn more, check out our new Responsive Web Testing Guide.

responsive web testing strategy

Responsive Web: Five Testing Considerations

With more and more consumers expecting to shop, bank, work and socialize across different devices, organizations are embracing responsive web design (RWD) as a tool to help them deliver a consistent digital experience on every screen.

multiplatform-1024x636

Growth of cross-device transactions (Source: Criteo’s State of Mobile Commerce Report)

But due to the complexity of digital environments and user experiences — responsive web is easier said than done. Organizations that develop RWD sites often face challenges when testing to assure smooth website navigation and a great user experience across multiple devices and platforms.

For more information, read our new Comprehensive Guide to Building a Responsive Web Testing Strategy

To get there, we recommended including the following five building blocks as part of your RWD test plan.
RWDTests-1024x368

Testing for these five areas will help achieve sufficient test coverage, a great user experience and higher traffic to your site.

To download the complete guide for testing RWD Site, go here

responsive-web-testing-strategy-2-600x315

User Conditions Importance in Web App Testing

Usually when talking about Web testing, QA teams tend to stick with functional, layout and performance testing. In this blog we would highlight the importance of testing for web apps from the end-user perspective and assuring that the experience is also covered in the test plans.

In today’s competitive landscape many web app sites are exposed to a lot of interrupts whether from web add-ons or in the majority of cases adware popups which often interrupt the user flows within the web page or even cover existing and important content from your web page.

Screen-Shot-2015-09-09-at-11.54.14-AM-640x275









Source: Mashable


In such cases as mentioned above, Dev and Test team need to asses the impact of such pop ups from upper bar security and other add-ons interruptions together with larger ad pop ups covering your web content – as seen above, sometime even when blocking the ads would not solve the problems since the ad vendor will sense that they are on and will pop a different screen.

As a supporting fact to the pain ads are adding to web site owners, we see reports of more than 5.5 millions downloads of Adblock plus tool for FireFox browser just over the last 30 days.

As a quick tip in this blog – enhance your web test plan beyond functionality and cover the cases of environment conditions.

Tests to Include Within Automation Suite

When developing a mobile or desktop test automation plan organization often struggle with the right scope and coverage for the project.

In previous post, i covered the test coverage recommendations in a mobile project and now, i would like to also expand on the topic of which tests to automate.

Achieving release agility with high quality is fully dependent today more than ever on continuous testing which is gained through proper test automation, however automating every test scenario is not feasible and not necessary to meet this goal.

In the below table  we can see some very practical examples of test cases with various parameters with a Y/N recommendation whether to automate or no.

As shown below, and as a rule for both Mobile, Web and other projects the key tests by definition which should be added to an automation suite (from ROI perspective and TTM) are the ones who are:

  • Required to be executed against various data sets
  • Tests which ought to run against multiple environments (Devices, Browsers, Locations)
  • Complex test scenario’s (these are time consuming and error prone when done manually)
  • Tedious and repetitive test cases are a must to automate
  • Tests which are dependent on various aspects (can be other tests, other environments etc.)

Picture1

Bottom line: Automation is key in today’s digital world, but doing it right and wisely can shorten time to market, redundant resources and a lot of wasted R&D time chasing unimportant defects coming from irrelevant tests

Happy Testing!

 

 

Q1 2016 Calendar Overview

Q12016

The year just started but as you can see, the market is already busy and we can see as a continuation to 2015, that Apple is much more active on its bug fix releases compared to Android.

Sign up for my quarterly Digital Test Coverage index report to stay up to date with the market trends, top devices, OS versions and desktop browsers.

Digital Test Coverage Download Page

Happy Testing!