Criteria’s for Choosing The Right Open-Source Test Automation Tools

I presented last night at a local Boston meetup hosted by BlazeMeter a session together with my colleague Amir Rozenberg.

The subject was the shift from legacy to open-source frameworks, the motivations behind and also the challenges of adopting open-source without a clear strategy especially in the digital space that includes 3 layers:

  1. Open source connectivity to a Lab
  2. Open-source and its test coverage capabilities (e.g. Can open-source framework support system level, visual analysis, real environment settings and more)
  3. open-source reporting and analysis capabilities.

During the session, Amir also presented an open-source BDD/Cucumber based test framework called Quantum (http://projectquantom.io)

Full presentation slides can be found here:

Happy Reading

Eran & Amir

7 Mobile Test Automation Best Practices

Developing a mobile test automation scenario isn’t that complicated. Developers and testers use a variety of commercial test automation frameworks or open source tools such as Selenium and Appium to do automation. However, when trying to execute these tests on real devices or integrate them into an Agile or CI (continuous integration) workflow, things get a little complicated.

The major challenges around mobile test automation

The essence of developing test automation is to be able to use and re-use scripts many times, across platforms and environments. Test automation should be as maintainable as possible, especially as new platforms and product features are released. Many organizations that develop test automation for their mobile apps face the following challenges:

  1. Executing the tests against a variety of real mobile devices
  2. Executing these tests in parallel
  3. Leveraging existing test code (re-usability) for new tests
  4. Including real end-user environments/conditions (changing network conditions, low battery) in the tests
  5. Overcoming unexpected interruptions (incoming call, apps running in background)
  6. Running these tests unattended — over night, as part of a Jenkins CI job

These are just few of the challenges organizations confront when trying to progress from older SDLC processes and meet faster releases and enhanced Dev–>Build–>Deploy–>Test–>Deploy cycles.

7 practical test automation tips

Overcoming these challenges starts with few changes in the overall mobile app dev and test processes.

Consider these seven recommendations for building sustainable unattended automation.

Test automation

The key to mobile test automation is to start with a small number of test cases, automate them, and assure that they are robust enough and can be executed in parallel and unattended. Only then should you invest more and grow the test suite.

An important question to ask at the start is: What should I be automating? Organization often do not choose the right tests to automate, resulting in lost development time, weak ROI, and an over-reliance on manual testing.

To learn more about the 7 Ways to Overcome Test Automation Obstacles, please join us next week for a webinar hosted by myself, automation expert and author Daniel Knott, and Perfecto’s Director of Technology Uzi Eilon.

Responsive Web: The Importance of Getting Test Coverage Right

When building your test lab as part of a RWD site test plan, it is important to strategically define the right mobile devices and desktop browsers which will be your target for your manual and automated testing.

For mobile device testing you can leverage your own analytics together with market data to complement your coverage and be future ready, or leverage reports such the Digital Test Coverage Index Report.

For web testing you should also look into your web traffic analytics or based on your target markets understand which are the top desktop browsers and OS versions on which you should test against – alternatively, you can also use the digital test coverage index report referenced above.

Related Post: Set Your Digital Test Lab with Mobile and Web Calendars

Coverage is a cross organizational priority where both business, IT, Dev and QA ought to be consistently aligned. You can see a recommended web lab configuration for Q1 2016 below which is taken from the above mentioned Index – Note the inclusion of Beta browser versions in the recommended mix due to the nature silent updates of these versions deployment on end-user browsers.

WCReport
For ongoing RWD projects  – once defining the mobile and web test coverage using the above guidelines, the next steps are of course to try and achieve parallel side by side testing for high efficiency, as well as keep the lab up to date by revising the coverage once a quarter and assure that both the analytics as well as the market trends still matches your existing configuration.

As a best practice and recommendation, please review the below mobile device coverage model which is built out of the 3 layers of Essential, Enhanced and Extended where each of these layers includes a mix of device types such as legacy, new, market leaders and reference devices (like Nexus devices).

MobileCoverageLayers

To learn more, check out our new Responsive Web Testing Guide.

responsive web testing strategy

Selenium Is the New Testing Tool Standard

Seems like the debate in the world of test automation tools is over.

If few years back HP QTP/UFT (formerly WinRunner) was the standard and most commonly used tool for test automation in the QA space, those days are over.

The shift toward Agile, Devops and such trends together with the digital transformation which includes multi platform testing of Mobile, Web, IOT in a very short amount of times changed the tools landscape and the testing requirements.

See below a snapshot of the top required testing tools which show that the shift already started in 2011 where Selenium passed HP tools in the market adoption.

qtp vs selenium

Sourcehttp://www.seleniumguide.com/

The requirements today are that testing is done as early as possible in the project life cycle (SDLC) and to enforce this process, developers ought to play a significant role – Testing is now being developed and executed by all Agile team members including developers, testers, ops people and others.

In order for the shift and the adoption to grow the tools need to be tightly integrated into the developers environment (IDE’s) which in the digital space might be Eclipse, Android Studio, Visual Studio, Xcode or other cross platform IDE’s like PhoneGap or Titanium.

The additional aspect of test framework adoption such as Selenium and Appium lies in the Open-Source nature of these tools. The flexibility of such open source tools to get extended by developers according to their needs is a great deal compared to closed testing tools such as UFT which are disconnected from the IDE and development environments.

We shall continue to monitor the tools space and movement, but seems like the open source tools is becoming standard for Agile, DevOps practitioners which find these tools suitable for their shift left activities, keeping up with the market dynamics and competition, as well as great enablers for quality and velocity maintainability.

To get some heads up into what is the future of Selenium, and how are the efforts moving on toward making the web browsers drivers (Chrome, Firefox, IE etc.) standard and managed by the browser vendors, refer to this great session (courtesy of Applitools)

http://testautomation.applitools.com/post/120437769417/the-future-of-selenium-andreas-tolfsen-video