Selenium Is the New Testing Tool Standard

Seems like the debate in the world of test automation tools is over.

If few years back HP QTP/UFT (formerly WinRunner) was the standard and most commonly used tool for test automation in the QA space, those days are over.

The shift toward Agile, Devops and such trends together with the digital transformation which includes multi platform testing of Mobile, Web, IOT in a very short amount of times changed the tools landscape and the testing requirements.

See below a snapshot of the top required testing tools which show that the shift already started in 2011 where Selenium passed HP tools in the market adoption.

qtp vs selenium

Sourcehttp://www.seleniumguide.com/

The requirements today are that testing is done as early as possible in the project life cycle (SDLC) and to enforce this process, developers ought to play a significant role – Testing is now being developed and executed by all Agile team members including developers, testers, ops people and others.

In order for the shift and the adoption to grow the tools need to be tightly integrated into the developers environment (IDE’s) which in the digital space might be Eclipse, Android Studio, Visual Studio, Xcode or other cross platform IDE’s like PhoneGap or Titanium.

The additional aspect of test framework adoption such as Selenium and Appium lies in the Open-Source nature of these tools. The flexibility of such open source tools to get extended by developers according to their needs is a great deal compared to closed testing tools such as UFT which are disconnected from the IDE and development environments.

We shall continue to monitor the tools space and movement, but seems like the open source tools is becoming standard for Agile, DevOps practitioners which find these tools suitable for their shift left activities, keeping up with the market dynamics and competition, as well as great enablers for quality and velocity maintainability.

To get some heads up into what is the future of Selenium, and how are the efforts moving on toward making the web browsers drivers (Chrome, Firefox, IE etc.) standard and managed by the browser vendors, refer to this great session (courtesy of Applitools)

http://testautomation.applitools.com/post/120437769417/the-future-of-selenium-andreas-tolfsen-video

Does Your Enterprise Have a Central Strategic Mobile Resource?

Mobile technologies are much more than a trend, and are no longer something that organizations and enterprises can overlook or pretend to ignore.

Mobile is already a significant market platform that generates substantial revenues, driven by the ever-growing use of smartphones and mobile applications. However, alongside these exciting business opportunities, mobile technologies and applications also introduce new types of risks that were not an issue in the desktop application world.

As mobile grows, the accompanying pains and risks require (even dictate) the adoption of a robust mobile strategy built upon a centralized organizational solution. Such a strategic solution should be based on an appropriate set of tools that can be used by management, development and testing teams wherever they may be located around the globe.

Mobile applications require enterprises to take into account several mobile-specific components that are not relevant to the desktop space:

  • Governance and availability of devices (smartphones and tablets). The extremely dynamic and fragmented mobile market requires continuous testing of mobile products. To ensure timely delivery to market, you must have access to an up-to-date set of mobile devices at all times.
  • Global solution which enables offshoring, testing and collaboration among diverse and distributed teams
  • Robust manual testing solution to handle user interface and other “look & feel” issues
  • Cross-device test automation solution in order to accelerate TTM and to enable QA to efficiently perform continuous regression testing while moving from one version of the product to the next
  • The ability to plug a dedicated, mobile-specific testing platform into your existing ALM tools, processes and organizational skill sets (i.e., seamlessly integrate with existing ALM quality suites)
  • Mobile Application Performance & Monitoring (APM) must be taken into account. Availability and response time for your mobile application are critical for business success. Studies show that mobile users are less tolerant than desktop users to performance and availability shortcomings.
Figure 1: World Quality Report 2012-2013, CapGemini

When formulating your mobile strategy, do not take the risk of overlooking these important aspects of mobile testing. In addition, the latest World Quality Report (as illustrated above) stresses the importance of using a cloud-based, centralized organizational resource to enhance agility, competitiveness and cost-effectiveness.

Using this blueprint, Perfecto Mobile has deployed mobile testing solutions for leading enterprises and mobile carriers worldwide.  Our experience shows that addressing these key pillars assures ongoing mobile quality while cost-effectively streamlining the entire mobile lifecycle for the various mobility groups (Development, Testing, Performance and Operations).

So here’s the takeaway:

Mobile is a serious business platform which is getting more complex and fragmented every day. New devices will continue to pop up and new mobile OS versions will be released with increasing frequency, making mobile testing that much more difficult.

To thrive in such an environment, enterprises ought to focus their time, energy and resources on developing high quality mobile products that serve their customers’ needs. Integrating the right mobile solution with your existing tools and processes will allow your team to focus on developing real mobile testing scenarios, automation, etc., rather than dealing with organizational issues. By supporting the pillars listed above, your mobile testing solution can serve as a central strategic resource across the enterprise IT environment.

Best practices for iOS mobile application testing

Hi

iOS changed the mobility game, no doubt about it. It paved the way for the ‘mobile era’ by offering amazing functionality with a simple user experience.  However when it comes to testing and monitoring, working with the iPhone/iPad mobile application can be anything but simple…

As the iOS app market continues to produce record growth, challenges and complexities surrounding iOS application testing also continue to interfere with development. A key challenge of iOS testing is that, unlike the open-source Android OS, Apple iOS is a closed operating system. Added complexity during the development and testing stages arises with a closed system, since users can’t extract necessary data from low level objects, which are essential for test automation. So, what’s the best approach for getting the necessary level of access to the iOS device – rooting (jailbreaking) or compile-time source instrumentation? Should you base your testing on native objects or OCR-based screen analysis?

Let’s take a deeper look into some of these challenges and why a cloud-based hybrid approach is important to offer developers and testers the necessary coverage, capabilities and flexibility to deliver better iOS apps and deploy them with confidence.

Rooting (jailbreaking) vs. Source Instrumentation (compile-time)

There are two common methods used today in the mobile testing industry to address this challenge (i.e. access to the low level objects): rooting (jailbreaking) and source instrumentation (i.e. compile-time solution).

Jailbreaking refers to the process of removing the limitations placed by Apple on the iOS device in order to get low level (root) access to the operating system. This allows the tester to be able to recognize the objects within the application being tested.

Source Instrumentation is performed by compiling the application being tested with an additional piece of code that provides access (“back door”) to the low level OS for object recognition. This code enables the tester to execute the low level calls and get the Object ID’s from the operating systems (without the need to root/jailbreak the device).

The decision as what approach to adopt strongly depends on several considerations (below are just few):

1)    The used SDLC process

2)    Corporate policies

3)    Application under test

4)    Frequency of testing

Perfecto Mobile provides its end users with the freedom to choose what fits them best, while taking into consideration the advantages and disadvantages of each approach. When customers need to quickly test either a new iOS version or a new iOS device, the jailbreaking approach is less suitable. In such a case, the compile-time method is preferable – even though it complicates the SDLC by introducing additional code to the application being tested.

On the other hand, using a jailbroken device lets you test the application with the exact code by which it will be released (compile-time mandates that before store submission, you will remove the “back-door” or be exposed to serious security issues). This eliminates the need for compilation and intrusive operations which could potentially pose a risk to quality. Companies using a compile-time approach should also consider possible regulations (such as HIPAA) which enforce testing on the final binary (and not on debug version, test friendly version, etc.)

The combined (hybrid) approach lets you choose which type of tests to implement on which iOS device according to the nature of your application, project needs, and policy. When the test devices are deployed and securely managed in a “private cloud” (such as that offered by Perfecto Mobile), such a configuration guarantees that the jailbreak method does not introduce any risks or abuse of the platform for non-testing purposes. The jailbroken device is used only for testing purposes in a closed and secure testing environment. This is analogous to the use the way iOS devices used for development have a “developer signature,” as well as the way Android devices used for development have more levels of access than those required during the normal ALM cycle.

The Need for a Hybrid Approach to Object Recognition

Testing a mobile application requires strong object recognition capabilities. The use of visual analysis might not be sufficient, for example, the OCR technology can detect UI issues and glitches on the test devices, but cannot ensure 100% accuracy due to its heuristic nature. On the other hand, low level objects might “miss” the obvious qualification that a visual analysis can easily detect. That’s why a hybrid approach incorporating both visual and Native object analysis is imperative for covering all mobile business cases. Such an approach is supported by Perfecto Mobile.

Object level analysis vs. Visual analysis

This screenshot above shows the differences of using an object level analysis as opposed to visual analysis (object level analysis would not have detected the overlapping of the button on the text).

The Perfecto Mobile Approach: Go Cloud, Go Hybrid

Perfecto Mobile’s experience as a market leader has taught us that the best approach is to present each customer with all possible alternatives making them available inside the cloud.

Real devices + emulators (in the cloud),  OCR screen analysis + OS level native objects (in the cloud), rooted/jailbroken device + non-rooted/jailbroken devices (in the cloud)

With hundreds of thousands of automation hours running every month on our platform, we are well-positioned to suggest and guide, but not to “judge” what’s best for everyone…

Perfecto Mobile hybrid object support on a rooted android and a non-jailbroken iPhone

Regards.

Eran Kinsbruner

Latest news from the mobile world

Hi,

In the last couple of days and with the new iPhone5 announcement i wanted to gather the main highlights in one place:)

  • iPhone 5 (http://www.apple.com/iphone/) which was announced shall be available next week together with a possible launch of the new iOS6!
  • Details around the app store applications –> Total of 700,000 apps available in AppStore, while ~250,000 also ported to iPad
  • battle in the mobile world continues, and few great smartphones put themselves as leaders and competitors to the new iPhone 5:

** HTC One X – http://www.htc.com/www/smartphones/htc-one-x/ (4.7” Screen)

** Samsung Galaxy SIII – http://www.samsung.com/global/galaxys3/ (4.8” Screen)

** LG Optimus G – http://reviews.cnet.com/smartphones/lg-optimus-g-unlocked/4505-6452_7-35427825.html (4.7” Screen)

** Motorola Razr Maxx HD – http://www.engadget.com/2012/09/05/motorola-droid-razr-hd-maxx-hands-on-bigger-battery-beautiful/ (4.7” Screen)

** Sony Xperia S – http://www.gsmarena.com/sony_xperia_s-4369.php (4.3” Screen)

** Nokia Lumia 920 – http://www.nokia.com/global/products/phone/lumia920/ (4.5” Screen)

Eran Kinsbruner

Windows Phone 8 handsets are starting to pop out

Hi,

As many anticipated, we are starting to see more and more investment in the new Windows Phone platform, by many OEM’s and not only Nokia which is collaborating for a while with Microsoft.

In this short post, i will list the new upcoming Windows Phone 8 phones which you will soon start to see.

HTC:

HTC is announcing the launch of its new Windows Phone 8 phone called HTC Accord.

The phone comes with a 1.5 GHZ dual core SnapDragon processor, 4.3 ” Screen, 8 MP camera, External microSD card, NFC support, as well rumors says that the phone will have support for LTE communication.

Read more at: http://www.htcaccord.com/

Samsung:

Samsung announced its new ATIV-S Windows Phone 8 phone with the following characteristics: Super AmoLed 4.8 ” Screen, 1.5 GHZ dual core processor, Full HD 8 MP rear camera, with a 1.9 MP front camera. support for an external MicroSD card (which is new in the WP platform) and NFC support!. The phone which is running the WP8 will support Internet Explorer 10 browser, Mobile Office suite and the new cloud storage service SkyDrive.

The ATIV brand actually starts a new line of products by Samsung for WP8 (ATIV Tab 10.1”, ATIV Smart PC and more)

Read more at: http://www.samsung.com/global/ativ/ativ_s.html

Nokia:

Nokia which is of course the WP platform pioneer, is also announcing on 2 new phones running WP8 called Nokia Lumia 920 and Nokia Lumia 820.

The Nokia Lumia 920 will come with a 4.5” screen, and the Nokia Lumia 820 with a smaller screen of 4.3”

The news around these 2 phone is about their support in the new PureView camera technology, which for these 2 phones will give a 21 Mega Pixel support.

Read more about these 2 new handsets at: http://www.theverge.com/2012/8/31/3281985/nokia-lumia-920-specs-pictures-leak

http://www.engadget.com/2012/08/31/nokia-lumia-820-920-leak/

Summary:

As I always state, the mobile world is dynamic and constantly changing, and we are already seeing that even for the new Windows Phone platform the biggest OEM’s are starting to dive in so it will be interesting to see how such change impacts the mobile market, and the existing iOS/Android and the RIM platforms.

From testing perspective we also see variety of new screen sizes which was and will always be a challenge for testers and test automation (Above we mention already 4.3”, 4.5”, 4.8”). Tablets for WP8 as you saw above are also starting to be deployed extending this platform market.

Regards,

Eran Kinsbruner