Optimizing Android Test Automation Development

Now that we are a few weeks away from Google I/O, and we understand that the complex Android landscape is becoming, even more, complex let’s explore a way Android teams can optimize and plan their test automation across the different platforms and devices.

In the past, I’ve聽written about the need to connect the 3 layers:

  • Application under test
  • Test code itself
  • Device/OS under test

I related back to my old patent that I jointly submitted years ago in the days of J2ME and also wrote a chapter about it in my newly published聽book (The Digital Quality Handbook)

Problem Definition

Android OS families support different capabilities and the gap is growing from one Android SDK to the next. As an example, Android devices older than 6.0, cannot support Android Doze for battery usage optimization, or cannot support App Shortcuts (see below example from Google Photos app). These diffs introduce a challenge to Dev and test team that innovate and take advantage of these features since the test code that shall run against these聽features and devices needs to be turned only towards devices can actually support it.

How can teams sustain a test automation suite that runs specifically on the right devices per supported features?

Proposed Approach

While I don’t have a bulletproof, magic pill to address all challenges that may occur as a result of the above problem, I can surely recommend an approach as described below.

Important to note, that being aware of the problem, is a step toward resolving it 馃檪

Assess Your App and DUT:

  • Map the different features that your app supports or requires the users to grant permissions for
  • Examine your device test lab and filter the devices that support and does not support these specific features

To manage the above, teams can leverage the following:

  • Use an existing ADB command that extracts from a connected device/s the supported feature
    • ADB SHELL
      • PM LIST FEATURES

After running the above command, you will get an output that looks like the below …

Compare The Outputs

Once you know your DUT’s capabilities, as well as your App, features to be tested, you can run a simple output comparison and see what can and can’t be tested – From that point, the optimization should be mostly manual – you will setup your test execution and CI in the lab accordingly. While it isn’t simple enough, it still offers a sustainable approach + awareness to both dev and test team that can be useful throughout the development, debugging and testing activities. In the below visual you can see a capabilities diff between a Samsung Note 5/Android 7.0 (left column) and an older 聽Samsung running Android 5.x device capabilities (right column). An immediate diff out of a larger list that I have shows the fingerprint functionality that is supported on the Note 5 but not on the other Samsung device. Such insight should be used when planning the feature testing across these 2 devices (this is just one example).

Bottom Line

As Google continues to innovate and add more features, the existing devices and test framework will find it hard to close the gaps and that’s a challenge that teams need to be aware of, plan for, and optimize so their release vehicles and velocity remain solid.

Happy Optimization!

Advertisements

3 Motivations That Made Me Switch From iOS to Android

As a mobile evangelist at Perfecto, i foresee the entire mobile and web space for the past 10+ years, following major trends both in the device/hardware front as well as the platform/OS (operating System) front.

I was an Apple user for the past 2 years, using an iPhone 6 Plus device both for my personal as well as my work daily activities. Last month i decided it’s time for a change and i replaced my iPhone with a Google Nexus 6P phablet.

Let me explain some of my reasons to that switch:

  1. Quality and Innovation
  2. Platform Restrictions
  3. Future Looking

vpcqffbw

Quality and Innovation

In the front of quality and/vs. innovation i found out that as a 2 year trend, Apple’s iOS was constantly straggling with quality that mostly came on top of innovative features and end user -experience. For the past 2 years Apple released 10聽versions of iOS 8 stopping at a stable GA of iOS 8.4.1, while for iOS 9 Apple released 10+ versions stopping at a recent 9.3.5 GA release that addresses security issues. To compare this trend to Android platform – Android 5.0聽Lollipop released in November聽2014 and was enhanced till latest version of 5.1.1 (~5 versions in 2 years). Android Marshmallow6.0 was released in October聽2015 and since than only had an additional version of 6.0.1 release. Last month (August 22nd) Google released its new Nougat 7.0 release that is available to users (like me) that hold a Nexus device. iOS 10 is just around the corner with the iPhone 7 devices, but based on the current trend and enormous public Beta versions, it seems like no major changes are expected in the quality/release cadence.

In the Android history we see some major enhancements around sensor based capabilities for payment, logging in as well as UX (user experience) features such as multi window support (see below image), android Doze (battery saving capability). In iOS we also see enhancements around sensors like force-touch, apple pay however these features IMO come in short compared to the platform stability over the past 24 months and the platform constrains which i’ll highlight in the next section.

20160823_142250 Screenshot_20160823-141941

Platform Restrictions

From an ens user perspective, some of the important platform features involves the ability to customize his UX and look and feel of his personal device. Also having the ability to easily manage his media files such as photos and music with a reasonable storage availability. Apple flagship device with massive market share across regions聽is the iPhone 6/6S with a聽default storage (un-expandable) of 16GB – I hardly know a person who has this device/storage size that is happy with that, and does not need to constantly delete files, cancel auto savings of WhatsApp media files and alike. 聽In addition, continuously working with iTunes software as a dependency to media/songs sync is a pain and often i found myself losing my favorite music files or getting them duplicated by simply having to switch from 1 PC to another (people do that, and there are procedures that might have prevented this outcome but still). Compared to the above, most Android devices that are not coming with an external storage option are by default coming with a 64 GB internal memory, and in addition working with music file system is a simple and straight forward task to do.

Switching from my iPhone and iTunes to a Nexus device while having my Gmail account was a very simple thing to do, my music, photos and apps easily “followed” me to the Android device that is already running Android 7 in a stable way.

iOS is not all bad, don’t get me wrong – from an adoption perspective, and device/OS fragmentation this is by far a much better managed platform compared to Android that rolls out its latest GA version in a 4-6 months delay to a non-Nexus device (example: Samsung). In addition the iOS tablets are still a leader in that front with 4-6 years old tablets like iPad Air, iPad 2 that are the most commonly used tablets in the market that can still run iOS 9 OS versions. It is not the case when it comes to Android tablets that tend to be replaced by their end-users in a shorter period of time that iPads.

 

Market_Cal

Future Looking

From a future looking perspective, my opinion is that Google is still going to have a global market share advantage over Apple and will continue to innovate with less frequent releases due to quality than Apple. 2017 is going to show us a continuous battle between Android 7 and iOS 10 in a market that becomes more and more digital and mobile dependent, and with this in mind – the challenge of quality, innovation and less restrictions will be even more critical to independent users as well as large enterprises who are already today fully digital.

As an end-user, i would look at both Google and Apple and examine how their overall digital strategy will transform and enable easier connectivity with smart devices like watches etc., as well as less limited storage and device/OS customization. From a Dev and Test perspective i would assume we will continue to see growing adoption of open-source tools such as Espresso, XCTest UI, Appium etc. as a method of keeping up with the OS platform vendors – Only such open-source frameworks can easily and dynamically grow and support new features and functionalities compared to legacy/commercial tools which are slower to introduce new API’s and new capabilities 聽into their solutions.

Blog series: (1) Mobile Market Landscape: 2015 Highlights

As we wrap up another year, we thought it would be fun and informative to take a look back at the mobile devices, operating system updates and trends that hit the market this year.

Let’s start with this bird’s-eye view graphic of the important releases that made 2015 such an innovative year in mobile.

Mobile Market 2015 Retrospective Calendar, Source (Perfecto's Digital Test Coverage Index Report)
Mobile Market 2015 Retrospective Calendar (Source: Perfecto’s Digital Test Coverage Index report)

There were more than 30 significant smartphones and tablets released this year. Many of these devices such as the iPhone 6S and 6S Plus, Samsung Galaxy S6, the Galaxy Note 5 and the LG Nexus 5X quickly became popular on a global scale. When putting together our quarterly Digital Test Coverage Index we noticed that these newly-minted devices ranked high up in the indexes in both the U.S. and Europe.

But a device is nothing if it’s not running on an updated operating system — and this year brought 15 OS releases (major and minor), from Apple iOS 8.2 to 9.2, and Android 5.1.1 to Android 6.0 Marshmallow. Microsoft released what it hopes will be a mobile game-changer, Windows 10. All of these OS releases will have an impact on the already fragmented mobile space, keeping DevTest teams busy with more re-testing.

This year we also saw disruptive technologies take center stage, such as Apple Force Touch, Touch ID authentication, Mobile Payments, Voice commands and contextual awareness apps based on location.

Samsung Galaxy View Tablet
Samsung Galaxy View Tablet

Looking ahead, it’s clear that 2016 is going to be just as hectic, fragmented, and EXCITING. In the coming year, enterprises can expect mobile and web user engagement to be key business drivers, forcing many organizations to do rigorous testing of new app features on mobile devices, OSes and web browsers to deliver a memorable user experience to customers.

As a sneak in preview in our latest Digital Test Coverage Index, we’ve added a 2016 calendar and we already see new trends emerging such as increasing tablet screen sizes that aim to replace today’s laptops (i.e. the Samsung Galaxy View tablet with its unique screen size of 18.4”, and Apple’s iPad Pro with its 12.9” screen size).

For more details on important mobile and web test coverage trends, download the Digital Test Coverage Index.

Happy holidays to all. Here’s to successful digital test coverage in 2016!

IDC 2012 report around mobile and additional post on mobile testing

Hi

Redirecting to 2 new posts i’ve made on PerfectoMobile blog:

http://blog.perfectomobile.com/2012/11/02/idc-report-mobile-operating-systems-stats-for-q32012/

http://blog.perfectomobile.com/2012/10/26/testing-mobile-enterprise-business-critical-applications/

Enjoy

Eran

Windows Phone 8 handsets are starting to pop out

Hi,

As many anticipated, we are starting to see more and more investment in the new Windows Phone platform, by many OEM’s and not only Nokia which is collaborating for a while with Microsoft.

In this short post, i will list the new upcoming Windows Phone 8 phones which you will soon start to see.

HTC:

HTC is announcing the launch of its new Windows Phone 8 phone called HTC Accord.

The phone comes with a 1.5 GHZ dual core SnapDragon processor, 4.3 ” Screen, 8 MP camera, External microSD聽card, NFC support, as well rumors says that the phone will have support聽for LTE communication.

Read more at: http://www.htcaccord.com/

Samsung:

Samsung announced its new ATIV-S聽Windows Phone 8 phone with the following characteristics: Super AmoLed聽4.8 ” Screen, 1.5 GHZ dual core processor, Full HD 8 MP rear camera, with a 1.9 MP front camera. support聽for an external MicroSD card (which is new in the WP platform) and NFC聽support!. The phone which is running the WP8 will support Internet Explorer 10 browser, Mobile Office suite and the new cloud storage service SkyDrive.

The ATIV聽brand actually starts a new line of products by Samsung for WP8 (ATIV Tab 10.1”, ATIV Smart PC and more)

Read more at: http://www.samsung.com/global/ativ/ativ_s.html

Nokia:

Nokia which is of course the WP platform pioneer, is also announcing on 2 new phones running WP8聽called Nokia Lumia聽920 and Nokia Lumia 820.

The Nokia Lumia聽920 will come with a 4.5” screen, and the Nokia Lumia 820 with a smaller screen of 4.3”

The news around these 2 phone is about their support in the new PureView camera technology, which for these 2 phones will give a 21 Mega Pixel support.

Read more about these 2 new handsets at: http://www.theverge.com/2012/8/31/3281985/nokia-lumia-920-specs-pictures-leak

http://www.engadget.com/2012/08/31/nokia-lumia-820-920-leak/

Summary:

As I always state, the mobile world is dynamic and constantly changing, and we are already seeing that even for the new Windows Phone platform the biggest OEM’s are starting to dive in so it will be interesting to see how such change impacts the mobile market, and the existing iOS/Android and the RIM platforms.

From testing perspective we also see variety of new screen sizes which was and will always be a challenge for testers and test automation (Above we mention already 4.3”, 4.5”, 4.8”). Tablets聽for WP8 as you saw above are also starting to be deployed extending this platform market.

Regards,

Eran Kinsbruner

Picking the right handsets for your project

Hi

We all know that the mobile world is dynamic, plenty of new handsets are being shipped at the same time in which we develop our product and testing on the (what we believe) is the “hottest” handsets in the market.

It is clear that being agile and fast in the way we develop, test and deploy our mobile products is a key to be attractive in the market, however it is also impossible to support all handsets and be ahead of the market.

So – The way to be up to date in the offering, is not simple but possible.

When you start developing your product keep in mind that by picking the “right” 10 handsets which are “hot” in the market you can reach the coverage of ~50% of the market (Note that there are lead devices which represent a whole family of handsets and can give you a lot of value by testing on it), as well if you go to ~30 devices you may reach up to ~80% coverage of the market.

How should you decide than?

The way to do the picking of handsets should combine the 2 following aspects:

– Market research

– Right family identification/lead devices

Market Research: The way to determine what is relevant in the market is to do some research and analysis – either through leading mobile blogs, or even simple – going through the leading mobile operators in the world, and seeing what they are currently selling (e.g. Vodafone global lists today in the top list of devices in Germany: Samsung Galaxy SIII, Samsung Galaxy SII, SEMC聽Xperia Arc S etc. – http://shop.vodafone.de/Shop/smartphones/, if you go to Vodafone UK you will see mostly the same ones, as well as HTC One X and others http://www.vodafone.co.uk/brands/android/index.htm)

Doing a matrix and unification of handsets between the world leading carriers聽in Europe/U.S/Asia should give you the lead handsets which you would like to support and test in the 3-6 months ahead (Per OS – Android, iOS, Windows Phone and BlackBerry).

Families: The aspect of family should be a subset of the above list聽If e.g you reached a common list of lets say 50 handsets, i am mostly certain that the list can be cut into half by doing proper comparison between the various handsets by their OS, Screen resolution and OEM (This can be done through sites like GSM Arena – http://www.gsmarena.com) 聽and minimizing the list by leads, members and families.

Please find attached to the post an up to date list of common handsets by OEM which is sold world wide these days to ease your pain 馃檪聽 –> As you will see, there are a lot of similar handsets across all large operators which can show the main devices to focos on.

MobileWorldHandsetsDistribution

P.S: With regards to the leading Android/iOS tablets these days:

iOS – iPad 2 and iPad 3

Android – Samsung Galaxy Tab 10.1, Motorola Xoom, Asus Nexus 7, Dell Streak 7, Samsung Galaxy Tab 7, Sony Tablet S. Asus Transformer TF300, Asus Transformer TF700

Regards,

Eran

Cross Platform mobile development tool – Titanium

All

This is a Testing blog however i聽recently get a lot of questions around Mobile development/porting for cross platform聽projects (Android/iOS etc.).

For this cases there are several good and free tools out there which can be used such as PhoneGap (http://phonegap.com/download/), EggPlant (http://www.testplant.com/products/eggplant/for-cross-platform-testing/)聽and Titanium.

In this short post i will just point you out to the tool and give a very high level details about it for you guys to go and try it out for yourself

Titanium is a free tool by Appcelerator (http://www.appcelerator.com/platform/titanium-sdk) which allows quite quickly to develop a cross platform聽mobile application which can be than deployed on iOS and Android Phones/Emulators and also PC Web browsers.

The tool allows to develop in Java Script and customize your resources pending the operating system you which to deploy on:

You need to have on your machine the Android SDK and point its location in the Titanium SDK to allow execution and debugging on the Android Emulator.

You have through the SDK options to create new project from samples or templates (HTML5, Tabbed APP and more).

Feel free to give it a try and comment

Regards,

Eran Kinsbruner