Mobile Testing Is Different
Mobile Testing or Mobile QA has taught us that the traditional testing approaches and methodologies need tweaking or changing. What worked as a defined process for testing desktop, client-server or web applications didn’t exactly fit mobile.
If you are a tester who is just starting to test on mobile or simply want a fresh perspective (even dare I say it a mobile Developer) then this is for you. Of course, if you are simply interested in the subject feel free to read on as well.
12 Critical Mobile Testing Issues:
- Fragmentation: as a tester you may have to support not only Android and IOS but, multiple devices and OS versions. Android especially is a market with hundreds of devices of varying specifications. Screen size, resolution, processors, memory and more all have impact on device performance and behavior. Now factor in that Apple and Google both release new devices at least once a year but, depending on the countries in your market, there may be many Android OEM devices you want or need to support.
- Mobile Device Labs: if fragmentation is a problem, this is a solution worth investigating. Some testers have their own devices onsite but for many who don’t have the budget, or want a more expansive solution then SAAS solutions where the tester can connect to live devices thru a web application this delivers nicely. The tester can choose the devices and with some solutions can run both manual or automated test scenarios.
- Compliance: mobile testing of apps requires the tester to be fully familiar with the rules of the App Store and Play Store. (Of course there are other stores beyond just Apple and Google) In addition, compliance has grown to include GDPR and Disability support.
- App Types: Mobile allows us to create apps of different types and technological base: native apps, mobile web apps, hybrid apps and more recently Progressive Web Apps (PWA). Each of these has their own testing challenges, scenarios and scope.
- In-app Purchase: IAP is a niche in E-Commerce where the customer can make purchases from within the app. Examples can be E-books, multimedia, game purchases (allowing the game player to progress in the game faster) and real-world items. IAP has it’s own compliance standards for each store and can be more complex to test that run-of-the-mill E-Commerce.
- Alpha / Beta testing: if you are testing an app intended for the App or Play Store then how do you get this into the hands of a small group of alpha or beta testers without publishing the app? Apple allows Developer Account holders to use Testflight to do this and Google has Play Store Alpha / Beta testing. Once these apps are published and distributed to your selected testing group, you can easily publish to the Stores on completion of your test review or roll to a new bug-fix version as needed. In the case of testing features like in-app purchase, the tester needs this option to facilitate test purchases.
- Crash Logs and Analytics: mobile testing requires a skilled tester to have a strong familiarity with logs on devices and services that provide crash and user analytics. Crash logs can be accessed from test devices by connecting IOS devices to XCode and Android devices to Android Studio. There are other methods but these often are the fastest when you want to add the critical information to the bug you are reporting. Crash analytics are frequently a SAAS solution that will show the tester frequency of bugs, and which devices or OS version(s) the bug reproduces on. Each provide actionable intelligence at different scales. Crashlytics is one popular example which can be added to your apps.
- App Distribution: there are cases where you may not want to publish an app to the App / Play Store or you want to distribute an app for internal or enterprise use. In these cases solutions like HockeyApp, now renamed and revamped as Visual Studio App Center (after being acquired by Microsoft) offer a way to do so. App Center combines several of the other solutions listed here and is cross-device and cross-platform.
- Automated Testing: testers will ultimately seek out a framework that enables them to test UI and function, both rapidly and repeatedly. Appium, Protractor and others allow these testing scripts to be run on real mobile devices. Automated testing on mobile has its own challenges and ROI. There are also cases where this will not work.
- UI, UX and User tolerance: these 3 are interconnected facets of the same issue. Ultimately users have a lower tolerance for poor UI or UX and are willing to remove an app that displeases them. Android and IOS have best practises for design and UI frameworks of their own. E.g. Material Design on Android OS. Testers should be aware of the design standards, what are acceptance criteria and be able to identify UI / UX bugs that can compromise the quality of the mobile app.
- Reverse compatibility: As mobile OS versions advance with time certain features, SDKs or APIs become deprecated and or out-of-date. Features and functionality that worked in previous versions may then stop working due to these changes. One example was the way IOS handled webview in Native apps; after IOS9 this changed (due to a major overhaul to remove serious bugs). Some developers and companies reduce this overhead by supporting only those OS versions after such a change, others only support the latest version of the OS.
- Non-functional testing: mobile apps live and die on their scalability, app performance and transparency if they function on low-spec mobile devices. This type of testing should play a critical part in optimal mobile testing to prevent nasty surprises when your app goes to production.
The Warptest POV
Mobile testing shares as much as it doesn’t to other platforms we test on. The biggest issue is perhaps how rapidly the mobile landscape changes. As processors, GPUs shrink and battery life grows, screens lose their bezel, handsets lose earphone jacks or data / charging cables change their connectors. Meanwhile, wireless charging is becoming mainstream and of course camera resolution increase. Augmented, Virtual and Mixed Reality impact the capabilities, functionalities and spec of mobile devices.
Mobile testing has to evolve as rapidly as the devices we test on.
This post is a sampler or taste-test. The solutions listed are not exclusive, each exist in a competitive ecosystem which mobile testers need to be continually learning about. Some solutions are offered by Apple and Google, others by 3rd parties or Microsoft. Hopefully this is a starting point that wets your appetite for exploration. I’m always happy to hear from you about alternatives and learn something new.
Some of what I have written about will get their own, more detailed posts or Vlogs so keep watching this space.