It is becoming more and more challenging to test apps and Web sites on different devices, with different screen sizes, running different operating systems, and with multiple browsers installed.
How can you guarantee your Web site or app works on every device and on every browser?
Nothing beats testing on the same devices as your users, as opposed to simulated versions of those devices. Physical interactions such as pinching, zooming, and scrolling, hardware features like the camera, GPS locations, the accelerometer, and factors such as battery consumption or site performance are all best tested using physical devices.
We established a device lab at Verdens Gang toward the end of 2013. Until then, we had a handful of mobile devices and tablets stored on a shelf in the office. They represented only a few of the many phones and operating systems our customers used. The majority of mobile testing was performed in emulators or on the device of the developer or designer.
One of our biggest problems was that the test devices were never charged when testers needed to use them, so the testers wasted a lot of time charging up devices before testing. We identified the need for a systematic way of acquiring, storing, and loaning devices and testing mobile sites and apps.
Hence, the establishment of a device lab.
Below are some of our experiences with setting up a device lab.
What is a mobile device lab? A mobile device lab is a collection of mobile devices available for testing new apps and services.
What were the mobile device lab requirements?
The device lab needs to represent the mobile devices, operating systems, and browsers of our users. Older devices are retired from the device lab in correspondence with the OS/browser/devices we support at that time. We also ask the testers not to update the operating systems on the devices.
The devices chosen for the device lab are based on:
- The most popular devices visiting our free Web site.
- The most popular devices running our Android, iOS, and Windows apps.
- The devices of users who have reported bugs.
- A range of devices close to retirement (a combination of devices running the minimum versions of OS and browsers we support).
- New devices we believe will be popular.
- The most popular devices visiting our free Web site.
The device lab needs to contain devices of different screen sizes.
The device, where possible, should support both Wi-Fi and cellular connections; we want to test the devices both on Wi-Fi and and the cellular network. The tester needs to use his own SIM card on the test phone if he wishes to test on a cellular network.
It should be possible to lock the device lab. Devices are expensive, and we don’t want to lose any of them.
Devices need to be constantly charged and immediately available for testing. There is nothing worse than having to wait for a device to charge each time you want to test something.
The device lab needs to be accessible for multiple departments, so it should be located where all departments can access it.
It must be possible for testers to borrow devices from the lab and test at their desks.
Who uses the device lab? Web developers, app developers, designers, testers, and the support department. The device lab is shared among all departments and internal companies at VG. For example, VGTV (Web TV company), Tactus (internal ad agency), E24/Dinepenger (global and personal financial publications), and VG (news site).
What are the alternatives to owning a physical device lab?
There are a number of commercial and open-source alternatives to owning a device lab and physically testing on devices. They range from changing the user-agent in your browser to appear as if it’s a mobile device; testing on remotely accessible device labs; crowd-sourced testing; or using emulators and simulators to test.
The most frequent and easiest way to test is by switching your user agent in your browser to simulate a certain device, e.g. User Agent Switcher.
There are numerous commercial and non-commercial emulators and simulators available to help test devices. They include Web sites that offer browser emulation, physical device labs in the cloud, and Google and Apple simulators. (See this list of emulators available, or “19 tools to test your site for mobile devices.”)
Crowd-sourced testing: A very effective way to test multiple devices with different network connections and at different locations is through crowd-sourced testing. For example, “releasing alpha and beta versions of your Android” app to a limited number of your Android app users.
- There are numerous device labs available where external developers and designers can go and test their products. Check out OpenDeviceLab.com.
How are the devices set up? All devices are installed with the default factory settings. The devices are constantly connected to a power source. We have created an Apple ID, a Google account, and a Windows Live account to be used on the different platforms.
Each device is registered in a document displaying device name, operating system version, and screen size, and each device has the same information displayed on a sticker on the back of the device.
Constantly powering up the batteries might expedite the degradation of the batteries. It might be worth using a wall socket electricity timer to limit the frequency of top-ups. For example, turn off the charging of devices at night.
How do people test devices? People are asked to sign-in and sign-out devices. They register with their names, the date they took the device, and the name of the device they are testing. When they return the device, they register that they have returned the device.
Where do we get the devices? The majority of our devices have been bought or donated to the device lab by employees. We try to buy the newest, most popular brands when they are launched.
What does the device lab look like? There are several possibilities, ranging from making a device lab to buying a commercial solution. We opted for the latter and chose to buy a “TabCabby Charge & Sync,” a movable and lockable unit connected to a power supply that can store 25 devices.
The unit also has the ability to sync devices from a master device or compile apps on multiple devices at once from one central connection (for example, from xcode to iOS devices).
The shift in readership traffic from desktop to mobile platforms and the resulting focus on being a mobile-first newspaper has placed a greater importance on delivering the best possible user experience on all devices. Tools like a device lab have made it easier for us to deliver on this promise.