Testing has been with me all my adult life. In college I was required to understand testing when I took programming classes. In my career I’ve had to do testing for console games and portables. Today, as a member of TestBird, I get to work with a great team, eager to help in the testing of today’s latest mobile games in China and around the world. As a leader in functional game testing using our cloud services and our phone lab, I learned a bit of more about the subject and why functional testing is one of the most essential things in mobile gaming today.
What is Functional Testing?
When we talk about testing in the game life cycle, I usually put the many types into three different groups:
Most early testing is for basic functionality issues and simple bugs that come up in all gaming life cycles (commands not completed, loading errors, incomplete task completion, wrong score, etc.).
End Testing usually begins regression testing and certification testing. Regression testing checks to make sure all major game ending bugs that were in the bug reports are fixed, and that most other bugs are either gone or incredibly hard for the player to reproduce. Certification testing prepares the game for the standards a console/handheld maker has for their games.
One of the last things tested is functional testing. This is the final major testing phase that makes sure the games loaded properly, display properly, save properly, have no load issues and don’t have game ending bugs. When I worked with consoles, this was a pretty painless (and very boring) process as the number of SKUs for a certain console was very limited and there wasn’t a lot of OS updates.
Functional Testing with Smartphones
As Android continues to allow free access to their OS, many companies today have built their smartphones on the opportunity to find the right form factor, screen size, and other cool add-ons to attract customers to purchase their phone or tablet. This has created a large array of SKU’s on phones and fragmentation of the industry has happened. Adding an array of OS versions and you can see this is getting complicated.
Many in the game industry are now worried with the amount of phones if they will need to test every phone for functionality and playability of their games. Some of the larger game companies purchase hundreds of phones every year to make sure they have enough phones for their own QA teams to use while in early testing.
Segmentation in Android is massive as seen in this chart from OpenSignal.com
Why is this important for game companies?
Making sure you catch as many players in the market as you can is extremely important. Many players in the early days of iPhone and iPod Touch were very surprised when new OS versions would come out and the newest games wouldn’t support their older devices. I sure was when it happened to me and it continues to worry me today if my phone will one day be too obsolete to play the newest mobile games.
Having a high saturation rate for your game is a must for developers. Making a game just for one type of phone is no longer enough to create the biggest and most popular games. The top 5 games in China tested at TestBird comply with over 300 different mobile devices, accounting for 95% of the Android market in China.
What is TestBird doing to help?
One of the reasons to love TestBird is that it has over 1,000 different devices ready for testing and can test games in 24 hours. The amount of time I spent in testing games were incredibly long and probably spent the company I worked with thousands of dollars. Also the amount of money, space, and security needed for purchasing all these phones could break the budget for many small companies and not close to possible for indie developers.
Functional testing to me is a necessary thing that can have many in the industry overwhelmed with the possibilities and challenges with the mobile landscape of today and tomorrow. But the great thing for me is knowing that there are solutions available for companies that talk to me.