A/B Testing — 25 Apr 2021
How to Improve Marketability of a Game with Concept Testing
Anastasia Sidoryk
Anastasia Sidoryk 
Game publishers may have no idea that they have weak marketability until they soft launch their product, and at this point they have already invested huge resources into the development. Yet, there’s a way to make sure that the game can hit its business goals before writing a single line of code. I’m referring to successful game concept testing.
Concept testing happens on emulated pages of the App Store and Google Play. Basically, you show the product page of a game that you intend to develop to your target audience to see how many of them click the “Get” button.
The numeric results of such game concept tests, or metrics, provide insights into whether your app has a good chance of success whatsoever. Overall, successful game concept test is a data-driven approach to launching games and a viable instrument to continuously improve your marketing message.
Now let’s dig deeper into…
The best thing about game concept test is that it saves you a good deal of time, money and other resources for the development. On the pre-launch stage, you get data-driven insights into how the target audience would respond to your game in total or to the new feature.
You can start with A/B testing the game subcategory to answer the question “Should I develop a card game or a puzzle game?” This is the most general thing you can A/B test.
Additionally, mobile game publishers can compare a few mobile game concepts that they are considering and select the most well-performing one. You can also choose between a few positioning strategies or identify the most well-performing audience, like should it be a mobile game for kids or adults, etc.
Details can be validated too. For example, you can check which of the game features resonates better with your target audience and once you understand them, get down to the development.
Overall, there are two major applications for game concept test.
Theme testing can be done early on, at the prototyping stage of the game development when you are figuring out what type of game you should be creating.
Here, you A/B test the core gameplay, or setting, as well as the elements in it. For example, the theme can be Middle Ages or Fantasy, and the elements can be dragons or zombies, etc.
Tests help game publishers conduct games user research see what users are more interested in, as well as find a less resonating theme, which is an equally important insight.
When you manage to increase the IPM (installs per thousand of ad impressions), you can lower your CPI. In other words, the better your ad and the app store page performs, the cheaper it is to acquire users.
By experimenting with different styles and app store optimization and tying the creative to the gameplay, after games user research you can arrive at something that might resonate better than your current elements, and thus impact your IPM in a positive way.
To build one, start with analyzing the market, games user research and check if there are similar games out there, whether they are in demand and what features they have. For the A/B tests to have really meaningful results, the tested concepts should differ distinctively, not just by the color of the button on the third screenshot.
At this stage, make sure you know your target audience. One of the mistakes that some game publishers make using the SplitMetrics platform is A/B testing on a very small audience. For example, instead of targeting the United States or several countries in Europe, they run A/B tests on, let’s say, Switzerland. As a result, they get a very low conversion and, moreover, the conclusions based on such are far from being meaningful.
Another mistake related to the target audience is choosing the wrong age. We saw some publishers failing their game concept test for a younger audience, while they could launch the same concept for an older segment of users and that would work. The only recommendation to give here is to study and know your target audience.
After you’re done with the hypothesis and games user research, you have to decide what your game should look like.
That’s right, a pretotype that comes before the prototype.
Let’s say you’re hesitating between the two settings – a cybercity and an alien planet before start building your game. To understand the best direction cost-efficiently, you could purchase the two packages from an asset store. Then you may want to modify them, put on screenshots and run tests to see which one is more resonating with your target audience.
Thus, instead of investing time and money in creating your assets from scratch, you just buy them for a hundred dollars and get numerical user feedback behind each of the environments under A/B testing.
To do that, you have to design the app store visuals based on the pretotype of your game. These are a few versions of the product page with elements – an icon, screenshots, videos (potentially) and UA banners.
Next, you create a copy, or text variations for the app – a title, subtitle and app store description.
Once you’ve got the visuals and copy prepared, you’re all set for showing the app page variations to potential users.
There are two ways to do that using SplitMetrics.
Firstly, you can run a one-variation experiment to check the overall performance and conversion rate. This type of test will give you the right direction in a very cost-effective way: you don’t need to waste money buying the traffic to test several variations while you’re not sure whether your overall concept will work or not.
Secondly, you can perform multivariate A/B tests, where one variation stands for one concept, and check which one would perform better.
After you’ve found the top-performing variation, build a strategy based on this games user research to improve it until you’re happy with your results and ready for the launch.
To assess the performance of variations under A/B testing, you have to consider their conversion rate, time on page, bounce rate, engagement rate, direct installs against the screenshot view, and the screenshot full-screen view, which basically means an open rate.
At the highest level, look at the IPM, or installs per thousand impressions. When assessing the metric, you may benchmark your own IPM against the game genre standard, but most importantly, compare it against your past performance. The IPM will change either way as you’re making changes to your assets.
Improving CVR will directly increase the install rate and ROI of your game and, most importantly, optimize the cost of acquiring users if you will follow our step-by-step instructions. So what are the ways to achieve the app store conversion rate improvement snd achieve your goal?
The major CVR improvements come from optimizing screenshots and previews of your game. However, our findings show that two and more videos may decrease the conversion rate checklist, so you might want to check the hypothesis that you could perform better on the app stores with no videos whatsoever.
You can improve the conversion rate by adding the brand name to your game’s title on the App Store or Google Play. However, this might work for big brands in the first place.
Take into account seasonality: adding two snowflakes to your icon or screenshots before the Christmas holidays might make a big difference. Just test if it works for your game.
Another thing that you can test is localization: find out if you need to adjust your app page to a particular storefront, and that may give you a major boost in another region.
Concept testing is not a one-time event, but a continuous process of detailed improving your game with concept test and games user research. It’s really important to keep running tests at the prototyping, pre-alpha and alpha development stages before start building your game to make sure that the marketability suits the core loop of your game.
Thinking about A/B testing your game concept? Give the SplitMetrics platform a try! Request a free demo with one of our specialists.
This article is based on the podcast organized by the Deconstructor of Fun. Listen to us sharing even more valuable insights on how to improve mobile game marketability.
