The Secret to Winning in Search: How To Run Category Split Tests

Ever thought about how category and search ranking affects your app’s conversion rate? ASO hackers are using a new type of split tests that can precisely tell you all about it.

Many mobile publishers are testing icons, screenshots and descriptions via Google Play experiments or with advanced tools like SplitMetrics. Screenshot tests lead the way here; icons come next. App page design matters – we’ve seen amazing results with Angry Birds, Paper by Fifty Three, Fresh Planet, and hundreds of other publishers.

However, before people get to your app store page, they first make a choice between dozens of similar apps. Category tests empower marketers to optimize conversion beyond the app store page, providing context to collect data on the whole visitor journey.

Interested to learn more? Let’s dive in.

Competitive Intelligence for App Store Pages: Category Split Tests

With regular app store a/b tests, you drive traffic to your app’s landing page. With category tests, you take visitors to a search page with several other app alternatives. From there, you can measure your conversion from search or compare scenarios when your app ranks, say, second versus fourth. 

So How Do You Run a Category Test on the App Store?

First, you create an ad banner with your app next to some of the competing apps in the category. Below is a simple ad I designed to test the conversion rate for fitNshare, a social networking app for fitness fans from I put it next to its competitors: BurnThis, Peak Physique, FitUp, and StayWow.

Next, I used the banner to run a Facebook campaign.


Screen Shot 2016-04-24 at 4.40.18 PM



When people click on my banner, they will get to a page that opens in a mobile browser and perfectly mimics the app store experience. Once I drive visitors to this page, I can now track how long they stayed there and what app they clicked on. After they hit “Install”, they’ll be redirected to the “real” app store. Meanwhile, I’ll be collecting data points for further analysis.

If Your App Ranks Lower, You Can Still Win

Perhaps, not surprisingly, the first position in search is a definite winner in all of the search & category tests we’ve run.

However, as you go further down the page, the results are not as obvious. The second position in search doesn’t necessarily rank second by conversion; the third doesn’t always perform better that the fourth one, and so on.

One of our clients, a gaming apps publisher, ran such test on its Slots app against competitors in the same category.



Even though the icon was in the fourth position, it converted better than the second and the third option.

The icon featured a bright background and simple contrasting graphics in the center; the colors were selected to make the app stand out.

See the Bigger Picture and Test Alternative Scenarios

Category tests give you actionable data to improve your conversion rate from search results. You see the bigger picture as you follow visitors’ journey from a category page to the app’s landing. Experiments with various search and category scenarios will give you valuable insights on how to optimize your page to beat the competition.

Ready to give Search & Category tests a run?

Sign up or request a demo


  • Anton Lebedev

    Thanks for interesting insights.
    Why don’t your clients add screenshots in search results like real Apple App Store search results look like?

    • Anya Pratskevich

      Hi Anton! Glad you noticed – SplitMetrics actually allows you to do both, together with the screenshots and simply icons as shown. The screenshot version, following with the example of fitNshare, will look similar to this: