A/B Testing Push Notifications for Maximum Results

Push notification A/B testing is the secret weapon that allows brands to derive massive ROI from push.

We absolutely love push notifications around here, and so do our users, who send hundreds of thousands of push notifications every month through their MobiLoud apps, to drive engagement, recover abandoned carts, promote hot offers, and much more.

But you can't just fire off push notifications into the void and hope for the best.

You need to follow a robust strategy, and the best strategies are shaped by data.

That's where A testing for push notifications is crucial.

In this article, we’re going to teach you the 80/20 of push notification A/B testing. We’ll show you why you need to test, what to test, and how to get the whole thing set up and running on autopilot.

At the end of this article, you'll have a thorough enough understanding of how to intelligently test your push notifications, to be able to go away and start generating amazing results from your push campaigns.

Native apps with push notifications are the best way to capitalize on the rising number of mobile-first shoppers today. Launching an app for your store is easy with MobiLoud. Click here to learn more about why MobiLoud is the best way to build your app.

What is A/B Testing?  

A/B testing is a crucial part of the conversion rate optimization toolbox, and is a scientific, experimental process.

Traditionally it means testing two versions of a web page, with one element different in each.

The original page is the control, while the altered page is the variation.

The tester makes a hypothesis predicting that certain changes to the page will affect metrics like dwell time, CTR, or conversions. 

By randomly sending traffic to both versions and measuring how users interact with them, the hypothesis can be tested, and the results used in future, higher performing designs. 

The typical workflow for A/B testing is:

1. Collecting performance data to find optimization opportunities
2. Building a hypothesis to propose changes to elements
3. Creating variations to test against a control 
4. Using A/B testing tools to run the experiment & analyze results
5. Collecting further data during to design the next tests & iterations

Basically, A/B testing allows you to make data-driven decisions. It's a proven method that produces great results, in almost every aspect of digital marketing - from emails, to landing pages, to ads, and more.

Now let’s move on to push notification A/B testing specifically.

A/B Testing Push Notifications 

So we know what A/B testing is in general, so what about push notifications? 

The overall concept is the same, we form hypotheses about what elements of a notification will improve performance, then we create variations to test whether our assumptions are correct. 

We use tools, typically provided by our push notification platform, to run the experiments and measure the results. Then we implement what we learn into future campaigns and come up with new hypotheses to test.  

The main difference is that there are simply fewer elements we can change for push notifications compared to emails, landing pages, and ads. 

This ultimately makes testing and optimizing easier, and we’ll cover an exhaustive list of what to test shortly. 

But just because push notification A/B testing is simpler, doesn’t mean it isn’t super powerful. 

Push Notification A/B Testing Tools

Generally, you'll have A/B testing included if you use any well-known push notification service provider like:

• OneSignal
• Airship
• PushEngage

So there should be no need for you to use separate tools just for testing push, and almost definitely no need for any kind of custom implementation. 

OneSignal, for example, is the platform we integrate with - allowing you to send unlimited push notifications to your users from your MobiLoud iOS and Android apps. 

Whatever features OneSignal supports, you can set up for your MobiLoud apps. 

For A/B testing, OneSignal has you covered. 

OneSignal allows you to A/B test up to 10 message variants on the Pro plan and even gives you A/B testing features on their Free plan!

You can conveniently track everything through their native dashboards.

If you use another push notification provider, check out their docs. All the market leaders will have similar A/B testing features with minor variations in functionality and pricing. 

There’s no need to worry about anything too “technical”, as the platform you use already built the necessary infrastructure. 

This is great, as it allows you to focus on designing intelligent tests. Let’s look at how to do this.

Push Notification A/B Testing Process

On a high level, here are the important steps for setting up push notification A/B tests. 

1. Goal Setting 

What does success look like? What metrics are you trying to improve or optimize for?

Typically, with push, we’re looking for higher CTR or conversions.

2. Hypothesis formation

Use your marketing intuition, industry best practices, or previous data to come up with a testable guess about the impact of some change. 

It should be simple, and easy to state in a sentence. Like this:

• Notifications with a clear call-to-action, like 'Shop Now', will have a significantly higher click-through rate than notifications with a generic CTA like 'Learn More'
• Incorporating emojis in push notifications will lead to a 20% increase in open rates compared to notifications without emojis.
• "Push notifications containing discount offers will see a 40% higher conversion rate compared to notifications with new product announcements."
• "Reducing the frequency of push notifications to once a week will decrease the unsubscribe rate by 15% compared to daily notifications."

The basic format: “If we change x (element) then y (result) will happen”.

3. Segmentation 

Using the push platform’s tools, you’ll divide users into two or more random, yet statistically similar groups to ensure a fair and unbiased test. 

This is fundamental to ensure that the outcomes of the test are relevant to your hypothesis - and  not due to underlying differences in the audience groups.

4. Variant Design 

Here you’ll actually develop the two (or more) variants with only one key difference - the variable that your hypothesis rests on. 

Examples are the message content, the call to action, or the timing of the notification.

This sole focus on the single variable is crucial for drawing clear conclusions from the test. 

5. Test Duration / Timing 

Finally, decide how long the test will run and when. 

The test must run long enough to gather a meaningful amount of data, but not so long that random events and factors can skew the results.

7 Variables to Test for Push Notifications

One of the most important parts of an effective test is what you test.

You want to test variables that can have a meaningful impact on results, and thus give you valuable insights you can use for future campaigns.

Here are a few suggestions for variables to test: 

• Frequency: push notifications are known for “fatiguing” users if they’re bombarded with them too much. Testing to explore the optimal notification frequency to engage users without annoying them is a good idea!

• CTA: Experiment with different CTAs to see which leads to higher click-through rates. Test the wording of the CTA, its positioning, its style, and anything else that could make an impact.

• Copy Style: test different versions of the notification text to see which hits hardest. Try varying the tone (formal vs. casual), use of emotive language, clarity and conciseness, and use of psychological triggers like scarcity and urgency.

• Media: the classic push notifications are text only but you can also test images, video, and audio to see how they perform versus text only.

• Timing: since push notifications are immediate, the time of day or day of the week that your users receive them can have a huge impact on their engagement, test it out and see how they react!

• Personalization: test the impact of personalized messages by using their name or referencing past behavior versus generic messages to see how the personal touch impacts target metrics.

• Length: Test short versus long notifications to determine the optimal length.

These seven variables should keep you occupied for quite some time, and optimizing all of them will dramatically improve the results of your push campaigns. 

Read more: Learn the Best Time to Send Push Notifications, for a starting point to use when crafting your A/B tests.

Push Notification A/B Test Tips 

Before we wrap up, a summary of the most important points and our top tips. 

Important: focus on testing one thing at a time. 

You should focus on single variables per test to make sure your results are valid.

Once you’ve proven (or disproven) your hypothesis, then you can move onto the next one. 

This makes it simple and manageable for small businesses without large teams of optimizers. 

It’s also crucial to get a large enough sample.

The groups you test on need to be large enough for statistical significance, so that you can be confident that the results are not from mere random chance. 

You can read more about sample size and statistical significance in this primer.

On a practical level, it means don’t worry so much about testing until you have a significant (thousands) number of users receiving your notifications. 

Until then focus on following best practices and acquiring more users!

A/B Testing Push Notifications with MobiLoud

We’ve seen that push notification A/B testing is generally as simple as: 

• Reading the docs of your push notification provider
• Making hypotheses to test 
• Running experiments 
• Implementing results and making new hypotheses in an iterative loop 

MobiLoud can make this easier for you in several ways. 

Firstly, MobiLoud converts your website into native apps for iOS and Android, allowing you to send native push notifications in the first place.

We build you high-end apps, handling the entire development process, and we integrate everything from your existing site. 

We've built thousands of apps for countless eCommerce brands - from small startups to multibillion dollar multinationals.

MobiLoud apps come pre-integrated with OneSignal, the best push provider on the market.

You can send unlimited push notifications to your customers through OneSignal, and also make use of OneSignal’s push notification A/B testing features. 

We can help you get set up with this as part of our full service where we handle all aspects of the app build for you, including ongoing updates and maintenance. 

We also have our own system for abandoned cart notifications.

These are extremely powerful for eCommerce brands. And we’ve created a system where an abandoned cart sequence is automatically triggered when we detect items in a users’ cart (when the app is closed).

Some of our users have recovered as much as $200,000 in revenue in just 30 days through abandoned cart notifications!

If you’re an eCommerce brand who wants to build mobile apps, abandoned cart notifications are all automated and handled by our team for you, so you can focus on testing and optimizing other kinds of notifications like promotions and welcome sequences. 

MobiLoud also integrates everything from your site into your apps. So whatever web tools you use for advanced analytics or landing page A/B testing will work in the apps too. 

You can literally transfer everything great about your web store into the apps!

Push notifications are just one way that a mobile app will drive more revenue for your brand. Get a full estimate of how much you could add to your bottom line with our free eCommerce App Revenue Calculator.

Get Started with Push Notification A/B Testing and Boost Your Push ROI Now

You now know enough to get started with push notification A/B testing, and use data to craft more informed and better performing push campaigns.

You can get started by A/B testing your web push notifications (as well as applying the same techniques to email, and other parts of your business).

But you should definitely start thinking about building your own eCommerce apps for iOS and Android, if you haven't already.

Apps give you access to native push notifications, one of the most powerful communication channels for modern brands - as well as a host of other benefits, that lead to higher AOV, conversion rate and LTV.

The best part is, MobiLoud makes it super easy to launch your app, with push notifications built in and ready to go.

It starts with a quick chat with one of our app experts. 

Just book a free consultation today, and we'll show you just how easy it is to build and launch your app, and start a major growth curve for your brand.