Optimizely’s decision to ditch its free plan suggests A/B website testing is dead

This post was originally published on this site

On December 20, 2017 Optimizely, the leading A/B testing platform, informed its users that it is planning to sunset its free Starter plan on February 28, 2018.

We estimate that this change will impact around 70,000 websites* which will soon be left without a tool that allows customers to test and alter site experience.

Here’s the official message as sent out by Optimizely via email:

Hi there,

We are notifying you about an important change to our pricing plans that will impact your Optimizely account. As of February 28, 2018, our Starter plan option will no longer be available as the Starter plan is based on a version of Optimizely that is being sunsetted.

You will be able to view and run experiments through your Starter plan through February 28, 2018. After this date, you will still be able to access your Optimizely account and results data, however any actively running experiments will be paused, and you will not be able to launch any new experiments or campaigns.

The Optimizely X platform provides organizations with the necessary tools to experiment across every channel, application and feature. We hope you will take this opportunity to consider moving to our new SaaS platform, Optimizely X. Our Starter option was designed as an easy way for customers to get started with Optimizely as they began their journey with experimentation. Over time, we’ve found that the companies that see the greatest success are the ones who are able to commit to experimenting over a longer term, in an ongoing and iterative fashion. For that reason, we will be focusing our efforts and now offer annual plan options going forward.

If you’d like to learn more about our annual subscription plans, please visit optimizely.com/plans.


The company states that they will discontinue the ‘old’ platform, Optimizely Classic, giving users only the option to move to Optimizely X – a solution heavily focused on enterprise customers.

Optimizely X positions itself as an experimentation platform. It offers versatile features such as product recommendations, personalization and server-side testing but comes at a high price that most users of Optimizely’s free plan will not be able to effort.

At this point, it is unclear what will happen to the paid legacy plan types – bronze, silver, and gold. Still, it is interesting to see how Optimizely shifted focus.

In its early days, the company aggressively promoted its free A/B testing platform to establish market dominance but now seems to move further and further away from its roots.

When Optimizely introduced Optimizely X in 2016, we saw how the company shifted its focus from A/B testing towards more sophisticated experimentation. In its communication, Optimizely positions itself as a platform for continuous optimization and personalization. With Optimizely X, the company has also dropped its free Starter plan and has since been focused exclusively on enterprise sales.

Sunsetting the free plan was only a question of time and yet the decision to make such a cut tells us more about the future of conversion optimization than Optimizely cares to admit.

A/B testing is dead

For years it seemed A/B testing was the universal answer to conversion optimization. The marketing world believed that a scientific approach to testing would revolutionize how we do business and through A/B testing any website could become a cash-making machine.

The only catch is that setting up a scientifically valid experiment is more complex than we were led to believe and in 2017 reality finally caught up with us.

It isn’t for a lack of effort or a lack of time instead it is for a simple lack of knowledge about statistics. Frequently, A/B tests are not set up correctly, polluted by external factors, or never reach statistical significance. Not to mention the ones that do but are effectively worthless since they have no sustainable impact on business and growth.

Trained statisticians would tear the average marketer’s A/B test to shreds, and it seems Optimizely has finally decided to call time of death on oversimplified CRO attempts. After all, bad results also reflect badly on the tool itself, and with Optimizely X the company seems to distance itself from its ambivalent past.

Why is A/B testing so error-prone?

For starters, statistical significance, the certainty with which one is able to declare a winning variation, is an issue. On average, an online experiment will need more than 25,000 visitors to reach significance thus many experiments fail due to insufficient data.

Often, something goes wrong in the preparation. Using classical statistical techniques requires setting a sample size and committing to a minimum detectable effect in advance. Further, underestimating how testing several goals and variations at once can increase errors and finally data pollution may cause false positives.

Not to mention that an A/B test takes more than just creating a tweaked variation of the original page. It’s about identifying a problem and providing a hypothesis, backed by data and to test a new experience based on that.

In other words, A/B testing is a rich man’s game. It requires enough traffic, experience, and information to test fast and with high confidence. If you don’t have the necessary resources to set up A/B tests correctly, you are better off not taking the risk of operating under false assumptions.

Okay, A/B testing is hard, but why did Optimizely do this?

No one can deny that A/B testing has the potential to make an impact, but it is not the jack-of-all-trades CRO marketers hoped it would be. It seems Optimizely wants to shield itself from unsatisfied users who simply can’t do what they hoped to achieve. Hence, the push towards more sophisticated experimentation.

However, above email does not explain why Optimizely has chosen to move towards an enterprise-only business model. After establishing market dominance Optimizely has decided to no longer cater to a big chunk of its original user base which leaves many looking for an alternative.

So what can companies do?

From Optimizely’s shift towards more elaborate experimentation, we can definitely learn one thing, namely that A/B testing is not the secret weapon it was ought to be.

Often enough the only solution to false results is to stop looking at your customers as a homogeneous group and take a different approach towards optimization.

In fact, with Optimizely X, the company gives us a hint at where we should be looking next; personalization. Creating a custom user experience and tailoring the site to each user individually is no longer reserved for big-data companies. Although personalization is often associated with complex algorithms, some Optimizely competitors have made it available to SMEs.

What are the alternatives to Optimizely?

Google Optimize

If your heart is really set on A/B testing, take a look at the free version of Google Optimize. It offers a powerful editor for both, visual editing and more advanced changes through code editing. Google Optimize is deeply integrated into the Googleverse which makes performance analysis using Google Analytics and optimization for AdWords campaigns particularly easy.

Potential downsides might be that you can run no more than three concurrent experiments and the tool doesn’t offer audience targeting. So, audience-based experimentation and personalization are not its strong side.


Unless.com is a newcomer among personalization solutions and yet it has quickly gathered a fanbase by providing extensive customer support.

This company puts a clear emphasis on personalization – primarily focusing on creating unique experiences for each of your visitors, with a little bit of A/B testing mixed in. Their platform lets you choose from 20+ targeting conditions to group your visitors into audiences and use 1:1 personalization to further improve relevance.

Be aware that you might run into limitations using their WYSIWYG (What-You-See-Is-What-You-Get) editor which is less powerful than a visual editor.

It is easy to implement (you can get started without adding a script), integrates with Google Analytics and pricing starts at 9 USD/month, making it accessible to anyone with a website.


VWO offers an extensive feature palette, covering all kinds of optimization use cases.

If you are looking for a testing platform that is more powerful than Google Optimize, VWO has got you covered for 49 USD/month. Its optimization and personalization solution will set you back 299 USD/month.

There is a catch though. Due to their implementation using asynchronous loading, speedy websites might see a FOUC (flash of unstyled content) potentially polluting experiments. Still, having all tools – testing, surveys, and personalization – wrapped up into one platform, certainly facilitates optimization.


If A/B testing has not brought you any luck in the past, it might be a good idea to invest in a tool that helps you better understand user behavior.

Starting at 89 USD/month, Hotjar offers a wide range of solutions such as heatmaps, visitor recordings, and feedback polls that help you uncover hidden opportunities. Bear in mind that additional tools to put your new ideas into action might be required.

Final verdict

If you are just looking for a new A/B testing tool, we recommend to go with Google Optimize. Even though it is limited to three concurrent experiments those might be plenty for low-traffic websites.

For marketers who feel their tests might be failing due to a lack of insights into user behavior, a heatmap and user recording tool such as Hotjar will be a good option.

If you want to give a new optimization technique, namely personalization, a try, start with Unless.com. It will help you create a better experience for each of your visitors.

You might also want to check out bestpersonalizationservices.com – a website that lists, reviews, and rates different personalization solutions.

* Using https://builtwith.com/ we analyzed the number of websites that installed the Optimizely script between Mar 1 2015 and Aug 1 2016 – the time range when the free Starter plan was available. Next, we subtracted the number of paying Optimizely users (as estimated by market insiders), leaving us with about 70.000 websites affected by the change. 

Yvonne Koleczek is CMO at Unless. She writes about personalization and growth marketing. Previously, she worked for the Frankfurter Allgemeine Zeitung. In her spare time, she loves to redecorate other people’s apartments.

Comments are closed.

Proudly powered by WordPress | Theme: Baskerville 2 by Anders Noren.

Up ↑