The problem with most client-side A/B testing tools

The Problem

Making a JavaScript tool for running A/B tests on web pages and apps isn't very hard. Any junior web developer could write you a script that will swap out elements on the page, maybe based on some URL parameter that gets passed to the page. You go to https://page.com?v=1 or https://page.com?v=2 and see some different copy or a different hero image, and you're done. The hard parts come before any DOM manipulation happens, and some are glossed over by the above assumption that you have the users split into buckets already. Building a robust system that will bucket users, display the relevant page variation, and send the events somewhere for tracking without slowing the page to a crawl can be a huge challenge.

Here we'll talk about some of the problems we've seen in client side A/B tracking implementations, and discuss how we've attempted to improve on them with Engauge.

Take Optimizely, the leaders in A/B testing on the web, for example. When a user hits your page, they download a hash of all of your active experiments for the page. From there, their scripts determine which bucket needs more views, adds you to it, and changes the DOM accordingly. While this approach has the benefit of placing users evenly into experiment buckets, it's fantastically slow, as you can easily see by loading their home page and watching the DOM jump around like a fourth grader at Chuck-e-Cheese.

Created from a visit to optimizely.com on 5/29/2017.

One key contributor to this failure of UX is a lack of developer control. If, rather than just clicking around in a web-based tool and deploying these changes to a production site, you were forced to implement the DOM changes with the help of one of the site's developers, the developer would likely make the decision to hide the image that may be changed before a selection is made. This would reduce the glaring impact of the changes being made, and likely result in less bias in your experiment (not to mention an overall improved User Experience). Now sure, you can use the code editor in Optimizely to write custom code, but that's just one more thing you will need to download and run rather than just having the code run right on your site.

And while, with some considerable effort, you can keep your page load times pretty quick with Optimizely, is it really worth spending all that time fighting a broken model when you could just bypass all the run-around and have everything take place right on your site? That's certainly for your team to decide, but it's important to have the conversation before committing to a tool that can require such a level of effort just to avoid a poor UX.

Can We Do Better?

You may think that, since Optimizely is the most popular tool for A/B testing with more resources than they know what to do with, that they must have an, um, optimal implementation of client-side A/B testing and that's as good as it's going to get. I submit that this is not the case, and that there are several improvements that can be made to decrease the overhead and inherent UX issues, resulting in Engauge's superior process for running an A/B test.

1) Don't download all the things

Instead of asking the server for a comprehensive list of experiments on each page request, why not just write the experiments you need right inside the page. This may be done in a script tag, an external js file hosted with the rest of your content, or along with other rendering code in an SPA. Now, instead of calling Optimizely to find out which experiments need to run on a page, the page itself contains all the information, and theres just that much less waiting that needs to happen before a decision is made and the DOM is updated.

2) Randomly assign users to buckets

In a similar vein, Engauge randomly assigns users to a bucket when they hit the page. Now we don't have to download and parse information about each experiment to decide where to put the user. This is critical for the first time user experience, as they have to wait for this decision to happen to see the UI corresponding to their bucket. It's probably a wash the second time around since the bucket selection has been stored in a cookie, but there may not be a second visit if the user has to wait too long to see what they came for. The obvious drawback here is not having a true 50/50 split, but the randomly filled buckets should be very close to 50/50 which is good enough to draw meaningful conclusions.

3) Let the developer control the UI changes

Rather than downloading an external list of DOM changes, Engauge lets you change your UI however you want by providing a callback function that runs once the user is put into an experiment bucket. One obvious benefit of this approach is that these changes stay in your source control repo along with any other code changes. This helps prevent a whole class of bugs that are hard to find during development such as not picking up a class name change. It also lets you decide when it is best to hide an element before making a change, or if it's below the fold to just swap it out on the fly. You can swap out or add css files for each experiment, and really take control over each experiment in a way that just isn't possible with web-based experimentation platforms.

By following these three principles, we can empower our users to run meaningful experiments without sacrificing page load speed or User Experience. We hope this was a valuable look into some of the common practices in setting up an online A/B test, and the effects those practices have on performance and UX. Please reach out to us on twitter @engaugeab if you have any questions or comments, we would love to hear your feedback. Thanks and happy testing!