A/B Testing lets you compare two quizzes and see which one performs better with your store visitors. You can test different quiz designs, question flows, results pages, offers, or recommendation strategies by splitting traffic between two quizzes.
This feature helps you understand which quiz gets more impressions, completions, conversions, revenue, and average order value.
Before You Start
Before starting an A/B test, create the two quizzes you want to compare.
For example:
Quiz A can be your current quiz, or the version you already know performs well.
Quiz B can be a new version you want to test, such as a quiz with different questions, a different design, or a different results page.
Make sure both quizzes are published and ready before starting the test.
Start an A/B Test
To start an A/B test, go to the Quizzes page.
Under the list of quizzes, you’ll find the A/B Testing section.
From here:
Select the quiz you want to use as Quiz A.
Select the quiz you want to use as Quiz B.
Choose the traffic split.
Click Start A/B Test.
By default, A/B tests use a 50/50 traffic split. This means visitors have an equal chance of seeing Quiz A or Quiz B.
You can also choose a different split. For example, you may want to send 90% of traffic to Quiz A if it is your proven quiz, and only 10% of traffic to Quiz B if you are testing a new version.
How the Traffic Split Works
When an A/B test is active, visitors are randomly shown either Quiz A or Quiz B based on the traffic split you selected.
For example, if your split is 50/50, each visitor has a 50% chance of seeing Quiz A (Personalised Nutrition) and a 50% chance of seeing Quiz B (Personalised Nutrition animation welcome).
PLEASE NOTE: This does not mean the quizzes will alternate perfectly between A and B. The quiz shown is selected randomly based on the traffic split.
Which Quiz Should You Add to Your Store?
You can use and add either Quiz A or Quiz B.
When the A/B test is active, Lantern detects that the quiz is part of an active test and randomly shows either Quiz A or Quiz B based on your selected traffic split.
If you’re using an URL to link to the quiz, the URL shown in the browser may stay the same, even when the visitor is shown the other quiz in the test.
View A/B Test Results
After the test starts, you’ll see a test card with side-by-side results for Quiz A and Quiz B.
The A/B test card shows:
Impressions: The number of times each quiz version was viewed during the test.
Completions: The number of quiz takers who completed each quiz version.
Completion rate: The percentage of impressions that became completed quizzes.
Conversions: The number of purchases recorded from each quiz version.
Conversion rate: The percentage of impressions that led to a purchase.
Revenue: The total revenue recorded from each quiz version.
Average order value: The average order value from purchases recorded for each quiz version.
The card may also show a Winner label to indicate the quiz version that is currently performing better. This is based on the Conversion metric.
These stats are specific to the time period when the A/B test is running.
PLEASE NOTE: Some stats, such as conversions, revenue, and average order value, may take a short time to update because Shopify reporting data can have a small delay.
Stop, Restart, or Delete an A/B Test
You can stop, restart, or delete an A/B test.
Stop a test
Stopping a test pauses the traffic split. Visitors will no longer be randomly sent between Quiz A and Quiz B. The test stats are preserved, but the A/B test stops recording new data.
Restart a test
Restarting a test starts the traffic split again using the same Quiz A, Quiz B, and test setup. The existing stats are kept and the test continues recording new data once it's active again.
Delete a test
Deleting a test removes it from the A/B Testing section. PLEASE NOTE: Deleting a test also deletes its A/B test data. Only delete a test if you no longer need the results.
Run Multiple A/B Tests
You can run more than one A/B test at the same time, as long as you are testing different quiz combinations.
Each test will have its own card and results.
Tips for Better A/B Testing
For clearer results, test one major change at a time. For example, you can test a different welcome page, a different quiz flow, or a different results page.
If you change too many things at once, it may be harder to know which change affected the results.
We also recommend running your test long enough to collect enough impressions, completions, and conversions before deciding which quiz performs better.