Shopify Rollouts: How Native Split Testing Actually Works on Shopify

What Is Shopify Rollouts?

Shopify Rollouts launched in January 2026 as part of the Winter '26 Edition update. It's a native split testing and staged deployment feature built directly into the Shopify admin, accessible via Markets > Rollouts. No third-party apps to install, no external scripts injected into your storefront, no additional monthly cost. You get a native testing framework as part of your existing Shopify subscription.

The technical difference matters for performance. Rollouts runs server-side, meaning the split happens before the page renders to your visitor. This is fundamentally different from client-side testing tools like Shoplift or Convert, which inject JavaScript after the page loads. Server-side execution means zero page speed impact and no flickering. Visitors never see a flash of the wrong version before the correct one loads. In our experience building and optimising Shopify Plus stores, page speed directly affects both conversion rates and SEO rankings. Native server-side testing eliminates that trade-off entirely.

Rollouts also pairs with SimGym, Shopify's AI simulation tool introduced in the same Winter '26 Edition. SimGym uses AI shoppers trained on billions of real commerce sessions to predict how your changes will affect conversion rates before you go live. Together they form a two-stage validation pipeline: simulate your change with AI to estimate impact first, then confirm with real traffic using Rollouts. This approach reduces risk by catching poor changes before they ever reach your actual customers.

How Shopify Rollouts Works (Step by Step)

Creating and Configuring a Rollout

Start by navigating to Markets > Rollouts > Create rollout in your Shopify admin. Give your test a descriptive name like "Homepage Heading Test" or "Collection Grid Redesign", then select your published theme to create a variant from. Set your start and end dates. You can schedule a rollout to launch at a specific time in the future or deploy it immediately.

One powerful feature that often gets overlooked is market-level targeting. If you use Shopify Markets for international sales, you can run tests in one specific market without affecting others. Want to test a homepage redesign in your US market while keeping your UK store unchanged? That's straightforward with Rollouts. This lets you optimise for regional preferences without the complexity of managing separate storefronts.

Setting Your Traffic Split

The next critical step is setting your traffic allocation. Click the traffic allocation toggle and use the slider to decide what percentage of visitors see your variant versus the control (your current live theme). We recommend starting conservatively at 10 to 25% of traffic to catch any issues before scaling up. The slider displays estimated visitor numbers based on your store's actual traffic patterns, so you can gauge roughly how many people you'll be testing with before committing.

Visitors not in the test group see your store exactly as it is now. Shopify maintains visitor consistency throughout the test, so the same person always sees the same version on every visit. This consistency is crucial for reliable results. If a visitor saw version A on Monday and version B on Wednesday, your data would be meaningless. The system handles this automatically.

This is also where the Advanced plan requirement becomes relevant, which we'll cover in detail shortly. For now, understand that while you can create and schedule rollouts on any Shopify plan, accessing the split testing and analytics features requires an upgrade to Advanced or higher.

Making Your Theme Changes

With your rollout configured and traffic split set, click through to edit the theme. You'll see a clear banner at the top of the theme editor confirming you're editing the rollout variant, not your live store. This visual indicator prevents the common (and costly) mistake of accidentally modifying your production theme.

Inside the editor, make your changes exactly as you normally would. Update hero images and heading text, restructure section layouts, reorganise your navigation, redesign call-to-action buttons. Everything you change applies only to visitors in the test group. Your live theme remains completely untouched. When you're satisfied, save your edits. The rollout will go live at your scheduled time, or immediately if that's what you chose.

The separation here is the key benefit. You're not risking your live store while making experimental changes. The variant editor is completely isolated from your production theme, giving you the freedom to test bold changes without any downside if they don't perform.

What You Can Test with Rollouts

Rollouts covers the majority of theme-level testing needs. You can test your homepage hero section and full page layout. Redesign your navigation and menu structure. Experiment with collection page grids, product card designs, and filter placements. Restructure product page layouts, including where reviews, images, and pricing information appear. Create different content block arrangements. Test seasonal campaigns and promotions. You can even run full theme redesigns if you want to compare a completely different visual approach against your current store.

The market-level targeting capability deserves extra emphasis. If you operate internationally using Shopify Markets, you can run A/B tests per market. Test a layout change in your Australian market while your European store stays unchanged. This is particularly valuable when you have regional preferences to account for, or when you want to validate a change in a smaller market before rolling it out globally.

Beyond A/B testing, Rollouts excels at scheduled deployments for seasonal campaigns. Schedule your Black Friday redesign weeks in advance, set it to go live automatically at a specific date and time, and configure an automatic rollback when the campaign ends. Your theme reverts to the previous version without any manual intervention. No more scrambling to update your store at midnight during your busiest sales periods.

What Rollouts Can't Do (Yet)

Understanding the limitations is just as useful as knowing the capabilities. Rollouts currently cannot test product prices or discounts. If you need to A/B test pricing strategies, you'll need a dedicated pricing tool like Intelligems. Checkout and post-purchase page testing is also not available. Testing payment page variations or post-purchase upsell messaging still requires third-party solutions.

Rollouts cannot modify Liquid template files directly. You're working at the section level within the theme editor, not editing underlying template code. If you need to test custom code changes or alter fundamental template logic, you'll need a developer-focused approach or a third-party tool that supports code-level modifications.

Audience segmentation is not yet available. You cannot target tests to new visitors only, returning customers only, or segment by device type (mobile vs desktop). Traffic allocation is randomised across all eligible visitors in your chosen markets. Custom conversion goals are also missing. Rollouts tracks standard Shopify analytics (conversion rate, average order value, revenue) but won't let you define custom metrics like add-to-cart rate or scroll depth.

App embeds and third-party widgets cannot be tested either. If you use product recommendation apps, loyalty programmes, or custom widgets, testing their placement or behaviour requires those apps' own testing features. Global theme settings like typography, spacing, and colour palettes apply to both the control and variant versions. If you need to test a completely different visual language, you're limited by these inherited settings.

These are real constraints, but frame them in context: this is version one. Shopify has signalled plans to expand Rollouts' capabilities over time. For now, it covers theme-level visual changes, which handles roughly 80% of what most stores need to test on a regular basis.

Shopify Plan Requirements: What You Actually Need

This is the section most articles about Rollouts get wrong, so we'll be precise.

Rollouts as a scheduling and deployment tool is available on all Shopify plans: Basic, Shopify, Advanced, and Plus. You can access Markets > Rollouts on any plan, create rollouts, schedule theme changes, and deploy them automatically at specific dates.

However, to actually split traffic between a variant and your control version (to run it as a real A/B experiment and access analytics showing how each variant performs) you need an Advanced plan or higher.

This distinction is critical. On Basic or standard Shopify plans, you can use Rollouts for scheduled deployments, but you will not get the split testing capabilities or experiment analytics. You cannot adjust the treatment percentage to run a true A/B test on these plans. The traffic allocation slider and experiment data are locked behind the Advanced tier.

If you're considering upgrading specifically for Rollouts' testing capabilities, review our complete breakdown of Shopify pricing tiers to understand what else changes at the Advanced level and whether the upgrade makes commercial sense for your store.

Rollouts Analytics and Reading Your Results

Once your experiment is live (on an Advanced plan or higher), Rollouts tracks three core metrics: conversion rate, average order value (AOV), and total revenue. Each metric displays a confidence interval chart showing the performance range for each variant. Rather than a simple "variant B converts at 2.4%", you see the statistical range: "variant B converts between 2.1% and 2.7%, while the control converts between 2.0% and 2.4%." This gives you a far more honest picture than a single percentage ever could.

Statistical significance matters here. You need sufficient traffic for reliable results. A rough baseline is 1,000 visitors per variant as a minimum. If your store gets 10,000 monthly visitors, a two-week test gives you solid data. If you see 300 daily visitors, a 7-day test gives you roughly 1,050 per variant at a 50/50 split. That's tight but workable. For stores with lower traffic, extend your tests to 14 days to allow for traffic variance across the week.

Weekly traffic patterns affect results more than most merchants realise. Monday visitors often behave differently from Saturday shoppers. A 7-day test captures one full cycle. We recommend running for 14 days where possible, giving you two complete weekly cycles to smooth out anomalies. And resist the temptation to stop tests early. Seeing a promising uplift after three days and rolling it out immediately is one of the most common testing mistakes. Commit to your planned duration and let real patterns emerge before making a decision.

Rollouts vs Third-Party A/B Testing Apps

For a detailed comparison of testing tools currently available, read our guide to the best A/B testing tools for Shopify.

Rollouts handles the majority of theme-level testing needs at no extra cost. So when should you keep a third-party tool, or invest in one if you don't have it?

Use Rollouts when you're testing theme-level visual changes, scheduling seasonal campaigns, want zero page speed impact from your testing setup, need basic split testing without adding monthly costs, or are getting started with experimentation for the first time. If your store has never run an A/B test before, Rollouts removes every barrier to entry.

Keep or adopt a third-party tool when you need to test product prices or discount structures. Or when you want to run checkout and post-purchase page experiments, segment audiences by visitor type or device, modify Liquid templates directly, track custom conversion goals beyond standard Shopify metrics, run multivariate tests (testing multiple changes simultaneously), or need enterprise-grade statistical analysis with detailed reporting.

Popular alternatives in the Shopify ecosystem include Shoplift, Intelligems, Convert, and Optimizely. Each handles specific testing scenarios that Rollouts doesn't yet cover. These tools typically cost between £50 and £500+ per month depending on traffic volume and feature tier.

We've seen a hybrid approach work well for many of our clients at Charle. Use Rollouts for homepage, product page, and collection page layout variations. Use a specialist tool for pricing tests or checkout optimisation. This keeps costs down while giving you comprehensive testing coverage across your store. If you're looking to build a structured testing programme, our team can help. Learn more about our CRO agency services or explore our dedicated split testing services.

What to Test First: Practical Ideas

If you're new to A/B testing or just getting started with Rollouts, begin with high-impact, low-risk changes. For a deeper dive into testing methodology, read our complete guide to A/B split testing.

Homepage hero section. Test lifestyle imagery versus direct product photography. Try different headlines or call-to-action button copy. The hero section often has an outsize impact on conversion rates because it's the first thing every visitor sees. A stronger hero can lift performance across the entire store.

Navigation structure. Test a traditional dropdown menu against a mega menu with more categories visible upfront. Try reordering your main navigation items. Does leading with your top-selling category improve navigation flow and reduce bounce rate?

Collection page grid. Test three-column product grids versus four-column layouts. Test filter placement with sidebar filters versus top-of-page horizontal filters. Larger grids can improve browsing speed, but sometimes smaller grids with bigger product images drive more clicks. Rollouts lets you measure the actual impact rather than guessing.

Product page layout. Test moving your image gallery to a different position. Try placing customer reviews above the fold versus further down the page. Small layout changes on your product pages can sometimes yield surprising conversion improvements because these pages carry the most purchase intent.

CTA button styling. Test button colour, size, and copy variations. "Add to Cart" versus "Add to Bag" versus "Buy Now" might seem trivial, but we've seen these changes move the needle across client stores. Buttons are the final action point before a conversion, so even small improvements compound over time.

Implement a progressive rollout strategy rather than going straight to 50/50. Days 1 to 3: run at 10% traffic to catch technical issues and confirm nothing is broken. Days 4 to 7: expand to 25% traffic and start gathering meaningful data. Days 8 to 14: increase to 50% traffic to confirm trends hold with a larger sample size. Day 15 onwards: either roll out to 100% of traffic or roll back entirely. Your decision should be data-driven by this point.

This progressive approach balances risk and learning. You catch problems early with minimal exposure, validate your results with increasing confidence, and then commit to a decision backed by real data. For more strategies on driving performance, see our guide on improving your ecommerce conversion rate.