A/B testing is the fairest way to settle questions like “should the video sit higher in the article?” or “how often should ads appear in the feed?”. It is also easy to get wrong. A badly designed test gives a confident answer that turns out to be noise, and decisions made on noise can cost more than no test at all.
Here is how to run video tests you can trust.
Decide what success means before you start
Pick one primary metric for each test and write it down before it begins. For video experiences, useful primary metrics include:
- Revenue per session for monetization tests.
- Swipe-through depth or watch time for content and experience tests.
- Open rate for tests of placements and entry points.
Track other metrics as guardrails. A change that raises revenue per session but cuts session length or return visits needs a closer look before it is rolled out.
Split readers, not pageviews
If a reader sees version A on one page and version B on the next, you are not testing either version properly. Assign each reader to a group and keep them there for the length of the test.
Change one thing at a time
When a test changes the placement, the first clip and the ad ratio at once, a winning result tells you nothing about which change mattered. Test one variable, or use a design built for several variables at once and enough traffic to support it.
Run it long enough
Short tests are the most common mistake. Traffic on publisher sites changes by day of week, by time of day and with the news cycle. A test that runs from Monday to Wednesday misses weekend behaviour entirely.
As a rule of thumb, run tests for at least one full week, preferably two, and longer for changes whose costs appear slowly, such as ad frequency, where the effect on return visits takes time.
Beware of novelty
A new placement often performs well at first simply because it is new. Readers notice it, tap it, and then stop noticing it. If a variant wins early, keep watching to see whether the advantage holds.
Check that the difference is real
Small differences between groups often come from chance. Before declaring a winner, check that the difference is larger than the normal variation you see between two identical groups. A simple way to learn what normal variation looks like is to run an A/A test first: two identical versions, where any difference is noise by definition.
Watch for side effects
Changes in one part of the page affect others. A more prominent video module might reduce clicks on related articles; a higher ad ratio in the feed might reduce display revenue elsewhere if readers leave sooner. Look at the whole session, not just the module you changed.
Keep a record
Write down what you tested, why, for how long, what happened and what you decided. Over time, that log becomes one of your most valuable assets: it stops the same debate from being reopened every quarter and helps new team members learn what works for your audience.
Tests worth running on a vertical video experience
- Entry point position in the article.
- Poster image versus short muted preview on the entry card.
- Which clip opens the feed: the one related to the article or the most popular clip in the category.
- Ad ratio in the feed.
- Formats enabled: full-screen video only, or a mix including display and a unit below the player.
Built-in A/B testing for layouts, placements, formats and ad settings is part of our vertical player, because every audience is different and the right answer is the one your readers give you.
