As CRO specialists, most of us have at one time or another reported into a manager with significantly less experience in CRO than ourselves. And when that happens, there tend to be a number of questions that get asked consistently:
1. How can we predict ROI?
A very valid question, and yet notoriously difficult to do; of course it is possible to make estimates based on existing conversion rates, projected improvements & any associated changes in average order value but an “accurate” prediction of ROI before a CRO programme has started is highly unlikely.
As such, my preferred answer to this is that it is down to the type of testing we do and how open-minded we are willing to be; essentially, achieving an ROI is entirely within our power to achieve, as long as we aren’t going to restrict ourselves by saying that certain areas are “sacred” and cannot be touched. If we ring-fence areas, then we can expect not to make as strong a return.
2. How do we know that the results are reliable/accurate?
This is a prime moment to explain the principles of statistical significance to your boss; the “proof” of uplifts is a science, not an art, and assuming that they have a reasonable grasp of basic maths, it shouldn’t be too difficult to explain this one.
At the very worst, you can always dangle out there the possibility of a “bake-off”; this is the act of rerunning the Control experience against the winning experience again, just to prove that the results will be the same. There is always a small chance that the same results may not be seen however, so it is still vital that the statistical side of the results is explained.
3. Will we understand why our visitors do certain things?
The short answer to this is “Yes, absolutely”; the longer answer is similar to no.1, where it depends on how you test and particularly how you hypothesis and plan your tests.
If you just set up a test with different placements of buttons on a page without attempting to understand beforehand why the page may not be converting well at present, then any potential learnings will be lost. So if you can hypothesise as to why your new test experiences may affect visitors in certain ways, and prove that hypothesis across multiple tests, then you are likely to start to understand your visitors’ motivations in the longer-term.
4. Why can’t we run more tests more quickly?
Although this is at no.4, this is almost certainly the most common question of all but it does give you as the CRO specialist an opportunity to once again prove your worth.
Firstly, there is the issue of test clash that you must avoid; for the results of a test to be statistically valid, the only difference in the experiences should be what you are looking to test and therefore a visitor should not be permitted to enter multiple tests at once. This, therefore, will likely limit the number of tests you can run at any one time, as they will need to remain mutually exclusive.
Secondly, and very much tied into the first point, is that to reach statistical significance, it is likely that a test will need a certain level of views and conversions; conversions-wise, I personally favour a minimum of 125-150 per conversion point per experiment. But you have 10 tests running at once on your site, all of which are mutually exclusive, each test is only getting 10% of traffic, and therefore the time it will take to achieve the necessary sample level of data will be much greater. So is it really gaining you anything?
Third and finally is the matter of cohesion and a centralising idea. Some of the most successful tests you are likely to run will be built on the back of other test learnings. As we saw in no.3, it is most likely that multiple tests will give you the knowledge you seek around visitor behaviour and therefore churning out tests quickly isn’t always the best use of your resources; waiting for some insights from existing tests may actually serve you better in the long run.
5. Who controls the “kill-switch”?
At no.5 is another very common question and certainly not invalid. The “kill-switch” usually refers to two things:
1. How do we stop a test if we need to?
2. What happens if the optimisation tags start breaking our site?
The answer to 1. is simple and generally should exist within the user interface of whatever CRO platform vendor you use so as long as your boss understand how simple it is, you shouldn’t have any problems!
For 2., the answer can be a little more challenging, depending on their level of technical knowledge. Optimisation tags, by their very nature, are always viewed with suspicion by in-house IT teams, given that they generally don’t believe there to be anything wrong with their existing code and therefore why do things need changing? They can also take exception to not being able to control the code that is being displayed on “their” URLs. But to put their minds and your boss’ at ease, just tell them that you can implement a server-side switch to remove the optimisation tags; then at the very longest, it shouldn’t take more than half-an-hour or so before they are stripped from your site.
As you’re no doubt realised, these 5 things your boss will ask you about CRO can take some explanation; but in my experience, if you can explain them up-front, you are often freer to go about your CROing with their minds wholly at rest.