The importance of high velocity testing

guillaume-jaillet-403241-unsplash.jpg

I’m a big proponent of high velocity testing, and believe the more tests you run the more effective you are. This doesn’t mean that these tests are throw it at the wall kind of tests, they still need to be grounded in research with a decent hypothesis that you validate, but the more you run the more effective your testing programme will be:

  1. the more you test the more you learn

For every test you want a falsifiable hypothesis. This gives you the ability to achieve a learning whether your test wins or not. More learning increases your chance of your test being a winner the next time around.

2. the more tests you run, the more winners you will get

I know its dumb to say, but increasing volume means that you will run more tests, more tests mean you will get more winners (overall), although maybe not as a percentage of tests run. There is no point in running a testing program where you run a test a month and get a winner every two months. That’s dispiriting for the team and doesn’t add value to your bottom line.

3. wins are compounded

Four winning tests with a 5% uplift don’t work out to be a 20% uplift, but the wins are compounded,and it works out to a 22% uplift.

Your processes should remain similar in order to achieve this:

  • look at the slots you have on your site to fill (e.g. for SAAS you’ll have the homepage, signup page, first user experience funnel, payment page etc)
  • ensure that you have a wealth of data around visitor behaviour and challenges, you need to be able to generate a lot of hypotheses
  • Generate these hypotheses, wireframe and get tests ready for each slot (you need to have one or two ready to go in order to be able to switch out quickly)
  • Be ruthless with your testing, don’t be afraid to kill things that don’t look like winners (the other perspective with this is that your hypothesis may be right, but the execution could be wrong, further iteration may be required)
  • It can be useful to try smaller developmental test ideas in here (not red button/green button tests), but tests that are more value proposition/microcopy focussed that don’t require a lot of development

Although it takes a lot more resources in order to increase your testing velocity, the results are worth it in terms of wins and learnings.

Amazon’s dark patterns (and a light one)

As I’m South African based I don’t shop on Amazon that frequently so its always interesting to look at how they manage their site. I know they do a lot of testing, in 2011 they were doing 7000 tests a year and the richest man in the world said:

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day…” – Jeff Bezos, CEO at Amazon

I’m always surprised at how aggressively they try and upgrade you to Prime. On my most recent purchase you can see the interstitial they used to try and upsell me. In order to not take the Prime offer I have to use an unobtrusive text link rather than the obvious yellow button.

screen shot 2019-01-20 at 10.04.57

I would be worried about all of this dark pattern Prime upselling on the customer experience. I think it’s a little sleazy and underhand but at the same time it doesn’t dissuade me from buying from Amazon (and if you have Prime you stick around forever, so its great for retention). If you look at the last time I purchased from Amazon and somehow signed up for Prime they had some strange ways to try and make me stay.

 

doavk12w0aermvl

Things I find notable about this:

  • making people click “I do not want my benefits” to cancel
  • making cancellation the least intuitive of four buttons on the page
  • the phrase “Unlimited One-day delivery: Direct to your door” has so many great benefits
  • Red vs green type (and the fact that its £0.00 not £0)

Finally here is a nice piece of UI from Amazon for products I’ve previously purchased. It messages that I last purchased it and when. Its easy and allows me to shortcut choosing my product, which at least for me reduced any choice friction and hastened my conversion.

screen shot 2019-01-20 at 09.54.42

Sign up page case study- +14% conversion

As part of the Now Novel first user experience we are constantly testing the signup process. You’ll see our thinking from a few years ago here.

We arrived at the last version of our signup page through testing, but I thought it wasn’t great. There is a lot of explanatory content on the page, but its mostly superfluous and distracting. Rather than trying to funnel people into conversion it is doing the job that a page further up the funnel should be doing. It’s more like a landing page than a pure signup page. Don’t get me started on the FAQ that i think are actually causing doubt more than countering objections.

 

control

 

Looking at heatmaps we could see that people weren’t scrolling the page, there was a large amount of drop-off seen in GA and there were a lot of concerns about privacy and whether or not people had to pay that we got from analysing single question survey feedback.

In terms of evidence to change the page we used the following:

  • Most people were coming from the homepage, so had an understanding of the service (which means that all of this additional information was superfluous).
  • We have a lot of information about the benefits from paying users using long form surveys that we could incorporate on the page.
  • We had jobs-to-be done research and understood what role we are employed for.
  • Single question surveys and customer interviews gave us an idea about the concerns people had (concerns were mostly around their ability, cost and whether it would work).

This allowed us to put together a page that catered to countering objections, a headline that gives an expectation for the future on a short page.

screencapture-localhost-3000-users-register-2019-01-28-10_15_05

The above version of the page we ran for a month and we had inconclusive results. The feedback from visitors that we got from single question surveys on the page was that they were worried about:

  • whether they had to pay
  • if they could trust the service

With our next incarnation of the page we iterated a little further.

We cleaned up the expectation around “What do I get?” a little, but the major change was that we added a large compelling testimonial to help give people reassurance that this was a trustworthy service.

test

This test was a wholehearted success with a 14% increase in conversion at a 93% significance.

The major learning it gave us was that a strategic testimonial can give a lot more trust and increase conversion. We’ll be testing this in other areas of the site.

3 product development tips for 2019

Last year I had some good results with some of the (many) product development tactics I  tried, and thought I would share as with CRO tips I shared.

  1. With Now Novel we consistently try to understand and test the optimum first user experience, and we spent some time this year trying to perfect our onboarding flow.  One of the first thing we looked at was understanding the product through a Jobs-to-be-done  lens. Intercom have written a lot about this and their book is fairly useful. We undertook more surveys, spoke to customers to understand their motivations and intentions a little better. What we found was that there were three distinct groups:
  • people who wanted to start writing
  • people who had something and wanted to progress, and
  • people who wanted to learn about the craft of writing

Armed with this information we could construct an onboarding flow that catered to our prospective client’s job-to-be done and so improve their satisfaction by exposing them to a relevant part of the product and ultimately increasing conversion. Very useful in this was Samuel Hulick’s onboarding book (you may recognise him from his zany first time onboarding teardown’s).

The above image is a powerful distillation of that book (and the idea of onboarding)

2. Actually asking for the sale is something I have forgotten to do enough of. You think its implicit that your product is for sale and that you would be guiding people to purchase, but with freemium there are a lot of opportunity to reiterate and improve your selling. Reading this article was illuminating in that it showed all of the options for cross selling and a methodical process for achieving it. We’ve started a process of integrating nudges (step 1 is a simple header for encouraging unpaid members to purchase) which has accounted for 13% of all sales since we launched it. We have a few more planned for very soon, the one I am most interested in is the freemium pop-up: as you reach the border of the freemium offering you get a pop-up that outlines the benefits of paid membership. The line that needs to be walked is how many of these can you use that don’t piss your audience off too much, whilst encouraging them to convert (great copy and a clearly communicated value proposition is key here).

3. Pricing is often seen as more art than science, we undertook some research to add some science to it. We asked about the benefits of our product and the pricing that would be:

-too expensive

-getting expensive

-a good deal

-too cheap

and then plotted all of the responses on a graph.

screen shot 2019-01-23 at 05.45.29

the area between all lines is the range where you can price (in the above example between $27 to $43). This gives you a much more informed perspective about where you can price your product.

 

 

 

 

3 CRO tips for 2019

I was reviewing 2018 and the tests that were successful for me to see if there were specific areas that I could recommend for testing. I wouldn’t suggest that you simply roll any tests out without researching your audience, as with all CRO it is important to focus on your customer behaviour on your site rather than best practises. If research were to point in this direction I would recommend exploring these approaches.

1. Across numerous split tests I conducted this year I saw the value of  header testing. There are a number of ways to trial this:

  • mapping sitewide navigation to user monetisation and intent (this is a perennial winner for me)
  • showcasing value propositions more effectively
  • showcasing communication options (e.g. customer service numbers) more clearly
  • size of search bar and its relative prominence
  • sticky header

With regards to the sticky header this could be used to add a consistent site search (very useful for ecommerce), key navigation or even to reiterate a value proposition. Here you’ll see how we’ve used it on the Now Novel blog to keep the key messages front and centre (this doubled content marketing signups when we integrated it)

Screen Shot 2018-12-27 at 15.01.16

 

2. The power of iteration in testing. Having a hypothesis backed up by evidence is crucial for your testing process, but its only by validating your assumptions through testing that you’re going to see improvements. However once you have a result (win/lose) that isn’t the end of the process, the most important part is the validated learning that comes from this, and how it informs future testing.

If it wins then how can you exhaust the hypothesis further? How can you make your test more extreme? By running follow up tests you can really start to learn more about the problem.  Similarly if it loses, how could you tweak your execution to validate the hypothesis (my record is 8 iterations on a product page test before I got a win, luckily a double digit increase in revenue per visitor)

3. The importance of value proposition testing. Value proposition is often thought of as this nebulous marketing concept when its actually a tangible testable piece of communication. The definition I like the most is: ” In the mind of your ideal prospect how are you differentiated from your competitor”. Through research and competitor analysis you can find which attributes are unique to you and resonate with your user.

With iterative testing (see above :)) you can find out which messages are impactful on which pages of the site and test and improve on these. Value proposition testing isn’t a silver bullet immediate winner, unlike fixing a UI, it will take some iteration as you finesse your copy and placement. Similarly different pages will have different challenges (i.e. visitor concerns on the basket page are different to concerns on a product page), but by testing you will uncover the right message for the right position.

The benefits of value proposition testing are huge because once you uncover a lever that shows how you are different you can roll it out onto different channels and all your acquisition and retention activities will benefit.

Books read in 2019

13. The Three-Body Problem- Cixin Liu

I really enjoyed this, even though it was quite impenetrable and difficult to read. There was a lot of science and the story was complicated. It was written in Chinese and I could see that it had a different cultural focus to the usual western perspective. I was a big sci-fi fan in my youth (Heinlein, Asimov) and it was interesting to read something with a similar but different perspective.

12. A moveable feast- Ernest Hemingway

In the age of “me-too” and toxic masculinity its hard to know how to take Ernest Hemingway with his hunting, boozing and bullfighting. I read “For whom the bell tolls” a couple of years back and loved it. I also reread this long read from the New Yorker again this year. Its great and shows how life can play out for you when you love champagne and are Ernest Hemingway. This story was great. It had a real flavour of Paris about it, and a yearning for being young and unencumbered by life.

 

11. Another Kyoto- Alex Kerr

This was an interesting book to read. The format was unlike anything I’ve read before. It had different segments for e.g. walls, mats etc; what they meant culturally and what they looked like as a result of this. It had great pencil illustrations between the pages that brought the concepts to life. The best thing I learned was the concept of shin, gyo and so which deals with the level of detail and abstraction (this is across pottery, painting, bonsai).

shingyoso_shodo

10. Dance, dance, dance- Haruki Murakami

I hadn’t read any Murakami for a few years and this was in the Airbnb we’re staying at. I had to go on a long distance flight so I thought I would treat myself. I loved it. There are always numerous strands of stories that have a magical realism angle to it (but he’s not as miserable as Paul Auster). I like the details of real life; food, drink and music playing that anchor you even if there are sheep men around.

 

9. Man’s search for meaning- Viktor Frankl

This slim book is a fairly harrowing read. It talks about his experiences in the Nazi concentration camps and his quest to understand what kept him (and others) alive during those tough times. The book is split into two parts, the beginning tells of his experiences in the camps and the second half is about his theory of logotherapy (to live meaningfully we need to respond meaningfully to life’s challenges). The second half didn’t resonate as much as the first, but I would wholeheartedly recommend the book.

8. Other voices, other rooms- Truman Capote

This was Truman Capote’s first book (apparently written when he was 23). It’s a coming of age story of a 13 year old boy whose mother dies and he is sent to live with his father in the middle of nowhere. The characters are well sketched, but as a curiosity, not as someone you actually like and empathise with. Entertaining, but not on the “read again” list.

 

8. Tools of titans- Tim Ferriss

I’ve read most of Tim Ferriss’ books since the 4 hour work week and listened to a good few of his podcasts. He is an earnest if somewhat irritating character. This summarised his podcasts and interviews and offered up some pithy insights on how to live your life from various famous people.

A useful summary of what people do:

  • meditate
  • don’t eat breakfast
  • use the chilipad for cooling at night (??!??)
  • some book recommendations (Man’s search for meaning, Influence, Poor Charlie’s almanack)
  • listen to songs on repeat for focus
  • create work on spec and then sell it
  • turn weaknesses into strengths
  • failure is not durable 

7. A supposedly fun thing I’ll never do again- David Foster Wallace

I’ve tried to read Infinite Jest a couple of times and got bogged down in his prose. All of the detail and footnotes and breaking down the fourth wall can get a little tiring. I really enjoyed some of the stories in this selection of his articles (specifically the cruise ship and Illinois state fair) and struggled with others (Lynch and television). His writing is amazing and infuriating in equal measures, but I would recommend this. (and I know it may be a cliche, but his This is water commencement speech is inspirational and sad in equal measures, especially as a result of his suicide).

6. Autobiography- Morissey

My 12 year old self would never have forgiven me for not reading this. Unfortunately the older Morrissey is less of the sad songs with bouffant hair with the gladioli in his back pocket and more of a curmudgeonly old racist. It was interesting to dip in and out of it. His tone is one of always being the victim (which is actually quite entertaining) and his fixation on the damage that school caused him was tiring.

5. The Book of Genesis- Robert Crumb

I’ve been a big fan of Robert Crumb’s cartoons for many years. The documentary about his life was very illuminating to show where his vision came from. It was interesting to see one of the most classic books of the bible illustrated in his style. Still slightly pervy (it is Crumb after all) but beautiful illustrations. I wish he had taken a slightly newer translation, all of the people’s names and sons of as well as the language was a bit dry, but still a good read.

4. Music for chameleons- Truman Capote

This was only the second Truman Capote book that I had read (after “In cold blood”, which I really enjoyed). There were a few short stories, and I wholeheartedly recommend this. The central story is another true crime story (which unfortunately may have a little artistic license take with it), but is gripping nonetheless. He has a way with creating characters that are truly alive, and he ends some of these stories without a clear conclusion, but the ending still makes you think and is satisfying.

3. Surely you’re joking Mr Feynman- Richard Feynman

Very interesting autobiography of Renaissance man Richard Feynman: Nobel prize winner, bongo drum player and fan of the ladies. Very inspirational story of his way of approaching a problem. He teaches himself to paint using the same way he approaches science. He is intensely curious, and not afraid to question the status quo.

2. Atomic Habits- James Clear

I thought I would enjoy this book, and I loved it. I’m a big fan of trying to understand habit formation, to try and improve my own habits as well as for Now Novel to help people build a sustained writing practise.

I’m waiting for BJ Fogg to bring out a book on this, his theories of behavioural change and the power of tiny habits are echoed in this book, but this is well worth reading as it gives simple actionable steps to understand habits, decrease bad habits and set new habits in place.

A couple of things that really resonated with me:

  • Behavioural change has 3 layers: outcomes based, processes and identity. Identity is the strongest method, as you don’t want to be lying to yourself (i.e. there is a huge gulf between saying: “I’m trying not to smoke” which is a process vs ” I’m not a smoker” which is your identity). Identity is more powerful
  • Like BJ Fogg he talks about habit stacking. Attaching a new (extremely small) habit to an existing behaviour (e.g. before you brush your teeth, floss one tooth) Easy to do and no one only flosses one tooth once they’ve started. He compounds this by adding rewards (to make it stick) and stacking multiple habits so that it flows together
  • Ensuring that new habits are obvious, easy, attractive and satisfying (it sounds simple, but is more difficult to achieve)

I thought the book was very actionable and the layout was great too, a lot of reiteration of the key methods. Pretty soon I’m going to end up with a morning routine like Mark Wahlberg’s

_103392798_39a75c55-64ec-4f74-988b-6adc067ab562

 

  1. Blood, bones & butter- Gabrielle Hamilton

I nearly put this down after reading the first 10 pages (and sporadically throughout the book), however I did end up finishing it.

She describes her unorthodox upbringing and life and how it maps to her love of food. From her childhood foraging with her French ballet dancer mother, through living in France and Greece and summers with her Italian mother-in-law. All of these impact the restaurant Prune she opens.

Her descriptions of what she ate are interesting, the inter-personal interactions less so.

 

I’m back!

10 years after my last blog post I thought it was time to revive my blog. Before I move onto a more monochromatic theme I wanted to show you the blog header I previously had:

image (14)

Magical…

Combined with a web2.0 tag cloud it is a sight to behold (unfortunately retired onto blogspot for posterity now)

3 little known (but effective) tools for understanding your user

(i wrote this post for Ventureburn recently)

If you’re like me, you’ve always been fascinated by startups but not sure how to go about actually starting one up. At the beginning of this year I noticed a gap in the online market and started working on filling it by creating an online novel-writing programme for people who really want to write a novel, but don’t know how to start it.
I’ve been working for a large corporate in South Africa and in the Netherlands, so it’s been fairly eye opening having to do things myself (not having a team of people to execute on things for me), as well as focusing on the user for requirements — not strategy that needs to be approved high-up and may not be particularly appropriate for the challenge at hand.
I have been trying to follow the lean startup and customer discovery methodology. In addition, I have been chatting to many people smarter than me, and spending an inordinate amount of time on startup blogs. In chatting to people I received three very useful processes that I didn’t see mentioned anywhere else, and I thought it would be good to share with you, so that you can ensure you’re building a product that your users want, rather than what you think is a cool solution.
1. Questionnaire design
We had an online questionnaire that we got people to fill in and I also conducted many in-person discussions with aspiring writers to try and understand what their problems were. Some of the most useful questions were things like:
“What are the most frustrating aspects of…”
“What is the process you go through today in order to…”
“Why is … frustrating?”
“If you had a magic wand to help you, what would it do?”
“How much would this magic wand cost?”
A great way to start is to ask people a question about the beginnings of their passion for this particular topic (e.g. When did you first start writing was our question). This loosens them up and makes them less skittish before you start asking them further questions.
2. Dump n’ sort
Once you have all of the answers above you’ll have a large amount of issues that people have with your topic. Individually look at these and write what you think their problems are on individual pieces of paper. I used post-its. Sit down in a team and then group them into sections. Isolate which are the most important/biggest problems that the users have. Then look at the features that support the solution of that problem. This gives you a small great nugget of a product (or MVP in the words of the Lean Startup).
3. Customer verification
Now you’ve been through the first two steps and you need to know if you’re on the right track. Write down (in the first person), what you believe the user’s challenge is. Then, write down what their desire is around this, and finally you put down your solution. Our summarised version looked like this:
Problem
Writing a novel is one of those bucket list things for me, I have to write a book before I die. It’s just really hard to find the time to get it done. Life is real busy, and finding writing time is difficult – especially with a project as big as this.
Solution
What I’d like is someone to hold my hand through the writing process. Help me find and finesse my idea, show me what structure I need and help keep me motivated throughout the process.
Product
The Now Novel is an online novel writing course that helps you finally write your novel. It gives you the structure to find your idea, finesse the plot, character and setting and the motivation and guidance to write it.
Finally, sit down with your potential users and ask them to go through your document. They need to highlight what they agree and disagree with, and leave blank what they feel ambivalent about. This will then give you an idea of how you’re progressing relative to people’s desires and where you need to put more effort in. Then its back to the drawing board to begin the loop again if you have too many disagreements.

3 Online marketing tips for startups/cheapskates

If you’re like me, you’ve always been fascinated by start ups but not sure how to go about starting one up. At the beginning of this year I noticed a gap in the online market and started working to fill it, by creating a custom-made online novel-writing programme for people who really want to write a novel, but don’t know how to start it. We’re just launched NowNovel and acquiring users has been another interesting part of the story
Having left the comfortable home of Naspers, and their larger budgets, I have been contending with marketing budgets that are self funded, so I need to keep my cost per acquisition low and  user quality high.
I’m going to outline a couple of tips that I’ve found that can drive you cheap users;
1.     smart Facebook advertising
2.     Stumbleupon
3.     Content marketing
Our challenge is that we’re focused on a female US audience who has on their bucket list that they want to write a book. One of the more contested audiences to acquire on the internet. However, knowing your tight targeting is an advantage as you can then acquire users who are more likely to convert.
1, smart Facebook advertising
Facebook is  great platform (especially according to the abovementioned targeting).  One trick that I gleaned was to use highly targeted ads, with highly targeted copy. To take our example;
Our target was women, aged 25+, who had an interest in creative writing and writing in Texas. Our headline was “want to write a book in Texas”, we talked about product benefits (not features) in the ad copy and then had a strong call to action.  The fact that our copy is targeted to a specific geographic area jumps out at people and makes them feel like it is more relevant. In terms of images, try and aim for something orange (very different to Facebook’s blue), make it a person’s image (pretty girl’s always work) or make it a zany picture that grabs attention.
2. Stumbleupon
Stumbleupon has very cheap clicks for targeted traffic; we paid $0,10 for females of a certain age with an interest in literature. They also bounce a lot more (because they’re a casual user), but we got some committed users. You also have the ability to have your content go viral, which happened with us (to the tune of 14,000 free global stumbles, extra server requirements and Friday night chaos). I think it was because of our good design  and logo/icons that triggered the sharing.
3.content marketing
 Content marketing isn’t a secret, essentially what we’re looking to do is create great content in interesting formats that users will enjoy. Its what you should be doing online and not creating spammy, keyword stuffed content for search engines. If you read any of the exhaustive content on copyblogger, the kissmetrics blog or seomoz you’ll get the idea of the content you need to create (and you’ll get some good guides from the same destinations). Its relevant content that is useful for the target audience (and coincidentally search engines) When the user is thinking of converting they’ll think of your site because you are a knowledge leader in this area. We haven’t created this yet, (because its a lot of work) but watch this space.
What else have I left out that works? Let me know in the comments 
(this is a crosspost to a post that i wrote for Bandwidthblog)

Now Novel

Since i left DSTV/Naspers I have been focussing my efforts on developing Now Novel. Now Novel is a guide to writing a book; essentially a “WordPress for novel writing”. It’s a step-by-step guide on how to write a book; with an overview of how to find your idea, hone your idea till you like it, and then develop the character, plot and setting around it. It also provides you with the motivation and structure to plough through the tough grind of getting your book written.

The initial idea came from my sister who wrote a novel a few years back and had it published here and in the US. People kept on harassing her on how she did it and she created a pdf guide on how to write your novel. From this basis we’ve been creating a course that seems like it will be useful.

We’ve been following the lean startup methodology, to try and verify as many assumptions as possible before we build version 1 of the product. We’re so lean in fact that in addition to my sister and i doing the product work, i have enlisted my wife to do the development (unfortunately my kids are still too small for customer service…)

We launch version 1 in June, watch this space for more updates