Full-Stack Experimentation
Background
The world of A/B/n testing has become one of the signature accomplishments of the era of Internet business. The fact that companies no longer have to rely solely on their gut instincts, but can instead immediately test a theory or product, is immensely powerful and has reshaped the way much of marketing is conducted.
Yet, as beneficial as A/B/n testing has been, it has largely been confined to marketing, and stayed on the surface of the software and product development process. It’s sort of like A/B/n testing has trapped us: we can see the potential for how testing and experimentation could reshape the creation and innovation cycle, but we haven’t been able to drill down and do testing at the engineering level where it could really make a difference.
Larger companies with considerable financial, engineering, and computing resources have long recognized the power of experimentation at all levels and have used these resources to test deeply how their applications and services work. While this isn’t always broadcast widely by the businesses, stories have filtered out about just how extensive this type of testing is at companies like Netflix, Google, Facebook, Etsy, Amazon, and their ilk. It’s likely that any time you use any of these services, they’re conducting a massive number of tests that are looking to address incremental, rather than just surface level, improvements to their portfolios.
This type of constant testing at all levels of a business is now being referred to as full-stack experimentation. To implement full-stack experimentation, you need agile development and continuous delivery supported in a stable system that can change and respond to the results of testing. Because the fact is, if you want to be a company that conducts a lot of experimentation, you need continuous deployment and the ability to test and make hundreds or thousands of changes a year. You also need to be able to change your development process so that you can do testing on small segments of your user base, and conduct tests that get down to the engineering and coding level of a product, rather than just the way it’s marketed. This is testing the fundamental nature of a product or process. The implications of such testing are profound and that’s why large Internet companies have used full-stack experimentation for so long. It’s also why now that new technology could make full-stack experimentation available to businesses of all sizes, companies should be thinking about how to incorporate it into their product development processes.
The mission
We’ve seen that leading Internet companies have proven that full stack experimentation is a way to make a service better for their users. These companies have built custom stacks that are ornate and can go deep and wide. They utilize stable deployment of rapid changes for real-time iteration and product improvement. They use site-wide monitoring, feature flagging, and continuous deployment to empower full-stack experimentation, which brings testing to all levels of an organization.
Our research mission is to determine how and when a company should adopt full-stack experimentation to improve the business.
Questions this research will answer
This research will seek to address the following questions:
- What does full-stack experimentation include?
- How does it work in large companies?
- How will it work differently when it is productized?
- Why does full-stack experimentation need to live in a continuous deployment mechanism?
- What is the difference between A/B/n testing and full-stack experimentation?
- How will full-stack experimentation change software development?
- What impact can full-stack experimentation have on the business?
- Which vendors offer products that support full-stack experimentation?
- What are the victories companies have already achieved?
- How does full-stack experimentation change an organization technically and culturally?