A Product Manager’s Lessons From a Failed Feature

I had a great idea for a feature in the early days of CloudHealth: enabling customers to collaborate around their cloud resources. For example, a manager might want to let their DevOps engineer know they forgot to “turn the lights out” on some infrastructure; a developer might want to let their team know they were running some extra EC2 instances for a test; and so on. I thought of the feature as the “yellow sticky notes" for the cloud, and in a burst of inspiration one evening, I wrote the software for functionality called Comments. The feature allowed my users to add notes to any cloud resource, and then track and manage these notes like you would do in Google Docs. I was pretty excited when I launched Comments a few days later, and I evangelized it directly to all of my early customers. But what happened next was a real surprise: after the initial use by customers, they never again returned to it.

Without knowing it, I had committed the cardinal sin of product management: to build something that doesn’t matter. I allowed myself to fall into the trap of making a product decision based on a hunch or instinct. It had seemed obvious to me that customers would need this feature, since I had personally filled up notebooks and chat channels communicating this type of information in previous roles. But what I didn’t do was to test the critical assumption: that my customers would give up their notebooks and chat channels to use my product.

Being a product manager is one of the hardest jobs in the tech industry. A successful product manager needs to drive a portfolio of investments, balance the strategic needs of their business with the tactical needs of customers, satisfy the competing requests from customers and internal stakeholders, and do all this while driving toward a strategic vision. When a product manager gets all this to align, it is like magic. But more often than not, delivering consistent results as a product manager is really really hard.

The single biggest difference between successful product managers and everyone else is not in their management of products, but in their management of risk. Every decision a product manager makes comes with it risk - e.g. the risk you build something that doesn’t matter, the risk you solve the right problem but in the wrong way, the risk of solving a lower priority issue at the expense of a more important one. All product managers combat risk with knowledge, experience, and instinct. But the best product managers put another tool in their toolbelt: validated learning.

Validated learning was formalized by Eric Ries in 2011 with his book Lean Startup. The concept was simple: to subject critical assumptions to testing by customers. It is the scientific method for product managers, and was an evolution of concepts formulated in Steven Blank’s Customer Driven Development. Combined with the Minimum Viable Product (MVP), validated learning provides a powerful tool for product managers to manage risk. The process works like this: (1) identify the critical assumptions behind a decision, (2) come up with an experiment that efficiently tests the critical assumptions with customers, (3) run the experiment, and (4), assess the results. If I had applied validated learning to my Comments feature, it might have worked like this:

1. Identify critical assumption(s) - The critical assumption behind Comments was: customers needed a common system to share information on cloud resources within their organizations.
2. Define experiment - I would have then designed an experiment to test my critical assumption(s) in the most expedient way. For example, I could have asked a few customers to agree to use an existing shared system for capturing information on their cloud resources - e.g. a Google Doc. Success could be measured on the adoption of this shared system - i.e. engagement over time. Note: the key to a good experiment is to define the success criteria clearly, allowing you to dispassionately review the results after the experiment.
3. Run the experiment - I would have engaged customers in the experiment. I could have watched their engagement with their respective shared Google Docs, and seen real-time how they were (and were not) using it.
4. Assess the results - What I almost certainly would have seen was early engagement, followed by a rapid drop off in interest. Customer interviews would have told me why: the need to share this information broadly is the exception and not the rule, and thus a shared system was overkill. My conclusion from this validated learning would have been obvious: don’t build the feature.

A running joke within my early engineering team was when I would allow them to take my Comments feature out of the product. It was obvious to everyone it was not getting used, but yet it took me some time before I admitted my failure. But after finally relenting, I was able to admit the experience had given me a healthy appreciation for the complexity of being a product manager, and an increased comitment to leveraging validated learning to minimize my future mistakes.