How Raw User Feedback Saved Our Product (When Dashboards Failed)
We drifted from our users, and our product showed it. Raw WhatsApp messages fixed what dashboards could not.
Oct 19, 2023
Products improve faster when the people building them can hear directly from the people using them. I use a simple loop: build, fix, then improve.
The loop
Build. Get a useful first version in front of people. Do not wait for a perfect solution that nobody has tested.
Fix. Use early usage and conversations to remove the problems that stop the first version from doing its job.
Improve. Once the basics work, invest in the experience you wanted to create. User needs will keep changing, so improvement is a loop rather than a final phase.
The drift
The first two phases are direct, and since we stay closer to the end users there, we usually have a clear picture in front of us. The challenges surface during the third phase. The intent here is to improve, but as the application scales, we begin drifting away from the end users and rely more on product telemetry and statistics. These proxy methods are good but often miss out on the nuances.
The disconnect snowballs. We add more product and engineering effort, but extra hands do not restore the missing signal.
I was working on a B2B product for the past two years, and we found ourselves stuck in a similar situation some time back. We were trying anything and everything to "optimize." Not much had changed in terms of the metrics, and any communications with the end users ended up being informative but not insightful.
After about a hundred interviews and other user interactions, we found two problems.
-
The end user was not always the decision maker. Feedback from licensees was useful, but it was biased toward pricing and reporting. The people using the product every day gave us details about speed, confusing flows, and operational workarounds that the decision makers never mentioned.
-
Our roadmap had drifted from those daily problems. We were building add-ons and automations while users cared more about speed and operational efficiency.
The solution: put raw feedback in the team's path
The target was to gather feedback from over a hundred thousand DAU (daily active users). Since the users were not technically very aware, email was not an option. We thought of creating a voice call-based helpline, but the cost was high. Moreover, we did not want to lose out on the telemetry details we had. Nor did we want the user to input a lot of information while being on a call with us. Also, a call-based setup would not give us a firsthand insight into the suggestions/feedback we get and the issues they face.
The answer was chat.
WhatsApp was more accessible than email for this audience, and typing let users describe a problem without navigating a support form.
This solved the top of the funnel, but not the end of it. So we brainstormed what our teams (specifically engineering and product) are usually hooked on to.
Slack/Teams or any other org-level communication tool. Slack in our case.
- We created a channel in Slack.
- Exposed a web hook to post messages in the channel via a proxy.
- And directly pushed the feedback to this channel from our products.
The messages were raw, misspelled, and sometimes hard to interpret. That was useful. Product and engineering could see the user's words, reproduce the issue, and discuss it in the same channel instead of receiving a polished summary weeks later.
Impact
NPS increased by 11 points over the next three months and another 7 points in the following three. It stabilised around 41. The channel also surfaced bugs and broken releases before our monitoring did; twice, it helped us avoid monetary losses.
The lesson was not that dashboards are useless. They tell us what changed. Direct conversations told us why.