Unpacking Project Failures
Most of our failed projects died from bias and misguided assumptions, not bad engineering.
Sep 24, 2024
After successfully re-architecting our core frontend systems in just under a month and handling a traffic surge of nearly 1 million requests, the team decided to celebrate. During the celebration, an intern, just two months into the role, asked me a simple question: How many parties have I missed?
Initially, I couldn’t quantify it. But it got me thinking, not about the parties we’ve had, but the ones we missed. The celebrations we couldn’t have because of late project deliveries or underwhelming outcomes. As I reflected further and revisited past project failures, I noticed a clear pattern. Most projects from our engineering roadmap ended up in the "project graveyard" or did so because of biases or misguided assumptions.
Whose fault Is It Anyway?
Bias can come from anywhere: development, product, infrastructure, or anyone else close to the work. These are the places where bias, or an atomistic view of the system, cost us a few more reasons to celebrate.
Throughput Estimations
Estimating system throughput can be incredibly complex. Typically, we rely on historical patterns and projections to evaluate throughput and conduct peak benchmarking. In one instance, we assumed a seemingly high threshold as a safe limit for our system. What we didn’t anticipate was a sudden surge in app opens and site visits. We believed the system could handle anything unless we saw a surge of over 100x.
All the key engineering decision-makers were in agreement; everything seemed under control. It was a significant event, and every team was geared to support growth. The marketing team, however, triggered a series of staggered push notifications that coincided with the peak in site visitors. This, combined with an unusually high app open rate (~24% compared to the usual ~11%), created added pressure.
While that alone was manageable, the issue escalated due to a shareable landing page that allowed real-time interaction between users and their friends. This triggered even more app opens and active users. What we failed to account for was the user’s exit from the landing page, which compounded the problem.
The combination of push notifications, back navigation, and increased traffic caused a rise in system latency. With the higher latency came retries due to timeouts, overwhelming the system and causing the gateway service (think BFF) to crash. Fortunately, our load-shedding mechanisms kicked in, and recovery happened faster than expected.
This event forced us to realign our priorities. Although the system has not handled another large-scale event with full user interaction since, it was pushed into the icebox until recently, when it resurfaced for future consideration.
Learnings: honour Murphy’s Law
ReInventing the Wheel
Our velocity in shipping out features sometimes puts us in situations where a project appears fairly straightforward. There’s no need for fancy algorithms, complex UX, or cutting-edge technology. These projects seem direct and are easy to estimate. However, many of these projects still spill over deadlines.
Upon digging deeper, a clear pattern emerged. Projects in this category often had proof of concepts (POCs) and detailed documentation readily available in our digital workspace. This created an interesting scenario. New engineers, often early in their careers, were usually assigned these tasks by senior engineers since they were clear, well-defined, and had plenty of resources to help the younger engineers execute independently.
But this approach had an unintended consequence: the new engineers often ended up focusing on reinventing the wheel.
Younger engineers, particularly those early in their careers, tend to enjoy building things from scratch. There’s a certain excitement in the creative process, and it helps them feel a sense of ownership over the project. For them, it’s an opportunity to learn and demonstrate their skills. On the other hand, more experienced engineers typically prefer not to build from scratch unless absolutely necessary. They understand the trade-offs involved and often lean towards leveraging existing solutions to save time, reduce risk, and ship faster. Experience teaches them that the end goal is delivery, not reinvention.
This realisation made me start viewing interns and younger SDEs simply as 'engineers,' without the bias of preconceived notions about what they can or cannot do. By removing these preconceptions, I began to see their potential more clearly and understand that, while their approach may differ and sometimes omit nuances due to having less context, it can still be just as valuable in delivering impactful results.
Perfect Product Bias
This journey toward perfection doesn’t just apply to engineers. Product managers, designers, and others in the team are all striving for excellence in parallel, each from their own perspective.
Product perfection usually comes from good intent. Product wants the highest value version. Design wants the cleanest experience. Engineering wants the system to hold up under edge cases.
None of these are bad goals. The trouble starts when every team optimizes its own version of "complete" without pricing the combined cost.
That is when scope quietly grows. A small interaction needs another state. A state needs another API behavior. The API behavior needs another fallback. Each addition is defensible on its own, but the project starts drifting.
The healthier moment is when everyone names the core product again. What must be true for this to solve the user problem? What can wait? What decision are we making expensive by delaying it?
That conversation is not a downgrade in craft. It is how craft survives delivery pressure.
The key to shipping fast has been identifying the core product. We shifted away from micro-optimizing at the implementation level because it slowed us down. First, solve the problem. Then evaluate the solution and its impact. Only after that do we refine the offering.
By following this structured approach, we ensure that we ship faster while still delivering impactful, high-quality results. And, of course, there’s always hope for further improvement (and parties!).