Unpacking Project Failures
Most of our failed projects died from bias and misguided assumptions, not bad engineering.
Sep 24, 2024
After successfully re-architecting our core frontend systems in just under a month and handling a traffic surge of nearly 1 million requests, the team decided to celebrate. During the celebration, an intern, just two months into the role, asked me a simple question: How many parties have I missed?
Initially, I couldn’t quantify it. But it got me thinking about the celebrations we had missed because of late deliveries or underwhelming outcomes. When I revisited those projects, I noticed a pattern: many died from bias or a bad assumption before engineering quality became the visible problem.
Where our assumptions broke
Bias can come from development, product, infrastructure, or anyone else close to the work. The common failure was looking at one part of the system and treating it as the whole.
Throughput Estimations
Estimating system throughput can be incredibly complex. Typically, we rely on historical patterns and projections to evaluate throughput and conduct peak benchmarking. In one instance, we assumed a seemingly high threshold as a safe limit for our system. What we didn’t anticipate was a sudden surge in app opens and site visits. We believed the system could handle anything unless we saw a surge of over 100x.
All the key engineering decision-makers were in agreement; everything seemed under control. It was a significant event, and every team was geared to support growth. The marketing team, however, triggered a series of staggered push notifications that coincided with the peak in site visitors. This, combined with an unusually high app open rate (~24% compared to the usual ~11%), created added pressure.
While that alone was manageable, the issue escalated due to a shareable landing page that allowed real-time interaction between users and their friends. This triggered even more app opens and active users. What we failed to account for was the user’s exit from the landing page, which compounded the problem.
The combination of push notifications, back navigation, and increased traffic caused a rise in system latency. With the higher latency came retries due to timeouts, overwhelming the system and causing the gateway service (think BFF) to crash. Fortunately, our load-shedding mechanisms kicked in, and recovery happened faster than expected.
This event forced us to realign our priorities. Although the system has not handled another large-scale event with full user interaction since, it was pushed into the icebox until recently, when it resurfaced for future consideration.
Lesson: test the path users actually take, not only the load we expect.
Reinventing the Wheel
Our velocity in shipping out features sometimes puts us in situations where a project appears fairly straightforward. There’s no need for fancy algorithms, complex UX, or cutting-edge technology. These projects seem direct and are easy to estimate. However, many of these projects still spill over deadlines.
Upon digging deeper, a clear pattern emerged. Projects in this category often had proof of concepts (POCs) and detailed documentation readily available in our digital workspace. This created an interesting scenario. New engineers, often early in their careers, were usually assigned these tasks by senior engineers since they were clear, well-defined, and had plenty of resources to help the younger engineers execute independently.
But this approach had an unintended consequence: the new engineers often ended up focusing on reinventing the wheel.
Younger engineers, particularly those early in their careers, tend to enjoy building things from scratch. There’s a certain excitement in the creative process, and it helps them feel a sense of ownership over the project. For them, it’s an opportunity to learn and demonstrate their skills. On the other hand, more experienced engineers typically prefer not to build from scratch unless absolutely necessary. They understand the trade-offs involved and often lean towards leveraging existing solutions to save time, reduce risk, and ship faster. Experience teaches them that the end goal is delivery, not reinvention.
This realisation made me start viewing interns and younger SDEs simply as 'engineers,' without the bias of preconceived notions about what they can or cannot do. By removing these preconceptions, I began to see their potential more clearly and understand that, while their approach may differ and sometimes omit nuances due to having less context, it can still be just as valuable in delivering impactful results.
Perfect Product Bias
The same bias showed up outside engineering. Product wanted the highest-value version, design wanted the cleanest experience, and engineering wanted the system to survive edge cases.
Product perfection usually comes from good intent. Product wants the highest value version. Design wants the cleanest experience. Engineering wants the system to hold up under edge cases.
None of these are bad goals. The trouble starts when every team optimizes its own version of "complete" without pricing the combined cost.
That is when scope quietly grows. A small interaction needs another state. A state needs another API behavior. The API behavior needs another fallback. Each addition is defensible on its own, but the project starts drifting.
The healthier moment is when everyone names the core product again. What must be true for this to solve the user problem? What can wait? What decision are we making expensive by delaying it?
That conversation is not a downgrade in craft. It is how craft survives delivery pressure.
The practical rule is simple: identify what must be true for the feature to solve the user problem, ship that version, and then evaluate the result before adding polish. It keeps implementation detail from deciding the scope by accident.