Running Towards Fire: The Key Trait That Separates Senior from Junior Engineers

Production was down for an hour yesterday.

Clickhouse, Parqet’s chosen database for stock market quotes, had too many open connections and too little memory. It took a Google Meet, three developers, and about thirty minutes of looking at various graphs and logs to locate the issue. After another thirty minutes of scaling and redeploying our API, everything returned to normal.

During my first years of coding, I was quiet in these meetings.

I couldn’t make anything of the charts, graphs, metrics. Even if a spike in a chart seemed like a clue, I didn’t know whether it was an indicator or the usual pattern.

After a couple of outages and extinguished fires, I felt more confident.

But it took a mindset shift to arrive at that point.

Junior Engineers Run Away From Fires

When production breaks, are you the one who fixes it?

Most junior engineers aren’t. Instead, they expect more experienced colleagues to act. Understandably so: Seniors can quickly find the origin of a bug and fix it in a durable way, whereas juniors can waste hours running in the wrong direction.

But waiting for seniors can turn into a habit.

If an outage happens and your first action is to see whether your coworkers are online, or if a critical but inexplicable bug appears in your monitoring but you ignore it and wait for others to resolve it, you are hiding behind others’ experience.

You are running away from fires.

They are, however, where you learn most.

Fires Are Your Best Bet For Growth

Humans learn through feedback.

With implementing features, the feedback is clear and available. If it works, you did a good job. If it doesn’t, you didn’t. Everybody can work with that. That’s why a programmer’s skill isn’t visible in what kind of features they can build but in how they build them.

But where do you get feedback for the how?

Only time will reveal the quality of your code. Your code might be difficult to work with when adding new features. Your coworkers might not understand it. Or it might break production.

Whatever it is, your only source of feedback is when something bad happens.

But if you ignore the bad thing and run away from the fire, you’re skipping valuable lessons.

The nifty part is that you can also learn from others’ mistakes. An outage might not be caused by your code but by a coworker’s, by an external dependency, or by a setting of your database. To uncover the root of the problem, you must learn about all of these things.

You should, therefore, appreciate things going south — and run to the fire.

It’s your best bet to learn what matters.

Parting Words

Fires are annoying. They distract from shipping features.

But fires won’t cease to exist, and the experience necessary to fight and prevent them is what makes a great engineer. They are the best opportunity for growth.

The next time you have an outage or a critical bug, be the first to try to solve it.

Soon, your coworkers will look to you to fix it, and you’ll be more than capable of doing so.

That’s it!

All the best,

Julian