Engineering Leadership 101: Steering Clear of High-Risk Practices
In software engineering, we’ve all felt the pressure of a looming deadline, urgent requirements, and demanding customers, making it tempting to augment our processes in order to accelerate our work.
Not only is this behavior risky in itself, but it also introduces a dangerous precedent to the team: “Since we took a shortcut today, why shouldn’t we take another shortcut tomorrow?”
Sometimes, we don’t even know we’re missing something in our process, and accidentally have introduced risk into our development lifecycle.
After seeing these mistakes at many companies and clients over the years, I consolidated a list of 20 process shortcuts that can severely increase risk of a production release if left unconsidered.
1. Skipping Code Reviews
Rushing through or skipping code reviews introduces the risk of missing critical bugs or architectural issues. Even experienced developers benefit from a second pair of eyes on their code.
2. Ignoring Test Coverage
Failing to maintain adequate test coverage, especially for critical parts of the codebase, can lead to bugs going undetected. Automated tests — unit tests, integration tests, and end-to-end tests — provide an essential safety net before pushing changes to production.
3. Not Setting Up Rollback Plans
Deploying without a clear rollback strategy can turn small mistakes into major incidents. Always ensure you have a tested and reliable rollback plan for releases, in case something goes wrong.
4. Ignoring Technical Debt
Accumulating technical debt without addressing it increases maintenance costs and slows down future development. Prioritizing new features at the expense of refactoring or fixing old code can lead to long-term instability.
5. Deploying Large Changes at Once
Deploying significant changes in a single release increases the risk of failures. Break down changes into smaller, manageable units and deploy incrementally to make debugging easier and reduce potential impact.
6. Releasing on Fridays or Weekends
Releasing code right before the weekend is risky since fewer engineers will be available to handle issues that arise in production. It’s best to release earlier in the week when support is readily available to monitor and address problems.
7. Lack of Documentation
Failing to maintain up-to-date documentation for new features, architecture changes, and dependencies can leave the team in a bind when something breaks or when new team members join. Comprehensive documentation ensures smoother onboarding and quicker resolution of issues.
8. Not Involving QA Early
If Quality Assurance (QA) is only involved at the end of the development cycle, bugs may be caught too late, delaying releases. Engaging QA throughout development helps identify issues earlier.
9. Ignoring Performance and Scalability Testing
Releasing features without testing their performance at scale can lead to production failures under heavy load. Regularly conduct stress and load testing to ensure the system can handle real-world usage.
10. Over-Reliance on Hotfixes
Fixing production issues with quick hotfixes without addressing the root cause introduces long-term instability. Focus on solving problems systematically rather than relying on temporary solutions.
11. Hiding Bad News From Stakeholders
Things are never going to go completely smoothly, but hiding these issues causes bigger problems. Failing to communicate problems or risks to stakeholders, including product owners, executive sponsors, and other teams consuming your product, ultimately sows distrust, and prevents the team from discovering and improving points of friction.
12. Deploying Without Staging Environment Testing
Skipping the testing of new code in a staging environment that closely mirrors production is a common risk. Without this step, potential issues might only surface in production, affecting real users and potentially causing downtime.
13. Underestimating Production Monitoring
Deploying without proper monitoring tools in place is risky because it delays the detection of issues. Ensure that real-time monitoring and alerting systems are active and tailored to detect critical issues in production environments.
14. Caving to the HiPPO
It is tempting to bend to the HiPPO (Highest Paid Person’s Opinion), but it is critical to understand that they are ultimately just another contributing member just like you. Approach every conversation as a collaboration, where ideas are shared and challenged, and explore all ideas deeply before choosing a path forward.
15. Overloading Teams with Urgent Work
Constantly assigning high-priority, urgent tasks to your team can lead to burnout, sloppy work, and increased error rates. Balance urgent work with regular maintenance and improvement tasks to keep the team healthy and productive.
16. Lack of Backup and Disaster Recovery Plans
Relying on the assumption that everything will go smoothly without setting up a robust backup and disaster recovery plan can lead to catastrophic data loss or extended downtime in case of failure. Periodically testing these plans is just as important as having them.
17. Allowing “Shadow IT” Systems
Developers may sometimes implement unofficial tools or workflows (“shadow IT”) to speed up processes, but these often require a lot of upkeep, and can become risky if they bypass security protocols or are not properly maintained. Ensure all systems and tools used by the team are known, supported, and compliant with organizational policies.
18. Overly Aggressive Deadlines
Setting unrealistic timelines for project delivery forces engineers to cut corners in testing, reviews, or building a scalable implementation, leading to poor-quality code. Set achievable goals that account for development, testing, and stabilization.
19. Not Addressing Security Concerns Early
Pushing security considerations to the end of the development process rather than integrating them from the beginning creates vulnerabilities and a frenzy of unplanned work to address issues before a production release. Implement secure coding practices and regular security audits throughout the development lifecycle, and address issues early.
20. Allowing Scope Creep Without Reassessment
When new features are continuously added to a project without reassessing deadlines and resources, it introduces significant risk. Scope creep can lead to missed deadlines, lower quality, and incomplete features. Ensure scope changes are carefully evaluated.
If you can address all of these risky behaviors, you are likely to release much more stable, secure, and performant products on a much more reliable basis.
