One of the most common problems engineering leaders face is deciding what metrics to track and report to stakeholders. I receive questions about this frequently:
I personally experienced this challenge in my first job as a CTO. Our executive team had started meeting monthly to review each department’s progress. Our CEO wanted more than just the anecdotal release updates I had been providing, and asked me to come up with a hard metric to report.
The problem: I couldn’t think of a good metric to use. Our CEO suggested the number of features we released, but I resisted because I knew it didn’t accurately account for the size of each feature. Alternatives like the number of lines of code or pull requests felt even worse, and I feared losing the respect of our developers if they found out I was representing our work in this way. In the end, I decided to give in and report the number of features released in order to save face with my boss and other executives.
In hindsight, I wish I’d approached this problem differently and today I am able to give concrete advice to leaders that find them themselves in a similar situation. This advice takes the form of reframing the problem, and a simple classification framework for metrics that leaders should be reporting on. I’ve developed this approach based on years of research and experience advising leaders.
If your boss asks for engineering metrics, start by reframing what they’re really asking for. Leaders often fall into a rabbit hole of worrying about whether to measure individuals or teams, the legitimacy of engineering metrics products, or whether engineering can be measured at all. But this misses the big picture: CEOs don’t know or care about the technicalities of engineering measurement; what they really want is a way to have confidence that you’re accountable for the millions of dollars they are spending on engineering.
Once you reframe the problem in such a way where you’re thinking about how to demonstrate strong stewardship, choosing metrics becomes much easier. For example, your CEO is likely under constant pressure around engineering headcount costs, so a clear accounting of projects being worked on along with their business ROI can reassure them and keep them looped in. If stakeholders are worried about whether the “house is in order” in regards to the performance of certain systems or teams, metrics can help assuage concerns.
Ultimately, your goal should be to proactively address these types of concerns by providing a full picture of engineering. This be accomplished by reporting metrics in these three buckets:
The following table summarizes the three buckets and examples of metrics to use for each one. Metrics can sometimes fall into more than one bucket.
You shouldn’t need fancy tools to capture these metrics–most of the information should already exist through your existing tools and processes, or else manually captured with simple spreadsheets or surveys. It doesn’t matter which specific metrics you use within each bucket or how perfectly things are measured. The important thing is to show intention and effort to your stakeholders, and then continue to iterate on your approach.
There are many metrics being advertised across the internet, which can make the process of choosing metrics overwhelming. I’ve found that categorizing metrics into these buckets forces you to be intellectually honest about why you are using a certain metric: Does the metric actually convey something useful? Or is it just something that was mentioned somewhere?
I see two common mistakes occur when leaders try to come up with metrics without this three-bucket framework:
Take a moment to examine your current engineering metrics, and I think you’ll find that you already have metrics for these buckets represented. Consider reorganizing your reports around this framework to create more alignment around the purpose behind each of your metrics. If you’ve recently joined an organization as an engineering executive, this approach can help you strengthen your relationship with stakeholders from the start.