The whole dev cycle spans from ideation to running on production. Chop it up however fine you want as long as it fits your organization/team size.
Company of less than 5 ppl? All you need to care is whether the whole cycle is delivering something of value.
Team of 5 in a 500 engineers org? All you ever have to care is a small slice of the cycle. Maybe a stage of testing, maybe an aspect of deployment, maybe a piece of user journey, maybe a feature.
The gist is to know that the cycle exists and which part of the cycle matters to your team.
epolanski 619 days ago [-]
Not fond of these metrics. Feels easy to gamify and hack. I can just push commit a when it's more mature, good I improved the metric, but reduced tinkering, collaboration with devs and product and probably released a subpar solution.
While I sort of see the point of such metrics, in my experience they are nothing but data porn to show improvements to higher up management, which in time lead to yet more complex and beaurocratic organizations.
kqr 616 days ago [-]
> Not fond of these metrics. Feels easy to gamify and hack. I can just push commit a when it's more mature, good I improved the metric, but reduced tinkering, collaboration with devs and product and probably released a subpar solution.
This is not gamed/hacked -- it's the metric working as designed. It is meant to measure purely how fast you can get commits into production. If you have found a way to do that faster, then the metric should increase to reflect that.
So there are two aspects of the problem: are you delivering fast, and are you delivering something valuable?
The delivery lead time doesn't even pretend to measure whether you're delivering something valuable. You need a different metric for that. And in your example, that would go down by your practise.
sparsely 619 days ago [-]
I like seeing actual numbers about what Good is meant to be for this metrics - it's easy for a devlops advocate to highlight a metrics and push for further investment in it, despite it not being a major weakness of the organisation otherwise.
isthisit 619 days ago [-]
Author here. The targets you set for Delivery Lead Time and Deployment Frequency should be determined by your organisation and your team. Context matters. I've been at an organisation where upper management introduced the Accelerate/DORA targets verbatim. These numbers did not resonate at all with engineers and so saw little to no adoption.
Accelerate was published four years ago and understanding of the 'four DORA metrics' has matured. Štěpán Davidovič wrote a good piece Incident Metrics in SRE [1] where he pulls apart the Mean Time to Recovery Metric. I've written an overview of the DORA metrics and what to use instead - SLOs & SLIs [2]
External benchmarking is a tricky thing, especially with a number like this, which is derived from a self-reported survey. The important thing is that those teams perceive they can quickly deploy a change. Perhaps they do, at times, but as the author of the OP points out, that’s not really revealed in the source literature, and being able to isn’t the same thing as often or frequently flexing that muscle.
I would start with “can you even measure this”. Just getting something like this instrumented for your team is probably going to drive improvements.
Then you can put it on a Shewhart/PDCA Cycle driven primarily by the team itself through its retrospective and prioritization process. “Has this metric slipped? Do we feel like there are blocks in the deploy pipeline? Are those blocks more important than other work right now? What changes does the team feel most invested in?” Fostering that conversation is really important.
Grabbing these metrics, poorly measuring them, and then trying to hold a team accountable to some arbitrary, external, non-contextual measure is a sure way to kill morale and productivity, hinder future leadership efforts, and waste time and resources and non-priorities.
Of course, doing nothing will accomplish the same.
We do this kind of work with hospitals and their process metrics. Our measurements are always based on raw data, never self-reported metrics. Frequently, team’s self-reported perceptions prior to measurement is the most telling thing. When I worked in k-12 Ed, the same was true with teachers and their students: they perceived their students as great writers, but they were often much worse when measured quantitatively by normalized graders. What is important in both cases isn’t blindly pushing to improve those metrics, but developing an understanding of why that perceived difference exists, why the numbers are the way they are, and ultimately, whether and what kinds of differences you can make. In the case of hospitals, there are some units/teams where you don’t want high performance numbers, because it might lead to higher error rates. In Ed, there might be limits to what you can do as a single-subject teacher focused on a single year in a group of children’s lives. But you can understand each groups challenges and strengths a little better and maybe adapt accordingly.
Blindly pushing top-line metrics is not the same as blindly doing the same thing with no measurement or learning. Both are bad.
stevenally 619 days ago [-]
There is a general rule in social and economic life” wrote W. Brian Arthur, an economist at the Santa Fe Institute, “given any system, people will find a way to exploit it. Or to say this more succinctly: All systems will be gamed”.
Which is why MBO is ultimately not a good management tool and is clearly not serving many organizations as well as they need, especially when applied to the management of Systems/Software Engineers who are all too aware of this fact.
If you want to be successful, you have to focus on intrinsic motivations of these kinds of staff. By and large, they do not respond effectively to extrinsic motivations. They may respond "positively", but not necessarily "effectively". It's difficult, though, in a world where any non-quantifiable evaluation is rightly viewed with incredible suspicion. It forces managers and leaders to walk a hard line.
Company of less than 5 ppl? All you need to care is whether the whole cycle is delivering something of value. Team of 5 in a 500 engineers org? All you ever have to care is a small slice of the cycle. Maybe a stage of testing, maybe an aspect of deployment, maybe a piece of user journey, maybe a feature. The gist is to know that the cycle exists and which part of the cycle matters to your team.
While I sort of see the point of such metrics, in my experience they are nothing but data porn to show improvements to higher up management, which in time lead to yet more complex and beaurocratic organizations.
This is not gamed/hacked -- it's the metric working as designed. It is meant to measure purely how fast you can get commits into production. If you have found a way to do that faster, then the metric should increase to reflect that.
So there are two aspects of the problem: are you delivering fast, and are you delivering something valuable?
The delivery lead time doesn't even pretend to measure whether you're delivering something valuable. You need a different metric for that. And in your example, that would go down by your practise.
Accelerate was published four years ago and understanding of the 'four DORA metrics' has matured. Štěpán Davidovič wrote a good piece Incident Metrics in SRE [1] where he pulls apart the Mean Time to Recovery Metric. I've written an overview of the DORA metrics and what to use instead - SLOs & SLIs [2]
[1] https://web.archive.org/web/20210917022148/https://static.go...
[2] https://isthisit.nz/posts/2022/state-of-the-dora-devops-metr...
I would start with “can you even measure this”. Just getting something like this instrumented for your team is probably going to drive improvements.
Then you can put it on a Shewhart/PDCA Cycle driven primarily by the team itself through its retrospective and prioritization process. “Has this metric slipped? Do we feel like there are blocks in the deploy pipeline? Are those blocks more important than other work right now? What changes does the team feel most invested in?” Fostering that conversation is really important.
Grabbing these metrics, poorly measuring them, and then trying to hold a team accountable to some arbitrary, external, non-contextual measure is a sure way to kill morale and productivity, hinder future leadership efforts, and waste time and resources and non-priorities.
Of course, doing nothing will accomplish the same.
We do this kind of work with hospitals and their process metrics. Our measurements are always based on raw data, never self-reported metrics. Frequently, team’s self-reported perceptions prior to measurement is the most telling thing. When I worked in k-12 Ed, the same was true with teachers and their students: they perceived their students as great writers, but they were often much worse when measured quantitatively by normalized graders. What is important in both cases isn’t blindly pushing to improve those metrics, but developing an understanding of why that perceived difference exists, why the numbers are the way they are, and ultimately, whether and what kinds of differences you can make. In the case of hospitals, there are some units/teams where you don’t want high performance numbers, because it might lead to higher error rates. In Ed, there might be limits to what you can do as a single-subject teacher focused on a single year in a group of children’s lives. But you can understand each groups challenges and strengths a little better and maybe adapt accordingly.
Blindly pushing top-line metrics is not the same as blindly doing the same thing with no measurement or learning. Both are bad.
https://waldencroft.com/gaming-the-system-the-unintended-con...
If you want to be successful, you have to focus on intrinsic motivations of these kinds of staff. By and large, they do not respond effectively to extrinsic motivations. They may respond "positively", but not necessarily "effectively". It's difficult, though, in a world where any non-quantifiable evaluation is rightly viewed with incredible suspicion. It forces managers and leaders to walk a hard line.