by Jack Ganssle
jack@ganssle.com
Recently our electric toothbrush started acting oddly – differently from before. I complained to Marybeth who said, “I think it’s in the wrong mode.”
Really? A toothbrush has modes?
We in the embedded industry have created a world that was unimaginable prior to the invention of the microprocessor. Firmware today controls practically everything, from avionics to medical equipment to cars to, well everything.
And toothbrushes.
But we’re working too hard at it. Too many of us use archaic development strategies that aren’t efficient. Too many of us ship code with too many errors. That's something that can, and must, change.
Long ago the teachings of Deming and Juran revolutionized manufacturing. One of Deming's essential insights was that fixing defects will never lead to quality. Quality comes from correct design rather than patches applied on the production line. And focusing on quality lowers costs.
The software industry never got that memo.
The average embedded software project devotes 50% of the schedule to debugging and testing the code. It's stunning that half of the team’s time is spent finding and fixing mistakes.
Test is hugely important. But, as Dijkstra observed, testing can only prove the presence of errors, not the absence of bugs.
Unsurprisingly, and mirroring Deming's tenets, it has repeatedly been shown that a focus on fixing bugs will never lead to a quality product - all that will do is extend the schedule and insure defective code goes out the door.
Focusing on quality has another benefit: the project gets done faster. Why? That 50% of the schedule used to deal with bugs gets dramatically shortened. We shorten the schedule by not putting the bugs in in the first place.
High quality code requires a disciplined approach to software engineering - the methodical use of techniques and approaches long known to work. These include inspection of work products, using standardized ways to create the software, seeding code with constructs that automatically catch errors, and using various tools that scan the code for defects. Nothing that is novel or unexpected, nothing that a little Googling won't reveal. All have a long pedigree of studies proving their efficacy.
Yet only one team out of 50 makes disciplined use of these techniques.
What about metrics? Walk a production line and you'll see the walls covered with charts showing efficiency, defect rates, inventory levels and more. Though a creative discipline like engineering can't be made as routine as manufacturing, there are a lot of measurements that can and must be used to understand the team's progress and the product's quality, and to drive the continuous improvement we need.
Errors are inevitable. We will ship bugs. But we need a laser-like focus on getting the code right. How right? We have metrics; we know how many bugs the best and mediocre teams ship. Defect Removal Efficiency is a well-known metric used to evaluate quality of shipped code; it's the percentage of the entire universe of bugs found in a product that were removed prior to shipping (it's measured until 90 days after release). The very best teams, representing just 0.4% of the industry, eliminates over 99% of bugs pre-shipment. Most embedded groups only removed 95%.
Where does your team stand on this scale? Can one control quality if it isn’t measured?
We have metrics about defect injection rates, about where in the lifecycle they are removed, about productivity vs. any number of parameters and much more. Yet few teams collect any numbers.
Engineering without numbers isn’t engineering. It’s art.
Want to know more about metrics and quality in software engineering? Read any of Capers Jones’ books. They are dense, packed with tables of numbers, and sometimes difficult as the narrative is not engaging, but they paint a picture of what we can measure and how differing development activities effect errors and productivity.
Want to understand where the sometimes-overhyped agile methods make sense? Read Agile! by Bertrand Meyer and Balancing Agility and Discipline by Barry Boehm and Richard Turner.
Want to learn better ways to schedule a project and manage requirements? Read any of Karl Wiegers’ books and articles.
The truth is that we know of better ways to get great software done more efficiently and with drastically reduced bug rates.
When will we start?
Jack Ganssle has written over 1000 articles and six books about embedded systems, as well as one about his sailing fiascos. He has started and sold three electronics companies. He welcomes dialog at jack@ganssle.com or at www.ganssle.com.
