Scott Rosenthal and I go back about a thousand years; we've worked together, helped midwife the embedded field into being, had some amazing sailing adventures, and recently took a jaunt to the Azores just for the heck of it. Our sons are both big data people; their physics PhDs were perfect entrees into that field, and both now work in the field of artificial intelligence.

At lunch recently we were talking about embedded systems and AI, and Scott posed a thought that has been rattling around in my head since. Could AI replace firmware?

Firmware is a huge problem for our industry. It's hideously expensive. Only highly-skilled people can create it, and there are too few of us.

What if an AI engine of some sort could be dumped into a microcontroller and the "software" then created by training that AI? If that were possible - and that's a big "if" - then it might be possible to achieve what was hoped for when COBOL was invented: programmers would no longer be needed as domain experts could do the work. That didn't pan out for COBOL; the industry learned that accountants couldn't code. Though the language was much more friendly than the assembly it replaced, it still required serious development skills.

But with AI, could a domain expert train an inference engine?

Consider a robot: a "home economics" major could create scenarios of stacking dishes from a dishwasher. Maybe these would be in the form of videos, which were then fed to the AI engine as it tuned the weighting coefficients to achieve what the home ec expert deems worthy goals.

My first objection to this idea was that these sorts of systems have physical constraints. With firmware I'd write code to sample limit switches so the motors would turn off if at an end-of-motion extreme. During training an AI-based system would try and drive the motors into all kinds of crazy positions, banging destructively into stops. But think how a child learns: a parent encourages experimentation but prevents the youngster from self-harm. Maybe that's the role of the future developer training an AI. Or perhaps the training will be done on a simulator of some sort where nothing can go horribly wrong.

Taking this further, a domain expert could define the desired inputs and outputs, and then a poorly-paid person do the actual training. CEOs will love that. With that model a strange parallel emerges to computation a century ago: before the computer age "computers" were people doing simple math to create tables of logs, trig, ballistics, etc. A room full all labored at a problem. They weren't particularly skilled, didn't make much, but did the rote work under the direction of one master. Maybe AI trainers will be somewhat like that.

Like we outsource clothing manufacturing to Bangladesh, I could see training, basically grunt work, being sent overseas as well.

I'm not wild about this idea as it means we'd have an IoT of idiots: billions of AI-powered machines where no one really knows how they work. They've been well-trained but what happens when there's a corner case?

And most of the AI literature I read suggests that inference successes of 97% or so are the norm. That might be fine for classifying faces, but a 3% failure rate of a safety-critical system is a disaster. And the same rate for less-critical systems like factory controllers would also be completely unacceptable.

But the idea is intriguing.

Original post can be viewed here

Feel free to email me with comments.

Back to Jack's blog index page.

In the comments section of my 2020 embedded salary survey, quite a few respondents felt that much of the embedded world is being subsumed by canned solutions. Will OSes like Linux and cheap, powerful boards like the Raspberry Pi and Arduino replace traditional engineering? Has that already happened?

A number of people complained their colleagues no longer understand low-level embedded things like DMA, chip selects, diddling I/O registers, and the like. They feel these platforms isolate the engineer from those details.

Part of me says yeah! That's sort of what we want. Reuse and abstraction means the developer can focus on the application rather than bringing up a proprietary board. Customers want solutions and don't care about implementation details. We see these abstractions working brilliantly when we buy a TCP/IP stack, often the better part of 100K lines of complex code. Who wants to craft those drivers?

Another part of me says "save me from these sorts of products." It is fun to design a board. To write the BSP and toss bits at peripheral registers. Many of us got a rush the first time we made an LED blink or a motor spin. I still find that fulfilling.

So what's the truth? Is the future all Linux and Pis?

The answer is a resounding "no." A search for "MCU" on Digi-Key gets 89,149 part numbers. Sure, many of these are dups with varying packages and the like, but that's still a ton of controllers.

Limiting that search to 8 bitters nets 30,574 parts. I've yet to see Linux run on a PIC or other tiny device.

Or filter to Cortex-M devices only. You still get 16,265 chips. None of those run Linux, Windows, BSD, or any other general-purpose OS. These are all designed into proprietary boards. Those engineers are working on the bare metal... and having a ton of fun.

The bigger the embedded world gets the more applications are found. Consider machine learning. That's for big iron, for Amazon Web Services, right? Well, partly. Eta Compute and other companies are moving ML to the edge with smallish MCUs running at low clock rates with limited memory. Power consumption rules, and 2 GB of RAM at 1 GHz just doesn't cut it when harvesting tiny amounts of energy.

Then there's cost. If you can reduce the cost of a product made in the millions by just a buck the business prospers. Who wants a ten dollar CPU when a $0.50 microcontroller will do?

Though I relish low-level engineering our job is to get products to market as efficiently as possible. Writing drivers for a timer is sort of silly when you realize that thousands of engineers using the same part are doing the same thing. Sure, semi vendors often deliver code to handle all of this, but in my experience most of that is either crap or uses the peripherals in the most limited ways. A few exceptions exist, such as Renesas's Synergy. They go so far as to guarantee that code. My fiddling with it leaves me impressed, though the learning curve is steep. But that sort of abstraction surely must be a part of this industry going forward. Just as we don't write protocol stacks and RTOSes any more, canned code will become more common.

Linux and canned boards have important roles in this business. But an awful lot of us will still work on proprietary systems.

View original post here

For novel ideas about building embedded systems (both hardware and firmware), join the 35,000 engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe

By Jack Ganssle 

As Good As It Gets 

How good does firmware have to be? How good can it be? Is our search for perfection, or near-perfection an exercise in futility?

Complex systems are a new thing in this world. Many of us remember the early transistor radios which sported a half dozen active devices, max. Vacuum tube televisions, common into the 70s, used 15 to 20 tubes, more or less equivalent to about the same number of transistors. The 1940s-era ENIAC computer required 18,000 tubes, so many that technicians wheeled shopping carts of spares through the room, constantly replacing those that burned out. Though that sounds like a lot of active elements, even the 25 year old Z80 chip used a quarter of that many transistors, in a die smaller than just one of the hundreds of thousands of resistors in the ENIAC.

Now the Pentium IV, merely one component of a computer, has 45 million transistors. A big memory chip might require a third of a billion. Intel predicts that later this decade their processors will have a billion transistors. I'd guess that the very simplest of embedded systems, like an electronic greeting card, requires thousands of active elements.

Software has grown even faster, especially in embedded applications. In 1975 10,000 lines of assembly code was considered huge. Given the development tools of the day - paper tape, cassettes for mass storage, and crude teletypes for consoles - working on projects of this size was very difficult. Today 10,000 lines of C - representing perhaps 3 to five times as much assembly - is a small program. A cell phone might contain a million lines of C or C++, astonishing considering the device's small form factor and miniscule power requirements.

Another measure of software size is memory usage. The 256 byte (that's not a typo) EPROMs of 1975 meant even a measly 4k program used 16 devices. Clearly, even small embedded systems were quite pricey.  Today? 128k of Flash is nothing, even for a tiny app. The switch from 8 to 16 bit processors, and then from 16 to 32 bitters, is driven more by addressing space requirements than raw horsepower.

In the late 70s Seagate introduced the first small Winchester hard disk, a 5 Mb 10 pound beauty that cost $1500. 5 Mb was more disk space than almost anyone needed. Now 20 Gb fits into a shirt pocket, is almost free, and fills in the blink of an eye.

So, our systems are growing rapidly in both size and complexity. And, I contend, in failure modes. Are we smart enough to build these huge applications correctly?

It's hard to make even a simple application perfect; big ones will possibly never be faultless. As the software grows it inevitably becomes  more intertwined; a change in one area impacts other sections, often profoundly. Sometimes this is due to poor design; often, it's a necessary effect of system growth.

The hardware, too, is certainly a long way from perfect. Even mature processors usually come with an errata sheet, one that can rival the datasheet in size. The infamous Pentium divide bug was just one of many bugs - even today the Pentium 3's errata sheet (renamed "specification update") contains 83 issues. Motorola documents nearly a hundred problems in the MPC555.

I salute the vendors for making these mistakes public. Too many companies frustrate users by burying their mistakes.

What is the current state of the reliability of embedded systems? No one knows. It's an area devoid of research. Yet a lot of raw data is available, some of which suggests we're not doing well.

The Mars Pathfinder mission succeeded beyond anyone's dreams, despite a significant error that crashed the software during the lander's descent. A priority inversion problem - noticed on Earth but attributed to a glitch and ignored - caused numerous crashes. A well-designed watchdog timer recovery strategy saved the mission. This was a very instructive failure as it shows the importance of adding external hardware and/or software to deal with unanticipated software errors.

The August 15, 2001 issue of the Journal of the American Medical Association contained a study of recalls of pacemakers and implantable cardioverter-defibrillators. (Since these devices are implanted subcutaneously I can't imagine how a recall works). Surely designers of these devices are on the cutting edge of building the very best software. I hope. Yet between 1990 and 2000 firmware errors accounted for about 40% of the 523,000 devices recalled.

Over the ten years of the study, of course, we've learned a lot about building better code. Tools have improved and the amount of real software engineering that takes place is much greater. Or so I thought. Turns out that the annual number of recalls between 1995 and 2000 increased.

In defense of the pacemaker developers, no doubt they solve very complex problems. Interestingly, heart rhythms can be mathematically chaotic. A slight change in stimulus can cause the heartbeat to burst into quite unexpected randomness. And surely there's a wide distribution of heart behavior in different patients.

Perhaps a QA strategy for these sorts of life-critical devices should change. What if the  responsible person were one with heart disease! who had to use the latest widget before release to the general public?

A pilot friend tells me the 747 operator's manual is a massive tome that describes everything one needs to know about the aircraft and its systems. He says that fully half of the book documents avionics (read: software) errors and workarounds.

The Space Shuttle's software is a glass half-empty/half-full story. It's probably the best code ever written, with an average error rate of about one per 400,000 lines of code. The cost: $1000 per line. So, it is possible to write great code, but despite paying vast sums perfection is still elusive. Like the 747, though, the stuff works "good enough", which is perhaps all we can ever expect.

Is this as good as it gets?

The Human Factor

Let's remember we're not building systems that live in isolation. They're all part of a much more complex interacting web of other systems, not the least of which is the human operator or user. When tools were simple - like a hammer or a screwdriver - there weren't a lot of complex failure modes. That's not true anymore. Do you remember the USS Vincennes? She is a US Navy battle cruiser, equipped with the incredibly sophisticated Aegis radar system. In July, 1988 the cruiser shot down an Iranian airliner over the the target wasn't an incoming enemy warplane, but the data was displayed on a number of terminals that weren't easy to see. So here's a failure where the system worked as designed, but the human element created a terrible failure. Was the software perfect since it met the requirements?

Unfortunately, airliners have become common targets for warplanes. This past October a Ukrainian missile apparently shot down a Sibir Tu-154 commercial jet, killing all 78 passengers and crew. As I write the cause is unknown, or unpublished, but local officials claim the missile had been targeted on a close-by drone. It missed, flying 150 miles before hitting the jet. Software error? Human error?

The war in Afghanistan shows the perils of mixing men and machines. At least one smart bomb missed its target and landed on civilians. US military sources say wrong target data was entered. Maybe that means someone keyed in wrong GPS coordinates. It's easy to blame an individual for mistyping! but doesn't it make more sense to look at the entire system as a whole, including bomb and operator? Bombs have pretty serious safety-critical aspects. Perhaps a better design would accept targeting parameters in a string that includes a checksum, rather like credit card numbers. A mis-keyed entry would be immediately detected by the machine.

It's well-known that airplanes are so automated that on occasion both pilots have slipped off into sleep as the craft flies itself. Actually, that doesn't really bother me much, since the autopilot beeps when at the destination, presumably waking the crew. But, before leaving the fliers enter the destination in latitude/longitude format into the computers. What if they make a mistake (as has happened)? Current practice requires pilot and co-pilot to check each other's entries, which will certainly reduce the chance of failure. Why not use checksummed data instead and let the machine validate the data?

Another US vessel, the Yorktown, is part of the Navy's "Smart Ship" initiative. Hugely automating the engineering (propulsion) department reduces crew needs by 10% and saves some $2.8 million per year on this one ship. Yet the computers create new vulnerabilities. Reports suggest that an operator entered an incorrect parameter which resulted in a divide-by-zero error. The entire network of Windows NT machines crashed. The Navy claims the ship was dead in the water for about three hours; other sources claim it was towed into port for two days of system maintenance. Users are now trained to check their parameters more carefully. I can't help wonder what happens in the heat of battle, when these young sailors may be terrified, with smoke and fire perhaps raging. How careful will the checks be?

Some readers may also shudder at the thought of NT controlling a safety-critical system. I admire the Navy's resolve to use a commercial, off the shelf product, but wonder if Windows, which is the target of every hacker's wrath, might not itself create other vulnerabilities. Will the next war be won by the nation with the best hackers?

A plane crash in Florida, in which software did not contribute to the disaster, was a classic demonstration of how difficult it is to put complicated machines in the hands of less-than-perfect people. An instrument lamp burned out. It wasn't an important problem, but both pilots became so obsessed with tapping on the device they failed to notice that the autopilot was off. The plane very gently descended till it crashed, killing everyone.

People will always behave in unpredictable ways, leading to failures and disasters with even the best system designs. As our devices grow more complex their human engineering becomes ever more important. Yet all too often this is neglected in our pursuit of technical solutions.

Solutions?

I'm a passionate believer in the value of firmware standards, code inspections, and a number of other activities characteristic of disciplined development. It's my observation that an ad hoc or a non-existent process generally leads to crummy products. Smaller systems can succeed from the dedication of a couple of overworked experts, but as things scale up in size heroics becomes less and less successful.

Yet it seems an awful lot of us don't know about basic software engineering rules. When talking to groups I usually ask how many participants have (and use) rules about the maximum size of a function. A basic rule of software engineering is to limit routines to a page or less. Yet only rarely does anyone raise their hand. Most admit to huge blocks of code, sometimes thousands of lines. Often this is a result of changes and revisions, of the code evolving over the course of time. Yet it's a practice that inevitably leads to problems.

By and large methodologies have failed. Most are too big, too complex, or too easy to thwart and subvert. I hold great hopes for UML, which seems to offer a way to build products that integrates hardware and software, and that is an intrinsic part of development from design to implementation. But UML will fail if management won't pay for quite extensive training, or toss the approach when panic reigns.

The FDA, FAA, and other agencies are slowing becoming aware of the perils of poor software, and have guidelines that can improve development. Britain's MISRA (Motor Industry Software Reliability Association) has guidelines for the safer use of C. They feel that we need to avoid certain constructs and use others in controlled ways to eliminate potential error sources. I agree. Encouragingly, some tool vendors (notably Tasking) offer compilers that can check code against the MISRA standard. This is a powerful aid to building better code.

I doubt, though, that any methodology or set of practices can, in the real world of schedule pressures and capricious management, lead to perfect products. The numbers tell the story. The very best use of code inspections, for example, will detect about 70% of the mistakes before testing begins. (However, inspections will find those errors very cheaply). That suggests that testing must pick up the other 30%. Yet studies show that often testing checks only about 50% of the software!

Sure, we can (and must) design better tests. We can, and should, use code coverage tools to insure every execution path runs. These all lead to much better products, but not to perfection. Because all of the code is known to have run doesn't mean that complex interactions between inputs won't lead to bizarre outputs. As the number of decision paths increases - as the code grows - the difficulty of creating comprehensive tests skyrockets.

When time to market dominates development, quality naturally slips. If low cost is the most important parameter, we can expect more problems to slip into the product.

Software is astonishingly fragile. One wrong bit out of a hundred million can bring a massive system down. It's amazing that things work as well as they do!

Perhaps the nature of engineering is that perfection itself is not really a goal. Products are as good as they have to be. Competition is a form of evolution that often does lead to better quality. In the 70s Japanese automakers, who had practically no US market share, started shipping cars that were reliable and cheap. They stunned Detroit, which was used to making a shoddy product which dealers improved and customers tolerated. Now the playing field has leveled, but at an unprecedented level of reliability.

Perfection may elude us, but we must be on a continual quest to find better ways to build our products. Wise developers will spend their entire careers engaged in the search.

Originally posted here.

For novel ideas about building embedded systems (both hardware and firmware), join the 35,000 engineers who subscribe to The Embedded Muse, a free biweekly newsletter. The Muse has no hype and no vendor PR. Click here to subscribe

by Jack Ganssle

jack@ganssle.com

Recently our electric toothbrush started acting oddly – differently from before. I complained to Marybeth who said, “I think it’s in the wrong mode.”

Really? A toothbrush has modes?

We in the embedded industry have created a world that was unimaginable prior to the invention of the microprocessor. Firmware today controls practically everything, from avionics to medical equipment to cars to, well everything.

And toothbrushes.

But we’re working too hard at it. Too many of us use archaic development strategies that aren’t efficient. Too many of us ship code with too many errors. That's something that can, and must, change.

Long ago the teachings of Deming and Juran revolutionized manufacturing. One of Deming's essential insights was that fixing defects will never lead to quality. Quality comes from correct design rather than patches applied on the production line. And focusing on quality lowers costs.

The software industry never got that memo.

The average embedded software project devotes 50% of the schedule to debugging and testing the code. It's stunning that half of the team’s time is spent finding and fixing mistakes.

Test is hugely important. But, as Dijkstra observed, testing can only prove the presence of errors, not the absence of bugs.

Unsurprisingly, and mirroring Deming's tenets, it has repeatedly been shown that a focus on fixing bugs will never lead to a quality product - all that will do is extend the schedule and insure defective code goes out the door.

Focusing on quality has another benefit: the project gets done faster. Why? That 50% of the schedule used to deal with bugs gets dramatically shortened. We shorten the schedule by not putting the bugs in in the first place.

High quality code requires a disciplined approach to software engineering - the methodical use of techniques and approaches long known to work. These include inspection of work products, using standardized ways to create the software, seeding code with constructs that automatically catch errors, and using various tools that scan the code for defects. Nothing that is novel or unexpected, nothing that a little Googling won't reveal. All have a long pedigree of studies proving their efficacy.

Yet only one team out of 50 makes disciplined use of these techniques.

What about metrics? Walk a production line and you'll see the walls covered with charts showing efficiency, defect rates, inventory levels and more. Though a creative discipline like engineering can't be made as routine as manufacturing, there are a lot of measurements that can and must be used to understand the team's progress and the product's quality, and to drive the continuous improvement we need.

Errors are inevitable. We will ship bugs. But we need a laser-like focus on getting the code right. How right? We have metrics; we know how many bugs the best and mediocre teams ship. Defect Removal Efficiency is a well-known metric used to evaluate quality of shipped code; it's the percentage of the entire universe of bugs found in a product that were removed prior to shipping (it's measured until 90 days after release). The very best teams, representing just 0.4% of the industry, eliminates over 99% of bugs pre-shipment. Most embedded groups only removed 95%.

Where does your team stand on this scale? Can one control quality if it isn’t measured?

We have metrics about defect injection rates, about where in the lifecycle they are removed, about productivity vs. any number of parameters and much more. Yet few teams collect any numbers.

Engineering without numbers isn’t engineering. It’s art.

Want to know more about metrics and quality in software engineering? Read any of Capers Jones’ books. They are dense, packed with tables of numbers, and sometimes difficult as the narrative is not engaging, but they paint a picture of what we can measure and how differing development activities effect errors and productivity.

Want to understand where the sometimes-overhyped agile methods make sense? Read Agile! by Bertrand Meyer and Balancing Agility and Discipline by Barry Boehm and Richard Turner.

Want to learn better ways to schedule a project and manage requirements? Read any of Karl Wiegers’ books and articles.

The truth is that we know of better ways to get great software done more efficiently and with drastically reduced bug rates.

When will we start?

Jack Ganssle has written over 1000 articles and six books about embedded systems, as well as one about his sailing fiascos. He has started and sold three electronics companies. He welcomes dialog at jack@ganssle.com or at www.ganssle.com.