Recovering from a system failure or a software glitch can be no easy task.  The longer the fault occurs the harder it can be to identify and recover.  The use of an external watchdog is an important and critical tool in the embedded systems engineer toolbox.  There are five tips that should be taken into account when designing a watchdog system.

Tip #1 – Monitor a heartbeat

The simplest function that an external watchdog can have is to monitor a heartbeat that is produced by the primary application processor.  Monitoring of the heartbeat should serve two distinct purposes.  First, the microcontroller should only generate the heartbeat after functional checks have been performed on the software to ensure that it is functioning.  Second, the heartbeat should be able to reveal if the real-time response of the system has been jeopardized.

Monitoring the heartbeat for software functionality and real-time response can be done using a simple, “dumb” external watchdog.  The external watchdog should have the capability to assign a heartbeat period along with a window that the heartbeat must appear within.  The purpose of the heartbeat window is to allow the watchdog to detect that the real-time response of the system is compromised.  In the event that either functional or real-time checks fail the watchdog then attempts to recover the system through a reset of the application processor.

Tip #2 – Use a low capability MCU

External watchdogs that can be to monitor a heartbeat are relatively low cost but can severely limit the capabilities and recovery possibilities of the watchdog system.  A low capability microcontroller can cost nearly the same amount as an external watchdog timer so why not add some intelligence to the watchdog and use a microcontroller.  The microcontroller firmware can be developed to fulfill the windowed heartbeat monitoring with the addition of so much more.  A “smart” watchdog like this is sometimes referred to as a supervisor or safety watchdog and has actually been used for many years in different industries such as automotive.  Generally a microcontroller watchdog has been reserved for safety critical applications but given the development tools and the cost of hardware it can be cost effective in other applications as well.

Tip #3 – Supervise critical system functions

The decision to use a small microcontroller as a watchdog opens nearly endless possibilities of how the watchdog can be used.  One of the first roles of a smart watchdog is usually to supervise critical system functions such as a system current or sensor state.  One example of how a watchdog could supervise a current would be to take an independent measurement and then provide that value to the application processor.  The application processor could then compare its own reading to that of the watchdog.  If there were disagreement between the two then the system would execute a fault tree that was deemed to be appropriate for the application.

Tip #4 – Observe a communication channel

Sometimes an embedded system can appear to be operating as expected to the watchdog and the application processor but from an external observer be in a non-responsive state.  In such cases it can be useful to tie the smart watchdog to a communication channel such as a UART.  When the watchdog is connected to a communication channel it not only monitor channel traffic but even commands that are specific to the watchdog.  A great example of this is a watchdog designed for a small satellite that monitors radio communications between the flight computer and ground station.  If the flight computer becomes non-responsive to the radio, a command could be sent to the watchdog that is then executed and used to reset the flight computer.

Tip #5 – Consider an externally timed reset function

The question of who is watching the watchdog is undoubtedly on the minds of many engineers when using a microcontroller for a watchdog.  Using a microcontroller to implement extra features adds some complexity and a new software element to the system.  In the event that the watchdog goes off into the weeds how is the watchdog going to recover? One option would be to use an external watchdog timer that was discussed earlier.  The smart watchdog would generate a heartbeat to keep itself from being reset by the watchdog timer.  Another option would be to have the application processor act as the watchdog for the watchdog.  Careful thought needs to be given to the best way to ensure both processors remain functioning as intended.

Conclusion

The purpose of the smart watchdog is to monitor the system and the primary microcontroller to ensure that they operate as expected.  During the design of a system watchdog it can be very tempting to allow the number of features supported to creep.  Developers need to keep in mind that as the complexity of the smart watchdog increases so does the probability that the watchdog itself will contain potential failure modes and bugs.  Keeping the watchdog simple and to the minimum necessary feature set will ensure that it can be exhaustively tested and proven to work.

Originally Posted here

5 Tips for Expanding your Embedded Skills

by Jack Ganssle

jack@ganssle.com

Recently our electric toothbrush started acting oddly – differently from before. I complained to Marybeth who said, “I think it’s in the wrong mode.”

Really? A toothbrush has modes?

We in the embedded industry have created a world that was unimaginable prior to the invention of the microprocessor. Firmware today controls practically everything, from avionics to medical equipment to cars to, well everything.

And toothbrushes.

But we’re working too hard at it. Too many of us use archaic development strategies that aren’t efficient. Too many of us ship code with too many errors. That's something that can, and must, change.

Long ago the teachings of Deming and Juran revolutionized manufacturing. One of Deming's essential insights was that fixing defects will never lead to quality. Quality comes from correct design rather than patches applied on the production line. And focusing on quality lowers costs.

The software industry never got that memo.

The average embedded software project devotes 50% of the schedule to debugging and testing the code. It's stunning that half of the team’s time is spent finding and fixing mistakes.

Test is hugely important. But, as Dijkstra observed, testing can only prove the presence of errors, not the absence of bugs.

Unsurprisingly, and mirroring Deming's tenets, it has repeatedly been shown that a focus on fixing bugs will never lead to a quality product - all that will do is extend the schedule and insure defective code goes out the door.

Focusing on quality has another benefit: the project gets done faster. Why? That 50% of the schedule used to deal with bugs gets dramatically shortened. We shorten the schedule by not putting the bugs in in the first place.

High quality code requires a disciplined approach to software engineering - the methodical use of techniques and approaches long known to work. These include inspection of work products, using standardized ways to create the software, seeding code with constructs that automatically catch errors, and using various tools that scan the code for defects. Nothing that is novel or unexpected, nothing that a little Googling won't reveal. All have a long pedigree of studies proving their efficacy.

Yet only one team out of 50 makes disciplined use of these techniques.

What about metrics? Walk a production line and you'll see the walls covered with charts showing efficiency, defect rates, inventory levels and more. Though a creative discipline like engineering can't be made as routine as manufacturing, there are a lot of measurements that can and must be used to understand the team's progress and the product's quality, and to drive the continuous improvement we need.

Errors are inevitable. We will ship bugs. But we need a laser-like focus on getting the code right. How right? We have metrics; we know how many bugs the best and mediocre teams ship. Defect Removal Efficiency is a well-known metric used to evaluate quality of shipped code; it's the percentage of the entire universe of bugs found in a product that were removed prior to shipping (it's measured until 90 days after release). The very best teams, representing just 0.4% of the industry, eliminates over 99% of bugs pre-shipment. Most embedded groups only removed 95%.

Where does your team stand on this scale? Can one control quality if it isn’t measured?

We have metrics about defect injection rates, about where in the lifecycle they are removed, about productivity vs. any number of parameters and much more. Yet few teams collect any numbers.

Engineering without numbers isn’t engineering. It’s art.

Want to know more about metrics and quality in software engineering? Read any of Capers Jones’ books. They are dense, packed with tables of numbers, and sometimes difficult as the narrative is not engaging, but they paint a picture of what we can measure and how differing development activities effect errors and productivity.

Want to understand where the sometimes-overhyped agile methods make sense? Read Agile! by Bertrand Meyer and Balancing Agility and Discipline by Barry Boehm and Richard Turner.

Want to learn better ways to schedule a project and manage requirements? Read any of Karl Wiegers’ books and articles.

The truth is that we know of better ways to get great software done more efficiently and with drastically reduced bug rates.

When will we start?

Jack Ganssle has written over 1000 articles and six books about embedded systems, as well as one about his sailing fiascos. He has started and sold three electronics companies. He welcomes dialog at jack@ganssle.com or at www.ganssle.com.