Provisioning, managing and securing devices in an IoT product requires careful planning at the very start of the process. Rigorous evaluation of options, then a Proof of Concept helps determine the right solution. Once the POC has been approved, the IoT product moves to production. Then the real fun starts and many strategic considerations come into play. We can list them as follows:

• Robust and secure OTA software updates

• Security by design

• Scalability

• Automation

• Remote terminal management

• Device configuration, monitoring & troubleshooting

Robust and secure OTA software updates

Robust and secure OTA software updates are essential for keeping IoT devices secure as the software on these devices will become outdated during their lifetime and vulnerabilities are certain to arise if left in their initial states. Therefore a secure, risk-tolerant, and efficient update mechanism must be at the core of each product development team from the inception of the project to the end of its life.

How about a homegrown solution?

Homegrown solutions are less likely to be best-of-breed, can be hard to scale, can suffer from over customisation and scope creep, come at an inherently high cost and can be left in trouble if the star developers behind their creation suddenly jump ship and leave the organisation.  They also often lack the requirements needed to ensure security and robustness of software updates. Various open source solutions exist, but none provide an end-to-end solution and lack the overall functionality to make them enterprise-grade. Generic public cloud IoT stacks wish to cater to the entire IoT value chain but fail to deliver a purpose-built solution for software updates. Proprietary and platform solutions cause lock-in to specific cloud infrastructure, operating system, or development tools.

The common thread among all of these solutions is the lack of a fully optimized end-to-end OTA software update and device management infrastructure that can minimize risk, increase efficiency and enhance security and uptime.

Security by design

A device security breach incident can interrupt operations, damage systems, and negatively impact both virtual and physical processes. This translates into unhappy customers and lost business. As Colin Duggan, the Founder and CEO at BG Networks says in an interview with the Device Chronicle, “It is difficult to add security after the design has been completed. There are a number of reasons for this. Embedded systems have limited MHz, memory, and limitations of network interfaces on embedded processors. Security features can be added after the fact but usually will not close off all the vulnerabilities.” That is why it is so important to ensure security by design, in the very early stages of the product’s lifecycle.

IoT product security should be approached holistically with a framework that addresses the people, devices and process. To help IoT professionals make the right decisions concerning their product development, we designed a simple framework based on these factors and called it the Triangle of Trust:

Scalability

There’s a significant difference between managing a small number of embedded devices and having thousands or even millions of devices deployed in the field. Microsoft’s new IoT Signals report found lack of scalability as a leading cause for IoT project failures. Complexity is one of the greatest scalability issues. As such, choosing the right solution with the right architecture is important to safeguard the long-term management viability of your fleet of connected devices. More on the topic of IoT scalability can be read here.

Automation

When one of the arms of the Triangle of Trust fails, the other two are endangered. To prevent any risks arising from human mistakes, automating some of the processes is a solution that might save your business thousands of dollars. Mender.io is an OTA software update manager for Linux-based embedded devices, and it also offers a wide range of automations to securely manage these devices. One of the features that Mender offers is automatic retry of failed device deployments. Deployments to devices might fail for various intermittent reasons like loss of power, network or device usage. Automatic retry upon failures reduces device deployment error rates up to 90%. This translates to time and money savings managing deployments, and also leads to customers receiving the updates faster.

Remote Management

Remote management is a necessity for any kind of embedded device. Any company rolling out its IoT products needs to have control of its systems from a central location. SSH, secure tunneling and remote terminal access is preferred by service providers to VPN access as they can assure their customers of security when accessing and troubleshooting devices. Furthermore, the management involves grouping and accessing embedded devices, provisioning, configuring, and monitoring remotely and securely.

Seeing the necessity for not only secure over-the-air processes, but also for reliable ways of monitoring, provisioning, configuring, grouping, and accessing the embedded devices, the team behind Mender decided to expand their offering by the mentioned remote management features. Mender is open source software meaning there are many contributors to make it better and support a variety of customer hardware and software such as NVIDIA Jetson and NXP's family of iMX processors. It provides flexibility in choosing your infrastructure, software, and hardware from prototyping to production which means there is no vendor lock-in. Mender supports all device software updates from a full disk image to application updates with the freedom to customize the update and installation process to fit your workflow. It is also integrated with Google Cloud and Microsoft Azure IoT for easy device authentication. 

Device configuration, troubleshooting and monitoring

A proper device management set up should never be overlooked. Robust and secure device management is a necessary cornerstone for an IoT product and therefore you need to find a high quality solution. Once you deploy thousands or millions of devices into the field you’ll need to be able to configure them properly, gather the data, and quickly troubleshoot any arising problems. Many organisations treat these capabilities as an afterthought. Engineers realize that they need some kind of device management solution right before their deadlines and product releases, which results in rushed fixes being made, that may have serious implications for the robustness and security of connected devices.

Conclusion

In order to roll out a successful, secure, and robust IoT product a few things have to be taken into consideration before the release. To ensure security by design from the earliest stages of the product life cycle, the team behind the IoT product needs to find a solution for deploying secure and robust OTA updates, remotely monitor, configure, and troubleshoot the devices, and automate necessary processes in order to avoid human-made mistakes.

From a salt shaker with a built-in speaker to smart water devices that bring clean water to communities with weak infrastructure, connected devices are increasingly advancing into all areas of our lives. But more connectivity brings more possibilities for crippling issues that can impact product development, operations, and maintenance. IoT developers must consider how to plan for firmware architecture that leads to a better, stickier product.

Competition among connected device manufacturers is swelling in every corner of the industry, and user patience for clunky products won’t get the benefit of the doubt that developers might otherwise have had in the IoT’s nascent days. As users become more dependent on connected devices, consumer demands that those devices consistently function well - and securely - become the expectation. There remains, of course, work to be done: a quick Google search reveals stories like the Fitbit firmware update that destroyed the device battery, or the Tesla key fobs that could be overwritten and hijacked until a patch was rolled out.    

These stories underscore that the IoT ecosystem’s connected nature requires that hardware developers approach product development differently - and take firmware updates seriously. It used to be that developers could write static firmware for specific device use cases or commoditized products and, once released, have no further interaction or engagement with the product. That system no longer works. To have a successful product, IoT device manufacturers need to invest in design and in firmware development equally.

Whether it’s BLE on phones or LTE or Zigby and other mesh networks, IoT devices are connected, regularly transmitting sensitive and personal data to and from the cloud. The near limitless reach of modern connected devices across all areas of our lives, paired with the high price point of most IoT devices underscores that IoT developers must have a plan (and not an after-the-fact reaction) for firmware maintenance. Putting that plan in motion requires three considerations:

Device monitoring

Ubiquitous connectivity brings with it major challenges, but it also brings opportunities. Among other things, it allows automated device health monitoring. The typical process of releasing a product relies on users’ reporting a problem and requiring them to physically return the device to be evaluated, repaired, and returned. Simply put, this is a huge waste of money and time, and it also risks frustrating the customer to the point of losing them entirely. Using customers as your testers is simply a terrible business decision. (Maybe you could get away with it if you were the only game in town, but IoT device makers don’t have that luxury anymore). Automated device monitoring is the solution. By regularly analyzing the health of devices and flagging potential problems immediately, a monitoring system can help device makers catch and fix issues in hours that would have otherwise taken them weeks to root cause. Designing embedded systems with such capabilities gives critical observability into performance, stability, and overall health - either of a single device or of a fleet of millions. 

Repair

Shipping products that require an update or patch is inevitable for even the most talented and thorough teams. Just ask NASA. While no one can avoid updates entirely, it is possible to detect fleet-wide issues and solve them without burdening users. The key is to roll out updates incrementally, starting with a small number of devices and ramping up over time. This limits the impact of any new issues and insulates most of your users from the churn of getting a few bugfix releases in a row.  Another good option is to implement an A/B update system if you have enough flash memory. This allows your device to download an update in the background with no user impact and simply prompts the user to reboot once the update is ready. Fast and simple update flows like A/B updates are key to compliance, and prevent too much fragmentation across your fleet. Last but not least, it is important to pair regular updates with a monitoring system so you can quickly identify problems with the update, and rollouts can be paused or aborted altogether.

Building with security in mind

The ubiquity of IoT devices has accelerated customer demands for robust device security in lockstep, with regulatory bodies becoming more serious (and punitive) about security requirements and standards. For those building smart devices, I would offer these principles as table stakes for security: 

1. Devices must be updateable. 
2. Trusted boot is no longer optional. You need a chain of trust to control the firmware running on your device.
3. Rotate secrets and don’t use a master secret. Whether that means a set of encryption keys or other secrets to make devices functional, they must be dynamically changed, so the compromise of one device does not lead to the compromise of others. 

Software teams have long embraced iterative processes, and IoT device developers can learn much from this process. Focusing on firmware architecture that is responsive, observable, and proactive, lets device manufacturers ship a better product and create a happier customer base.

François Baldassari is the Founder and CEO of Memfault, a cloud-based observability platfrom for hardware devices. Prior to Memfault, François worked on developer infrastructure initiatives at Pebble and Oculus.

Guest post by Jeffrey Lee.

In the early days of IoT, updating remote devices often caused intermittent disruption and performance degradation. As IoT platforms have matured, they have embraced a novel way to remotely and reliably update connected devices with little to no disruption: over-the-air (OTA) firmware updates.

Over-the-air firmware updates refers to the practice of remotely updating the code on an embedded device. The embedded hardware must be built with OTA functionality for this mechanism to work.

Why OTA Firmware?

Prior to OTA updates, you had to go out and retrieve the device, take it apart, connect it to your computer, reprogram it, put the device back together, and then take the device back.

However, this process is overly burdensome and unscalable for companies who have devices out on the field. Although, it hasn’t stopped some from trying . . .

• In 2015, Chrysler was criticized for patching a software vulnerability via mailed USB drives. Chrysler’s method put many consumers at risk because the USB drives could be intercepted, modified, and resent.

On the other hand,

• In 2016, Tesla drivers woke up to find substantial new features to their car after the company sent out an OTA firmware update. Consumers could now self-park their cars without having to manually update their vehicles.

You tell us which is the better headline.

OTA Firmware Benefits

• Bugs and product behavior can be continuously improved even after the device is in the hands of your consumers.
• Companies can test new features by sending updates to one or multiple devices.
• Companies can save costs by managing the firmware across their fleet of devices from a seamless, unified interface.
• Developers can deploy frequently and reliably, knowing that products will stay functional as updates are released.
• OTA firmware augments scalability by adding new features and infrastructure to products after they are released.

OTA Firmware & Device Management

To send out OTA firmware updates, you need a device management system that can interface with microprocessors and local software on IoT devices. This is complicated to build because few companies have an IoT software and hardware ecosystem that can process OTA firmware updates and manage remote devices.

Implementing OTA firmware updates

There are two options companies can take: you can build your own OTA firmware system or buy a managed OTA firmware system. For the build route, it is imperative that you research, plan, and consult domain experts to help you add OTA functionality to your hardware and software. Implementing the proper industry encryptions, finding the compatible hardware/software, and finding domain experts who can actually help you will be some of your biggest concerns.

However, due to the complexities of transmitting of the data and security concerns, you could harness a pre-built managed platform solution like Particle.

Getting Started with Particle and OTA firmware

Particle is a full stack IoT platform that offers the hardware and software tools to connect everyday electronics to the internet. Part of this platform, Particle cloud and console, also allows consumers to control fleets of devices and products with wireless firmware updates. Here are some of the benefits of using Particle for OTA firmware updates:

• Future-proof your products knowing that Particle is taking care of the infrastructure, hardware, and software.
• OTA firmware updates are sent in chunks so your device won’t brick. If your device loses connection during the update process, it’ll just resume when the connection comes back online.
• Firmware updates are delivered quickly because the update is just sent to the application layer and not the system layer. Particle only pushes parts of the application that have changed to the device.
• Easily scale from sending OTA firmware updates from 1 to 1,000 devices without hardware scalability or software issues.
• Test application updates by sending firmware updates to one or a controlled group of devices.
• Deliver updates securely knowing all communication channels between the device and Particle cloud are fully encrypted and authorized.
• Document each release throughly via Particle console to provide your team a comprehensive picture of what has changed in each version.
• Devices can be set into safe mode so it doesn’t execute any application code, which can be useful if new application code contains bugs that stop the device from connecting to the cloud.

All and all

OTA firmware is the critical driver for IoT success because it is powering the reliability and scalability of connected devices. Companies must decide whether building their own OTA firmware system is worth the time and potential costs, or if purchasing a platform that has OTA firmware functionality is a more efficient and effective way to update remote wireless devices.

This post originally appeared here.