A smart, highly optimized distributed neural network, based on Intel Edison "Receptive" Nodes

Training ‘complex multi-layer’ neural networks is referred to as deep-learning as these multi-layer neural architectures interpose many neural processing layers between the input data and the predicted output results – hence the use of the word deep in the deep-learning catchphrase.

While the training procedure of large scale network is computationally expensive, evaluating the resulting trained neural network is not, which explains why trained networks can be extremely valuable as they have the ability to very quickly perform complex, real-world pattern recognition tasks on a variety of low-power devices.

These trained networks can perform complex pattern recognition tasks for real-world applications ranging from real-time anomaly detection in Industrial IoT to energy performance optimization in complex industrial systems. The high-value, high accuracy recognition (sometimes better than human) trained models have the ability to be deployed nearly everywhere, which explains the recent resurgence in machine-learning, in particular in deep-learning neural networks.

These architectures can be efficiently implemented on Intel Edison modules to process information quickly and economically, especially in Industrial IoT application.

Our architectural model is based on a proprietary algorithm, called Hierarchical LSTM, able to capture and learn the internal dynamics of physical systems, simply observing the evolution of related time series.

To train efficiently the system, we implemented a greedy, layer based parameter optimization approach, so each device can train one layer at a time, and send the encoded feature to the upper level device, to learn higher levels of abstraction on signal dinamic.

Using Intel Edison as layers "core computing units", we can perform higher sampling rates and frequent retraining, near the system we are observing without the need of a complex cloud architecture, sending just a small amount of encoded data to the cloud.

Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. It will play a big part in the IoT. From our friends at R2D3 is a very interesting visual introduction to machine learning. Check it out here. 

As we move towards widespread deployment of sensor-based technologies, three issues come to the fore: (1) many of the these applications will need machine learning to be localized and personalized, (2) machine learning needs to be simplified and automated, and (3) machine learning needs to be hardware-based. 

Beginning of the era of personalization of machine learning

Imagine a complex plant or machinery being equipped with all kinds of sensors to monitor and control its performance and to predict potential points of failure. Such plants can range from an oil rig out in the ocean to an automated production line. Or such complex plants can be human beings, perhaps millions of them, who are being monitored with a variety of devices in a hospital or at home. Although we can use some standard models to monitor and compare performance of these physical systems, it would make more sense to either rebuild these models from scratch or adjust them to individual situations. This would be similar to what we do in economics. Although we might have some standard models to predict GDP and other economic variables, we would need to adjust each one of them to individual countries or regions to take into account their individual differences. The same principle of adjustment to individual situations would apply to physical systems that are sensor-based. And, similar to adjusting or rebuilding models of various economic phenomena, the millions of sensor-based models of our physical systems would have to be adjusted or rebuilt to account for differences in plant behavior. We are, therefore, entering an era of personalization of machine learning at a scale that we have never imagined before. The scenario is scary because we wouldn’t have the resources to pay attention to these millions of individual models. Cisco projects 50 billion devices to be connected by 2020 and the global IoT market size to be over $14 trillion by 2022 [1, 2].

The need for simplification and automation of machine learning technologies 

If this scenario of widespread deployment of personalized machine learning is to play out, we absolutely need automation of machine learning to the extent that requires less expert assistance. Machine learning cannot continue to depend on high levels of professional expertise.  It has to be simplified to be similar to automobiles and spreadsheets where some basic training at a high school can certify one to use these tools. Once we simplify the usage of machine learning tools, it would lead to widespread deployment and usage of sensor-based technologies that also use machine learning and would create plenty of new jobs worldwide. Thus, simplification and automation of machine learning technologies is critical to the economics of deployment and usage of sensor-based systems. It should also open the door to many new kinds of devices and technologies.

The need for hardware-based localized machine learning for "anytime, anywhere" deployment and usage 

Although we talk about the Internet of Things, it would simply be too expensive to transmit all of the sensor-based data to a cloud-based platform for analysis and interpretation. It would make sense to process most of the data locally. Many experts predict that, in the future, about 60% of the data would be processed at the local level, in local networks - most of it may simply be discarded after processing and only some stored locally. There is a name for this kind of local processing – it’s called “edge computing” [3].

The main characteristics of data generated by these sensor-based systems are: high-velocity, high volume, high-dimensional and streaming. There are not many machine learning technologies that can learn in such an environment other than hardware-based neural network learning systems. The advantages of neural network systems are: (1) learning involves simple computations, (2) learning can take advantage of massively parallel brain-like computations, (3) they can learn from all of the data instead of samples of data, (4) scalability issues are non-existent, and (4) implementations on massively parallel hardware can provide real-time predictions in micro seconds. Thus, massively parallel neural network hardware can be particularly useful with high velocity streaming data in these sensor-based systems. Researchers at Arizona State University, in particular, are working on such a technology and it is available for licensing [4].

Conclusions

Hardware-based localized learning and monitoring will not only reduce the volume of Internet traffic and its cost, it will also reduce (or even eliminate) the dependence on a single control center, such as the cloud, for decision-making and control. Localized learning and monitoring will allow for distributed decision-making and control of machinery and equipment in IoT.

We are gradually moving to an era where machine learning can be deployed on an “anytime, anywhere” basis even when there is no access to a network and/or a cloud facility.

References

1. Gartner (2013). "Forecast: The Internet of Things, Worldwide, 2013."

         https://www.gartner.com/doc/2625419/forecast-internet-things-worldwide-

     2. 10 Predictions for the Future of the Internet of Things

     3. Edge Computing

     4. Neural Networks for Large Scale Machine Learning