The range and depth of applications dependent on IoT sensors continues to swell – from collecting real-time data on the floors of smart factories, to monitoring supply chains, to enabling smart cities, to tracking our health and wellness behaviors. The networks utilizing IoT sensors are capable of providing critical insights into the innerworkings of vast systems, empowering engineers to take better informed actions and ultimately introduce far greater efficiency, safety, and performance into these ecosystems. 

One outsized example of this: IoT sensors can support predictive maintenance by detecting data anomalies that deviate from baseline behavior and that suggest potential mechanical failures – thus enabling an IoT-fueled organization to repair or replace components before issues become serious or downtime occurs. Because IoT sensors provide such a tremendous amount of data pertaining to each particular piece of equipment when in good working condition, anomalies in that same data can clearly indicate issues.

Looking at this from a data science perspective, anomalies are rare events which cannot be classified using currently available data examples; anomalies can also come from cybersecurity threats, or fraudulent transactions. It is therefore vital to the integrity of IoT systems to have solutions in place for detecting these anomalies and taking preventative action. Anomaly detection systems require a technology stack that folds in solutions for machine learning, statistical analysis, algorithm optimization, and data-layer technologies that can ingest, process, analyze, disseminate, and store streaming data from myriad IoT sources.

But that said, actually creating an IoT anomaly detection system remains especially challenging given the large-scale nature inherent to IoT environments, where millions or even billions of data events occur daily. To be successful, the data-layer technologies supporting an IoT anomaly detection system must be capable of meeting the scalability, computational, and performance needs fundamental to a successful IoT deployment.

I don’t work for a company that sells anomaly detection, but I – along with colleagues on our engineering team – recently created an experimental anomaly detection solution to see if it could stand up to the specific needs of large-scale IoT environments using pure open source data-layer technologies (in their 100% open source form). The testing utilized Apache Kafka and Apache Cassandra to produce an architecture capable of delivering the features required for IoT anomaly detection technology from the perspectives of scalability, performance, and realistic cost effectiveness. In addition to matching up against these attributes, Kafka and Cassandra are highly compatible and complementary technologies that lend themselves to being used in tandem. Not fully knowing what to expect, we went to work.

In our experiment, Kafka, Cassandra, and our anomaly detection application are combined in a Lambda architecture, with Kafka and our streaming data pipeline serving as the speed layer, and Cassandra acting as the batch and serving layer. (See full details on GitHub, here.) Kafka enables rapid and scalable ingestion of streaming data, while leveraging a “store and forward” technique that acts as a buffer for ensuring that Cassandra is not overwhelmed when data surges spike. At the same time, Cassandra provides a linearly scalable, write-optimized database well-suited to storing the high-velocity streaming data produced by IoT environments. The experiment also leveraged Kubernetes on AWS EKS, to provide automation for the experimental application’s provisioning, deployment, and scaling. 

We progressed through the development of our anomaly detection application test using an incremental approach, continually optimizing capabilities, monitoring, debugging, refining, and so on. Then we tested scale: 19 billion real-time events per day were processed, enough to satisfy the requirements of most any IoT use case out there. Achieving this result meant scaling out the application from three to 48 Cassandra nodes, while utilizing 574 CPU cores across Cassandra, Kafka, and Kubernetes clusters. It also included maintaining a peak 2.3 million writes per second into Kafka, for a sustainable 220,000 anomaly checks per second.

In completing this experiment, we’ve demonstrated a method that IoT-centric organizations can use for themselves in building a highly scalable, performant, and affordable anomaly detection application for IoT use cases, fueled by leveraging the unique advantages offered by pure open source Apache Kafka and Cassandra at the all-important data layer.

Considering that the IoT is in its infancy and due to the last years wasted in predictions that have not been fulfilled, in disappointing statistics of successful projects and with most companies without clear strategies, it is normal to think that R & D is today so necessary for boost and accelerate this increasingly sceptical market.

R&D should be an essential part of bringing innovation to any company via IoT projects. And though we can all agree how important R&D is, it requires a great deal of experience, senior experts, and specific toolsets—resources that not every company can say they have handy.

However, there is a risk when deriving the strategic decisions that the executive directors consider to be technological towards the R & D departments. Many times, oblivious to the reality of the markets, those responsible for R & D with the invaluable aid of the subsidies of the different Administrations, they launch to develop products and technologies for problems that do not exist, just for the fact of obtaining recognition or to continue living without pressures of the Top Management. I am enemy of granted subsidies granted most of the time by unqualified Administration organisms that does not understand that need to prevail the utility, the business model, the business case and the commercialization over the innovation that R & D said to be developed.

Now, if we ask the sellers of IoT technology, products and services, they may not be so happy with the idea of having to talk with the R & D areas instead of with other areas of the company more likely to buy. Most time, R &D departments decide to do it themselves. Vendors know, that with great probability, they will not to close deals due to lack of budget of the R &D or the low visibility of this area by the rest of the departments of the company.

The Importance of R&D for the Internet of Things

Innovation in IoT is a major competitive differentiator. See below some advices to have a decisive advantage over competitors:

• IoT-focused companies need to invest in R&D to keep up with the rapidly changing and expanding market. It is important that an organization’s R&D iteration turn times are quick, otherwise the company is not going to be able to keep pace with the expected IoT market growth. However, it’s not enough to simply speed up R&D—innovative IoT firms, both start-ups and established companies, must also make sure their R&D processes are extremely reliable.
• You can’t solve R&D speed issues just by increasing budget.
• Executives must maintain strong, steady communication with R&D regarding the department’s priorities over a particular time frame and how progress will be measured.
• Guidelines are invaluable: The more structured and streamlined R&D procedures are, the better IoT companies will be able to move from conception to delivery.
• Design innovative IoT products but accelerate time to market.
• Internal collaboration: R&D team should share real-time data across internal departments to spur intelligent product design
• External collaboration: Connect with customers and partners to ensure success
• Differentiation: Drive overall business value with IoT.

Outsource or not Outsource R & D for your IoT project

Just like any other technology, IoT products and solutions require thorough research and development, and it better be done by professionals. Despite the noise generated by analysts and companies around the IoT, the reality is that there have not been many IoT projects and therefore it is not easy to find good professionals with proven experience in IoT to hire.

When I think of Outsourcing IoT projects, Eastern European and Indian companies immediately come to my mind. No doubt because the R & D talent seems to be cheaper there. Spain could also be a country to outsource IoT, but at the moment I do not see it.

The benefits of Outsourcing R&D for IoT Projects:

• Expertise and an Eye for Innovation
• Bring an IoT Project to Market Faster
• Optimize Your Costs
• Control and Manage Risks

I am not sure about the quality of most of these companies or the experience of their teams in the development of IoT products or in the implementation of IoT projects, but there is no doubt that there are benefits to Outsource R & D for some IoT Projects. You should select any of these companies after a careful evaluation.

Recommendation: Do not stop your IoT projects if you do not have the skills and professionals in house. Luckily, there are companies who offer outsourcing R&D for IoT projects.

Note: Remember I can help you to identify and qualify the most suitable Outsource R&D for your IoT project.

Spain is not different in R & D for IoT

I have not believed in R & D in Spain for years. There are exceptions without a doubt, but it seems evident that the prosperity and welfare of Spain is not due to our R & D. Fortunately we have sun and beach and a lot of brick to put in houses that are not sold because of high prices and low wages.

With the entry into the EU, I thought that we had great markets open to us. I was also optimistic that we would have great opportunities in the Latin American market, thanks to the fact that our research and development capacity would have been consolidated effectively in our companies and universities because it would be profitable and worldwide recognized.

But it has not been that way. The technology developed in Spain and more specifically that relating to the IoT has little chance of being commercialized in France, Germany and not to mention in the UK. If we add the development gap of the countries of South America and that our local market is averse to technological risk, it is difficult to flourish R & D in IoT or Industry 4.0 here in our lovely Spain.

That does not mean that we do not have public R & D budgets for these areas. What happens is that the same thing that happened during the last 30 years has happened. The incentives and aids are few and for the most part used to finance large companies with little return to society. There is no rigorous control of the aid granted and, above all, there is no plan to encourage the local and global marketing of the products developed with the talent of our scientists and researchers.

I have stopped believing and trusting in our successive Governments for the change in R & D but there are exceptions that are worthwhile to follow and work with them. For this reason, I continue help them demonstrate that “SPAIN CAN BE DIFFERENT”.

Key Takeaway

After years of unfulfilled expectations, companies are sceptical of the potential growth of the IoT market or the benefits in their business. R&D department can be a cure to boost IoT initiatives but also a poison to kill IoT initiatives.

IoT may have started in R&D, but their benefits don’t have to end there. To drive overall business value, it’s important to share IoT data – both internally and externally. Facilitating open collaboration, discovering new ways to innovate products, and accelerating time to market, you can differentiate R&D and your business.

As fast turn times and reliability becomes a focal part of companies’ R&D processes, these companies will be well-positioned to thrive within the IoT market.

Thanks for your Likes and Comments