The range and depth of applications dependent on IoT sensors continues to swell – from collecting real-time data on the floors of smart factories, to monitoring supply chains, to enabling smart cities, to tracking our health and wellness behaviors. The networks utilizing IoT sensors are capable of providing critical insights into the innerworkings of vast systems, empowering engineers to take better informed actions and ultimately introduce far greater efficiency, safety, and performance into these ecosystems. 

One outsized example of this: IoT sensors can support predictive maintenance by detecting data anomalies that deviate from baseline behavior and that suggest potential mechanical failures – thus enabling an IoT-fueled organization to repair or replace components before issues become serious or downtime occurs. Because IoT sensors provide such a tremendous amount of data pertaining to each particular piece of equipment when in good working condition, anomalies in that same data can clearly indicate issues.

Looking at this from a data science perspective, anomalies are rare events which cannot be classified using currently available data examples; anomalies can also come from cybersecurity threats, or fraudulent transactions. It is therefore vital to the integrity of IoT systems to have solutions in place for detecting these anomalies and taking preventative action. Anomaly detection systems require a technology stack that folds in solutions for machine learning, statistical analysis, algorithm optimization, and data-layer technologies that can ingest, process, analyze, disseminate, and store streaming data from myriad IoT sources.

But that said, actually creating an IoT anomaly detection system remains especially challenging given the large-scale nature inherent to IoT environments, where millions or even billions of data events occur daily. To be successful, the data-layer technologies supporting an IoT anomaly detection system must be capable of meeting the scalability, computational, and performance needs fundamental to a successful IoT deployment.

I don’t work for a company that sells anomaly detection, but I – along with colleagues on our engineering team – recently created an experimental anomaly detection solution to see if it could stand up to the specific needs of large-scale IoT environments using pure open source data-layer technologies (in their 100% open source form). The testing utilized Apache Kafka and Apache Cassandra to produce an architecture capable of delivering the features required for IoT anomaly detection technology from the perspectives of scalability, performance, and realistic cost effectiveness. In addition to matching up against these attributes, Kafka and Cassandra are highly compatible and complementary technologies that lend themselves to being used in tandem. Not fully knowing what to expect, we went to work.

In our experiment, Kafka, Cassandra, and our anomaly detection application are combined in a Lambda architecture, with Kafka and our streaming data pipeline serving as the speed layer, and Cassandra acting as the batch and serving layer. (See full details on GitHub, here.) Kafka enables rapid and scalable ingestion of streaming data, while leveraging a “store and forward” technique that acts as a buffer for ensuring that Cassandra is not overwhelmed when data surges spike. At the same time, Cassandra provides a linearly scalable, write-optimized database well-suited to storing the high-velocity streaming data produced by IoT environments. The experiment also leveraged Kubernetes on AWS EKS, to provide automation for the experimental application’s provisioning, deployment, and scaling. 

We progressed through the development of our anomaly detection application test using an incremental approach, continually optimizing capabilities, monitoring, debugging, refining, and so on. Then we tested scale: 19 billion real-time events per day were processed, enough to satisfy the requirements of most any IoT use case out there. Achieving this result meant scaling out the application from three to 48 Cassandra nodes, while utilizing 574 CPU cores across Cassandra, Kafka, and Kubernetes clusters. It also included maintaining a peak 2.3 million writes per second into Kafka, for a sustainable 220,000 anomaly checks per second.

In completing this experiment, we’ve demonstrated a method that IoT-centric organizations can use for themselves in building a highly scalable, performant, and affordable anomaly detection application for IoT use cases, fueled by leveraging the unique advantages offered by pure open source Apache Kafka and Cassandra at the all-important data layer.

While IoT has become more of a reality than just an industrial buzzword, what made it impactful among the masses is its ability to build “Smart” solutions.

The IoT based home automation or smart homes endow us with more security, better control of our assets, and cost savings through judicious and efficient use of energy resources such as water & electricity, and real time monitoring.

The IoT enabled smart home solutions are of great help in preventing property damage through theft, water leakage or flood, events of fire break out  – to name a few.

The Basics/Fundamentals of an IoT smart home solution

While a smart home application consists of a set of sensors, gateways, networking channels, cloud framework and web-dashboard and/or a mobile app ; it is the sensor that adds life to an IoT system by sensing the all important data (in the form of temperature, proximity & more).

IoT Sensors: The Cool Guys in the Town

IoT sensors are one of the coolest inventions in the modern times after internet.
Why?

Not only because IoTsensors are:

1. Easy to set up/install
2. Easy fault detection
3. No messy ‘wired’ connections and hence offer advantages of better mobility and management.

But, because the sensors are the ones that render your smart home solution ‘smart’. IoT sensors are IP bases and hence can be connected to the internet.

IoT Sensor nodes sense and capture the real time data from your home appliances and the surroundings with the help of sensory nodes and send it to the cloud backend via the IoT Gateway Device.

Accuracy of the information communicated by the sensors is very important for a robust smart home solution.

Any delay or inaccuracy (due to IoT Sensor Nodes) in sensing the ambient information can be at time catastrophic; for example if a fire breaks out and the sensors fail to detect it, it is needless to say how costly it can prove to be.

Types of IoT Sensors for Smart Home Solutions

Today, various versions of Smart Home Solutions are available in the market with high end sensor technologies and advanced features for added comfort and security.

But at the core, every smart home solution application comprises of basic sensors that are capable of detecting changes in the ambient data based on various stimuli such as temperature, smoke, motion etc.

Most sensors come in two varieties:

• those that are in direct contact with the physical objects to sense any fluctuation, and
• those that are remotely connected to the objects

Let us look at some of the most commonly used smart home sensors:

• Temperature Sensors: Temperature sensors are capable of detecting any fluctuations of temperature in their surroundings

  The information from these temperature sensors are used by the  a home automation solution regulation of the temperature within the rooms to a desired level, to perform certain actions such as turning on the fans and air conditioners, rolling down the curtains etc. based on the user’s request.

  Some of the commonly used temperature sensors in smart home solutions are MSP430 series from Texas Instruments (TI), LM35 from TI, Maxim Integrated DS18B20and more.

• Humidity Sensors: Humidity sensors are a great way to keep in check the humidity levels. The ideal humid level within homes should range between30 percent and50 percent.

  If the moisture level goes below or above this range, it leads to allergy, dryness of the skin or at higher levels a feeling of heaviness and air becomes suffocating.

  Many of the smart thermostats now come integrated with humidity sensors to detect any change in the moisture level.

  These humidity sensors help in maintaining the air quality and alert you about presence of allergens, mold growth etc.  HTU21D from TE Connectivity, Honeywell Humidicon™HIH6100 series and NPA-700 Amphenol Advanced Sensors are the most commonly used humidity sensors in modern smart home solutions.

• Optic/ Light Sensors: Optic sensors are great way to detect the ambient light levels. These sensors are useful in measuring the external light levels and accordingly switching on/off of the lights to conserve energy.

  These IoT sensors can also be used for controlling all the lighting installations within your homes – turn them on/off or change their brightness as and when required. Some of the commonly used optical sensors are Adafruit TSL2591 and Addicore BH1750.

• Fire/Smoke sensors: When it comes to ensuring safety of people and property when a fire breaks out, the timing of alert is very crucial.

  It is here that importance of fire/ smoke sensors comes into light. Usually, smart homes come with CO (carbon monoxide) detector that alert you whenever there is unusually high level of CO inside the building. The Maxim Integrated MAX30105 is a widely used sensor for fire detection.

• Proximity/Motion Sensors: The motion sensors are crucial for ensuring safety of your home and property especially when you are not present at your homes.

  These IoT sensors can alert you of any suspicious activity inside or around your home. These sensors sense any motion or vibration and can respond to 2D or 3D gesture, UV Index, or heart rate. Some commonly used motion sensors are Si114x and Si1102 from the Silicon Labs.

  There are even more variations of IoT sensors such as pressure/gas sensors, sound detectors, sensors to detect water levels – that are installed in smart homes these days for added security and safety of your dear ones as well as you valuable properties.

Conclusion:
Thus, the power of IoT technology to make sense out of ‘sensor ‘data and to etch out smarter and comprehensive solutions is already transforming the world. The design and development of an IoT sensor node based on standard protocols is a critical factor in deciding the success and efficiency of the IoT implementations. Unless the data sent by the sensors is not accurate or timely, there is no point in having a high-end and extensive IoT setup.