Thank you to all the new members and contributors of IoT Central. Our membership is growing quickly and we all should be excited about the community we are building. In this issue Bill McCabe looks at IoT Services, Ben Dickson explains why the ransomware threat is more serious than you think, and Sandeep Raut explores the good, the bad and the ugly of IoT. If you're interested in being featured, we always welcome your contributions on all things IoT Infrastructure, IoT Application Development, IoT Data and IoT Security, and more. All members can post on IoT Central. Consider contributing today. Our guidelines are here.

Why Companies Should Care About IOT Services

Posted by Bill McCabe 

As with any new technology, businesses will need to find quantifiable benefits in the Internet of Things before the concept is embraced and implemented. It could be argued that connected devices are already being adopted on a wide scale: companies like Microsoft, Amazon, Qualcomm, IBM, and others already see IoT as a core part of their businesses. Even so, there are still some, especially small to medium sized businesses, that are weighing up the costs and benefits of ultra-connectivity in the world of the Internet of Things.

The potential of cellular technologies for the great world of IoT

By Rick Blaisdell

In the next five years Internet of Things communications will see unprecedented growth, and cellular connectivity will become even more valuable. Wireless cellular technologies have found enormous potential as key enablers for IoT, and the continuously increasing technology enhancements and innovations in cellular technologies are promising to be the major primary access methodologies to enable a great number of IoT applications.

The IoT ransomware threat is more serious than you think

By Ben Dickson

At the recent Def Con hacking conference in Las Vegas, two researchers from cybersecurity firm Pen Test Partners showed that they could inflict your smart thermostat with ransomware from hundreds of miles away, and force you to fork over cash (usually bitcoins) before you could regain control of the appliance. Ransomware has been around for a while. It’s a breed of malware that locks down access to your files by encrypting them and sells you the decryption key that will give you back access to the files. IoT ransomware is relatively new. However, this isn’t the first time that the topic of IoT ransomware has been brought up by cybersecurity experts

Start Building an IOT Solution

By Ashish Modi

To build an IOT application we required following things.

1. A problem where we required IOT solution. 
2. Identify and design IOT based solution (Hardware + software + connection).

A problem where we required IOT solution

Nowadays everything is connected to the internet.  We need to move our existing system into IOT based solution.

Hitchhiker's Guide to IoT Standards and Protocols

By Preston Tesvich

In this article, we focus on a framework of how you can think about this problem of standards, protocols, and radios.  The framework of course depends on if your deployment is going to be internal, such as in a factory, or external, such as a consumer product. In this conversation we’ll focus on products that are launching externally to a wider audience of customers, and for that we have a lot to consider.  Let’s look at the state of the IoT right now— bottom line, there’s not a standard that’s so prolific or significant that you’re making a mistake by not using it. What we want to do, then, is pick the thing that solves the problem that we have as closely as possible and has acceptable costs to implement and scale, and not worry too much about fortune telling the future popularity of that standard.

The Good, The Bad & The Ugly of Internet of Things

The greatest advantage we have today is our ability to communicate with one another. The  Internet of Things, also known as IoT, allows machines, computers, mobile or other smart devices to communicate with each other. Thanks to tags and sensors which collect data, which can be used to our advantage in numerous ways. IoT has really stormed the  Digital Transformation. It is estimated that 50 billion devices connected to the Internet worldwide by 2020.

Additional Links

Follow us on Twitter | Join our LinkedIn group | Members Only | For Bloggers | Subscribe

As with any new technology, businesses will need to find quantifiable benefits in the Internet of Things before the concept is embraced and implemented. It could be argued that connected devices are already being adopted on a wide scale: companies like Microsoft, Amazon, Qualcomm, IBM, and others already see IoT as a core part of their businesses. Even so, there are still some, especially small to medium sized businesses, that are weighing up the costs and benefits of ultra-connectivity in the world of the Internet of Things.

You do not have to dig deep to see why IoT is important. Business Insider's research division, BI Intelligence, has predicted that IoT will become the largest device market in the world over the next five years. Most analysts predict market value will reach in to the trillions, with possibly $7 trillion of total value by 2020. Any way you slice the pie, billions of dollars are on the table. These figures are promising for businesses directly involved in the manufacture and design of device services and hardware, but what about the companies that will purchase these technologies to incorporate them into operations?

Perhaps the single largest benefit will be in how Internet of Things devices can lower costs. The manufacturing sector provides an ideal case scenario. Machine to Machine (M2M) systems will allow for machinery to become more efficient, and more autonomous. Take a production line that was previously labor intensive. Sensors relying on IoT can receive orders, initiate fabrication, sign off work orders, and even package products using IoT, and with little human interaction. Even non-automated manufacturing will benefit. Orders can be taken from anywhere in the world, transferred through the cloud, and delivered to remote manufacturing facilities. These systems can collect valuable analytics that can benefit accounting, inventory management, and even resource procurement.

While this type of IoT will directly benefit businesses in manufacturing, it will also create new opportunities for project managers, engineers, and IT professionals who will be necessary in designing, implementing, and supporting these systems. It even creates the role of Chief Internet of Things Officer, the CIOTO, tasked with managing a network of connected systems, and connecting their efforts back to business goals.

Because IoT provides immediate data collection, businesses in all industries will benefit from improved decision making. Being able to analyze and distribute intelligence faster means that tedious data collection will be a thing of the past. Decisions can be made faster, and in some cases can be automated. What this spells for enterprise is, in essence, better decisions based on better data.

Hong Kong International Airport, and other mega-airports around the world, already rely on RFID technology to track luggage and freight throughout their sites. This enables luggage to be delivered by machine to the correct gate, the correct passenger carousel, or to the correct airliner, train, or delivery vehicle. Items are tracked via computer, and managed from a central control point. This reduces hands on management and labor costs. HKIA spent $50 million to develop the initial infrastructure, but widespread adoption of this IoT based technology could save the industry $760 million per year, according to the International Air Transport Association.

Imagine how a similar system could benefit a SMB. Goods delivery could be RFID or barcode tracked on handheld scanners. This tracking information could be uploaded to a cloud solution, from where dispatchers, couriers, and clients could track the location and progress of a delivery. These are the kind of innovations that are driving IoT, and making it a necessary technology in a market where cost and efficiency is key, and where end users and consumers demand constant, easily accessible information.

The opportunities are there for businesses who adopt IoT today. The benefits exist whether they seek to improve manufacturing efficiency, streamline logistics processes, or even provide new ways for customers to interact and receive information. In the growing world of IoT, the question is not why should we care, but is rather, can you afford not to?

Please give us your feedback or share how the Internet of Things has touched your business below. 

Guest blog post by Bill Vorhies

Summary:  Sensors that know how you feel?  Sensors that want to change the way you feel?  When did that happen and better yet how?

We’re getting used to sensors finding out what we’re doing.  Apparently they are now sufficiently sophisticated that they can even tell if I’m sitting up straight (yes Mom – BTW using a camera is almost cheating, you should be able to do this with just an accelerometer and a gyro). 

But what if I told you that those same IoT sensors can tell how you feel?  And now they’re even being programmed to change the way you feel!  A little creepy?  Feeling manipulated?  Hang on to your hat because it’s about to get worse or better depending on your point of view.

Mood Science

First of all I didn’t even realize that ‘mood science’ is a real thing.  Turns out it’s been going on a long time in design circles where designers and architects in particular have been making informed guesses at what chills us out.  Blue rooms relax.  Red rooms stimulate and arouse.  Pink rooms are most soothing. 

Interesting note: For years many prisons have been painting their walls bright pink based on early findings that prison inmates’ tempers were soothed when placed in pink-walled cells.  For what it’s worth these generalizations about room color have all now been overturned by the new practitioners of ‘mood science’.

I’ve been tracking the uses of IoT sensors particularly those with human interaction (think Fitbit) but I didn’t see the big picture until I came across this article “Design for Mood: Twenty Activity-Based Opportunities to Design for Mood Regulation” by Pieter M. A. Desmet, a member of the Faculty of Industrial Design Engineering, Delft University of Technology.  This is one of those articles you know you should trust because it contains a reference bibliography of 169 learned articles.

For the most part it seems that in academic circles the desire to determine how to ‘regulate mood’ is pretty benign and generally couched in terms like improving subjective well-being.  After all who doesn’t want an extra helping of well-being?

Then I found it.  About three pages in, buried in the text:

• Mood influences consumer behavior. Research has demonstrated that consumer mood influences buying behavior, product preference, and purchase decisions.
• When evaluating new products, people do so more favorably when in a good mood than when in a bad mood.
• Mood influences user behavior. For example, when using new products, individuals in a bad mood tend to explore fewer interaction possibilities than those who are in a good mood.
• A good mood increases one’s willingness and motivation to adopt and use new technologies.

OK, now it’s clear.  Just sending me a coupon when I’m standing next to the new flat screens isn’t nearly enough.  “They” want to know how I’m feeling, and better yet to make me feel in a way that positively disposes me to buy.

One more piece of foundational information before we move on to how this works.  Turns out that monitoring and manipulating mood (feelings) through just four quadrants and eight basic mood types is enough to make this happen.

When it comes time to model, one of these eight states will be our dependent target variable.

How It Works

How would sensors go about detecting mood?  It’s all about cleverly combining and interpreting the signals.  This is a fairly new field studying how to fuse sensor data to make it context aware.  Take for example heart rate as measured by a wearable sensor.

Dr. José Fernández Villaseñor is a medical doctor and electrical engineer studying the field of emotion analysis using sensors.  His research shows the rate at which heart rate increases can differentiate between exercise and increases due to adrenalin from excitation based on the slope of the increase.  Turns out that Heart Rate Variability (HRV) is one of the prime tells that can be used to differentiate one mood from another.

Here’s a simple example of how your Xbox or PS4 can not only tell how you’re feeling but manipulate those feelings. 

image source: mouser.com

You are playing a driving game.  Your game controller may contain sensors that can detect:

• Muscle relaxation (MR)—via a pressure sensor.
• Heart rate variability (HRV)—via a two-electrode ECG on a chip.
• Sweat (S)—via a capacitive sensor.
• Attitude (A)—via an accelerometer monitoring a person’s state of relaxation (jerky movements vs. steady hands).
• Muscle contraction (MC)—via a pressure sensor.

Suppose the combination of increased pressure on the controller, sweat, and the jerkiness of your motions (from the accelerometer) could be correlated (modeled) against your performance in the game.

Pressure and sweat increase.  Jerkiness increases.  Your game platform infers that you are both excited and stressed.  Your score is just OK.  To encourage you to play more, the system adjusts the difficulty of controlling the steering, braking, and the behavior of the other cars to reduce difficulty.

Your performance and score improves.  Pressure and sweat decrease and your hand movements become smoother.  The platform interprets that you are more relaxed and are mastering the game at this level.  To keep you involved it increases excitement by making input controls and the behavior of competing cars more difficult.

You’ve just been gamed in the new world of IoT mood manipulation.

It’s Not Just About Wearables

In a sense, if you’re worried about the intrusiveness of this technology you would think that wearables offer their own defense – just don’t wear them (leave the Fitbit at home).  Problem is it’s not just wearables.  There are at least four categories of things that supply data about ourselves, many of which you may not have thought of in this way.

Wearables:

Wearables is a big one.  It’s not just where you are and how fast you got there (GPS, accelerometers, altimeters, thermistors, gyros) it’s also sensors that measure physiological signals such as heart rate, skin conductance and temperature, and respiratory rate.  These already include finger rings, ear rings, wristwatches, wrist and arm bands, and gloves.  Soon to come, sensorized garments including shirts, shoes, and underwear.

Take a look at the W/Me wearable wellness monitor introduced in 2013 that claims to measure the four basic mood states: passive, excitable, pessimistic, and anxious.

Natural-Contact Sensors

These are sensors that are integrated into the devices and particularly the surfaces of objects we regularly come in contact with.  Likely you are interacting with these objects, not just brushing up against them. How about the steering wheel of your car that could easily have these sensors embedded and also transmit information about the smoothness or jerkiness of your movements.

It could be a chair that infers your stress or relaxation or a pen, cell phone, or mouse that can detect moods like stress, nervousness, and excitement based on hand movement.  Even your keyboard can give you away by interpreting the strength and cadence of your keystrokes or how many times you use the backspace key.

Non-Contact Sensors

Anything with a camera or a microphone: computer, phone, TV, or game console that could use visual signal processing (deep learning) to record facial and voice expression, body posture, pupil diameter, and eyelid closure patterns.  Law enforcement is hard at work adopting facial and emotion detecting software.

Self Expression

Sometimes we just tell machines how we feel.  I frequently tell my alarm clock how I feel when I give it a rough slap (there could be a sensor in there warning my family I’m in a bad mood when I come out for breakfast).  About 8 years ago Philips developed a ‘mood pad’ for hotel rooms that let you pick a mood (romantic, restful, let me sleep) that controlled ambient lighting.  And if you look in your app store, I’m sure you can find an app for creating a mood journal or for evaluating how you feel right now.  Who’s receiving that signal?

What Could Possibly Go Wrong?

This question is almost too rhetorical to even ask.  If you want to make me more likely to buy something, OK, maybe I can live with that.  And if it makes my game play more interesting that might go on the good list.  If your car tells you you’re suddenly suffering from road rage that could be helpful.  And certainly there are applications in healthcare, mental care, and elder care that we can easily applaud.

But when it comes to manipulating me I really want to know who’s doing it and with what motive.  What could the government or the IRS be learning about me or trying to make me do?  I don’t want to seem alarmist.  Sometimes the best thing we can do is just make ourselves aware that this is happening.  Maybe there will be an on-package or on-screen disclaimer (probably buried deep in the EULA). 

This is one of those technological advances that delights me as a data scientist and disturbs me as a citizen and human being.  Like all technological advances this one’s out of the bottle and trying to get it back in would make the loaves and fishes look like child’s play.  As much as anything, I just want to know if I’m getting some quid for my quo.

About the author:  Bill Vorhies is Editorial Director for Data Science Central and has practiced as a data scientist and commercial predictive modeler since 2001.  He can be reached at:

Bill@DataScienceCentral.com

Follow us @IoTCtrl | Join our Community

IOT and Assisted Living

It is most likely the you have heard the term “internet of things” or IOT in regards to everyday things such as our televisions and phones. That is not however where this new innovation is going to end. There has been a lot of talk about the IOT stepping into the healthcare industry with things like connected healthcare.

Another area where we can expect to see the IOT playing a large role is in assisted living. It is no secret that people are living longer than we ever have before. It has even been said that the first person to see the age 150 has already been born. It should come as no surprise then that nursing homes and senior assisted living facilities are full to bursting with elderly people whom are healthy but incapable or afraid to live on their own. The IOT could help with this.

We are all familiar with products such as Life Alert that have been used to give seniors a sense of security in their own home. These types of things allowed seniors to remain in their homes longer than before. They are not perfect though. The fact is that the technology behind these types of monitoring devices is out dated. It relies on a live person being available 24/7 to respond to the individuals call for help. What happens when the person in question does not have the capability of triggering the monitoring device though? This is where the IOT can step in.

Recently engineers have developed sensors that can be placed discreetly throughout the home. These sensors then monitor the resident’s movements and activities throughout the day. These sensors rely not on a live person monitoring them, but on algorithms and programming that over time learn the normal habits of the person living in the home. They monitor things such as…

• location of the resident within the home
• light sources being used
• bed time and awakening time
• television watching
• cooking
• bathroom usage
• leaving the home and returning
• heating or air conditioning temperature and adjustments

Then in the case of an emergency or variations to that pattern that do not fit the normal activity within the home can notify family members or medical professionals.

Another development is something similar to that of Life Alert but more sophisticated. Wireless vital sign monitors. These devices can notify first responders of medical emergencies such as stroke, heart attack and a loss of consciousness without the person suffering having to do anything at all. Further they could notify patients of an issue well before it actually happens, such as notifying a heart patients doctor that their heartrate has been erratic over a period of time, thus indicating that further investigation may be needed. It is not hard to see that very soon we could see the IOT playing a large role in the lives of our seniors, or anyone that needs some form of assistance.  

 For more information about IOT and Healthcare please check out our new website  www.internetofthingsrecruiting.com 

Guest blog post by Jason Li

Connected devices, Smart City, home automation, e-health, Big Data ... In recent years, the concepts of communicating objects have multiplied. In reality, they are all one facet of the same upheaval - the Internet of Things.

Cars can be driven without a driver, TVs are going online, and heating systems are activated automatically to the arrival of the residents. The Internet is making many processes in daily life easier. The Internet of Things, or IoT, which enable devices to communicate with people or machines, is actually working in many places in our daily life already.

Further reading:

• Smart Techs are around the Corner, Are you Ready to Adopt?
• At the Crossroads of the Third Wave of the Internet 
• 4 Strategies for the adoption of the IoT
• Artificial Intelligence will be a problem solver for the Internet of Things
• Why the Cloud is Essential for the Success of IoT?
• Big Data and IoT are Redefining the Information System
• From “If” to “When” – IoT is Transforming the Cybersecurity Landscape
• The future of Mozilla is the Internet of Things 
• Important Things for Startups Entering the IoT in 2016
• A Japanese Perspective on IoT – Opportunities and Challenges
• 5 Common Misconceptions About Internet Of Things (IoT)
• Is Our Smart Future Safe? Findings and Advices from the 2016 Norton Consumer Survey
• 5 ways to Overcome Common Hurdles to IoT Adoption
• 5 Internet of Things Startups to watch out for in 2016
• Preparing for the Connected World – What Skills are Necessary?

For People: Web-Enabled Electronics 

If phone and computer without Internet access, they become unthinkable devices. Meanwhile, this is also valid for televisions and audio devices or cameras. New electronic products will not come without Internet in the future. For example, wearables are the typical types of Web-enabled electronic devices which are worn directly on the body for health monitoring, have become the next big trend in healthcare.

Further reading:

• Wearables will be the Next Big Trend for Healthcare
• The Internet of Me : 11 Wearable Predictions to Watch In 2016
• 6 Smart Tips for Smart Devices Security

For Home: Smart Home 

Manufacturers not only produce Internet-enabled home appliances, but also care about apps and software portals. For example, Miele combines a range hood with the stove so that the fan motor is automatically adapted to the cooking process.

The smart home concept has created many great ideas - shutters can be controlled by smartphone, and lights and heats can be turned before you returning home from vacation. Germany’s digital association Bitkom assumed that there will be one million fully networked households by 2020.

Further reading:

• 4 Examples of Using Connected Devices to Make Your Home Smart
• 5 Robots that Will Make Our Cleaning Work Easier
• From Smartphone to Smart Home

For Transportation: Connected cars 

In a survey, the respondent indicated that the smartphone connection in the car was more important than a higher horsepower. In the world of connected cars, drivers, cars and infrastructures are all connected with each other, and are able to communicate among objects in the system in real-time to optimize routes and avoid accidents.

The concept of autonomous cars has taken the step further towards making self-driving cars. Major car manufacturers promised to produce at least one business model within the next five years.

Further reading:

• Will You See Millions of Autonomous Cars by 2020?
• How IoT Works for Us? Explained in Four Examples
• Autonomous and Connected Cars Write New Rules for Car Industry
• Big Data is the Underlying Big Story of Connected Car
• Moving Urban Transportation Forward with Big Data
• Taking the Pulse of Wearables, IoT and Analytics for Freight Transportation
• Infographic: Transportation in the Age of IoT , Big Data and Emerging Technologies
• From Smart Cars to Self-Driving Cars - Internet of Things (IoT) and Big Data are Transforming the Future Transportation
• Can Mobile Data Help Reduce Traffic Jams?

For Community: Smart City 

In 2050, our planet will be different from today – There will be nine billion people live on it, 70% of them live in cities. This growing trend has not only significantly increased demands in cities, but also created great opportunities to improve efficiency of energy, material and human resources.

The Smart City concept was created to exploit these opportunities with aims to integrate information and communication in various technical systems of a city to promote innovative solutions for mobility, management and public safety in the city – in particular, electricity, water, gas, and goods.

Further reading:

• “Virtual Singapore” Makes a Small Country with Big Intelligence
• What is Smart City and Who are the Key Players
• Innovative Use of Urban Data – Boston’s CityScore Example

For Agriculture: Digital Agriculture 

“According to the United Nations’ Food and Agriculture Organization, food production must increase with 60% to be able to feed the growing population expected to hit 9 billion in 2050. John Deere uses big data to step into the future of farming to help farmers achieve this ambitious target.”(Datafloq)

The networked agriculture can benefit in many places of databases and real-time monitoring – The balance of weather data with the plant growth data, and the complex structure of the forecasted demand and the current market.

Further reading: 

• Big Data Holds the Keys for Digital Agriculture

For Manufacturing: Industry 4.0 

Industry 4.0 represents the manufacturing future with IoT. High demands are made on the production of the future – you must be intelligent, changeable, efficient and sustainable. Industry 4.0 stands for the intelligent networking of product development, production, logistics and customer.

The Industry 4.0 Working Group define Industry 4.0 as “a network of autonomous, controlled situational itself, configure itself, knowledge-based, sensor-based and spatially distributed production resources (production machines, robots, conveyor and storage systems, resources), including their planning and control systems”.

Further reading:

Digital Industry 4.0 – It is All about the Manufacturing Future with IoT

Follow us @IoTCtrl | Join our Community

Guest blog post by Brian Rowe

Another big part of the food supply comes from ranches and farms that raise and slaughter various livestock. While ranching is sometimes bundled with agriculture, I discussed farming in Big Data in Agriculture, so we’ll focus on ranching this time around. Somewhat surprising is that big data usage in ranching appears more limited than in farming. That said, there are a number of novel uses of technology and data in animal husbandry.

Credit: Emilio A. Laca

Land Use Optimization

At a high level, the goals of ranching and farming are the same as any business: increase yields and lower costs. Production maximization has long played a role in large operations. A twist to the optimization problem is land use optimization and how that can affect yields. According to NASA, “Australia’s rangelands provide an opportunity to sustainably produce meat without contributing to deforestation” if properly managed. This sort of optimization is made possible by big data coming from satellites. The same article cites how some West African nations use satellite data “to identify areas with agricultural potential and to estimate the amount of food available.” Growing up in rural Colorado, the most advanced tech I saw at ranches were solar powered fences and artificial insemination. Clearly a lot has changed. From a supply chain perspective, these trends also demonstrate how just-in-time manufacturing can be extended to resource allocation.

From a technical perspective, crop and livestock rotation will become outputs of a multi-objective optimization problem. I imagine that the challenge will be less about the optimization and more about the inelasticity of “bioprocesses”. Aside from slaughter or transfer to somewhere else, there aren’t too many options for reducing “inventory”. Presumably these issues already exist, so any solution is bound to be an improvement. Ultimately, there is a race to avoid the outcome that the U.N. foresees: the majority of humans eating insects as a primary source of protein. Even if that future is unavoidable (not necessarily bad), presumably similar techniques can be used to maximize insect yields.

Sensors and IoT

Technology advancements are driving parralel trends in agriculture and ranching. While satellite imagery offers a big picture overview, sensors provide a micro view of individual plants and animals. RFID tags are a first step enabling real-time tracing of an animal. Equally important is the assignment of a unique identifier to facilitate storing electronic records that can be merged into a centralized dataset. RFID is fundamentally passive, whereas sensors are active. This is where biosensors and Precision Livestock Farming (PLF) come into play. PLF is a comprehensive approach to livestock management and animal welfare. The goal is “continuous, fully automatic monitoring and improvement of animal health and welfare, product yields and environmental impacts” Some of the sensors developed to achieve this are surprisingly simple and surprisingly clever, such as sensors that monitor the vocalizations of livestock to determine stress, illness, etc. These advances can also “raise milk yields, while also increasing cows’ life expectancy and reducing their methane emissions by up to 30%” (CEMA). The Biosensors in Agriculture workshop held in the UK presents even more exciting examples.

Other notable research around PLF include image analysis to monitor animal welfare and
classifying the behavior of cattle and fowl based on GPS movements. According to one paper, a decision tree was used to classify four behaviors: ruminating, foraging, standing, and walking. The features were based on distances and turning angles from the GPS data. Not surprisingly, the confusion matrix was pretty poor in terms of distinguishing between ruminating, foraging, and standing. So there’s lots of opportunities to whip out R and randomForest or party to conduct your own analysis (assuming you have access to the data).

Data and Accessibility

Big data is often synonomous with cloud computing and for ranching it’s no different. As with agriculture there are trends to centralize data to “help ranch managers track livestock, view production statistics, plan grazing rotations and generate reports that can offer insight into the health of a livestock operation.” Unlike in agriculture, it doesn’t appear that the machinery manufacturers are taking a role, although it wouldn’t surprise me if some PLF suppliers have cloud platforms for their customers. GrowSafe Systems is creating their own cloud-based dataset based on their customer data. Their system collects and forecasts “complex animal traits such as efficiency, growth, health, stress and adaptation.”

Europe has taken a different approach focusing on defining a comprehensive classification scheme for agricultural systems. Clearly the goal is data interoperability, so data can be widely shared and applied across farms and ranches. This goal is reflected in the three-level system that encompasses environmental factors and GIS data to site-specific measurements of individual animals that affect yields and animal welfare. Landcover data appears to be the most extensive, while biosensing is likely where the most immediate opportunities are to be found.

As data becomes more focused on individual sites and animals, scarcity is the word that comes to mind. In the USA public datasets don’t come anywhere near the level of detail to make a useful analysis. See data.gov for an example of a disappointing dataset. Of course it isn’t clear whether transparency of this sort is even possible. One rancher believes they have a right to privacy and shouldn’t be compelled to open their books to external scrutiny. This is understandable, but does this belief extend to data? Data privacy is a thorny issue, particularly balancing privacy, ownership, and the need for transparency vis a vis food security/safety. Eventually I think economics will force a change of heart if yields and margins increase significantly with the help of open data. However, this may take the shape of data cartels as opposed to truly open data. As big data and centralized data stores become more wide spread, this debate over data ownership will continue to be visited.

Know of some public datasets available for ranching and animal husbandry? Post links in the comments!

This post first appeared on cartesianfaith.com. Brian Lee Yung Rowe is Founder and Chief Pez Head of Pez.AI // Zato Novo, a conversational AI platform for guided data analysis and Q&A. Learn more at Pez.AI.

Follow us @IoTCtrl | Join our Community

By Abjijeet Banode. This article originally appeared here.

Fleet Management System (FMS) is one of the essential parts of businesses which directly or indirectly deal with automobiles. Precise fleet management minimizes various operational risks and increases cost efficiency. With proper utilization of analytics, alerts, and diagnostics, fleet management converts businesses to be more reliable and sustainable. Like any other business, predicting risks and working towards mitigation is essential for fleet businesses. Ample use of data analytics for early detection of faults and predictive mechanism helps business to reduce maintenance cost and downtime.

Typical modular fleet management unit consist of OBD-II (On-Board Diagnostics – Standard revision – II) module which connects to Controller Area Network (CAN) bus. Microcontroller, sensors, and various devices from vehicles use this bus (communication channel) to communicate with each other. OBD-II module captures diagnostic information from the CAN bus example, data engine control unit, and transmission. 

Figure 1: Typical architecture of Fleet Management System (FMS)

GNSS (Global Navigation Satellite System) receiver unit like GPS, GLONASS, assist to capture geographic coordinates. Synchronous capture of GNSS data and diagnostic data can help to immediately identify exact location of a vehicle breakdown or other events. Mapping it against reference data from the department of transportation can be utilized to analyse driver behaviour and their adherence to traffic regulations.

Every business has its unique requirement and objective behind Fleet Management System.  Organizations need to upgrade FMS module based their particular use case.  Trailer transporting food needs additional sensors to monitor temperature of on-vehicle refrigerators whereas a trailer carrying hazardous liquids has its own sensor requirements.

Cellular module is essential for real-time monitoring of a vehicle’s essential parameters, theft detection, driver safety, and to report breakdown. This data needs to be sent to cloud (or physical storage) so that fleet operators can analyse required parameters in real-time, perform predictive analysis, and identify mitigation requirements for smooth operation of fleet.

Effective use of data analytics and visualization tools – dashboards is the brain of intelligent fleet management system. Visualization parameters varies with respect to business needs but a few basic conclusions like driver behaviour, fuel efficiency, fleet health, maintenance requirements and breakdown will be always there on dashboards. 

Figure 2: Benefits of effective fleet management

The cost of E2E fleet management system has been reduced due to efficient analytics platform based on quality open source solutions (e.g. MongoDB, Hadoop), reduction in cost of electronic assemblies (chipset cost, antenna cost), economical cellular data connectivity (eUICC, dedicated data plans for M2M, IoT), reduction and flexibility in cloud storage cost due to competition, and most importantly cross domain interest of companies from various vertical (e.g. Cellular operators), electronic product manufacturers, and IT services are exploring additional revenue streams in automotive domain.

Guest blog post by Vishal Sharma

A buzz word around us for quite some time now is Internet of Things (IOT). To Simply define it:

“The internet of things (IoT) is the network of physical devices, vehicles, buildings and other items—embedded with electronics, software, sensors, actuators, and network connectivity that enable these objects to collect and exchange data.” (Wikipedia)

It simply means for me all devices that are connected to internet forms part of global network, producing data, that can be utilize for the betterment of services or customer experience or the way one can use it, some examples

• Send real time alert, Smart wear to your doctor or from a machine nearing permissible temperature for over heating
• Real time diagnostic like heart rate, pulse, Temp or SO2 Levels.
• Security breach detection Etc.

How this is done is not what I am focusing around, once it is implemented and if it’s done with integration of all your devices / networks which work as entry point e.g. your mobile, GPS you use etc.

What will be level of privacy remains in a complete IoT world?

All devices are in connection and all are talking to each other with some kind of BIG data tool and Analytics working together. What will happen?

Some scenario

• You are running out of fresh milk in your fridge and now smart fridge will send an order to your grosser for replenishment of the same.
• You have a Smart watch or a fitness device which help your Coach to monitor your activity and hear rate and other vitals, helping him or her to identify the best fit regime for you.

All above are good but let’s go little forward and think a real life scenario that can happen,

You went to your favorite food joint and at point of sale you provided and identifiable information specific to you what will happen if everything is connected in true IOT scenario, Person on the sales counter will have lots of information and POS machine will not take your order if you have any disease and your doctor have said no without giving any details only a small bit information,

E.G. I want fries, Person at POS will say “Sorry Sir can’t, as your doctor has instructed that no high salt/ deep fried items for you, so please pick other item from menu” and then again you go for selecting other things.

Now imagine how much your privacy is at risk, for a total stranger knowing about your health.

Another Scenario of some card company calling you saying you are using at X POS service use of Y gives more incentive for shopping.

Do I really want world to know about it, may be not directly but through different ways.This is just one/two example however there can be many other one can think off.

Questions remains is IoT good and i will say definitely it is as per my view its adaptability will give far more benefits than risk caused.

However level of integration will tell how much personal the use can be called invasion of privacy and how much is actually required.

Note till the time I was writing this post there was no Single protocol that connect different devices and make it part of Global or Actual Term IOT, which I know of; hence integration can take long time.

Follow us @IoTCtrl | Join our Community

The advent of smartphones, and the rise of mobile internet and mobile apps disrupted and transformed the way we live and do business. Thanks to the millions of mobile apps you can buy or download from app stores, you practically have your mailbox, office, photo album, TV, game console, shopping cart and much more at your disposal any time you like.

Now, thanks to the Internet of Things, the phenomenon that is already triggering the next digital revolution, your car will become integrated with your increasingly-connected life and will be added to the collection of things that fit in that little gadget you carry in your pocket all the time. Already, the combination of IoT gadgets and mobile apps in vehicles is gaining popularity among consumers and fleet operators, providing functionality and opportunities that were inconceivable a few years ago, which make them more efficient, safer to drive, more resistant to crime and theft, and less costly to maintain.

The current possibilities are virtually endless, and the future is even more exciting. Here’s a glimpse of how IoT connectivity, smart sensors and gadgets, edge computing, mobile apps and cloud services will revolutionize how you interact with and use your car.

IoT provides improved access and security

With every part of your vehicle being connected to the internet, you’ll have better remote access and control over your vehicle’s functionality with your phone. Ignition, windows, lights, trunk, everything can be manipulated through your smartphone while you’re busy elsewhere.

So you can start the engine with a tap on the phone and let it warm up in winter while you’re having breakfast and going over news headlines.

BMW puts this functionality to display with its My BMW Remote App, which enables car owners to remotely unlock or lock their cars, sound the horn, flash the lights, and turn on the auxiliary heating/ventilation system.

Viper SmartStart is an example of how you can integrate IoT with legacy technology. The kit, comprised of IoT gadgets, a mobile app, and a mobile app will give you enhanced control on your vehicle. After installing the IoT devices in your car, you can use the SmartStart app to start, lock, unlock and locate your car with a swipe and tap on your phone.

But mobile access surpasses convenience and also enters the realm of security.

Today’s mobile devices protect your data with state-of-the-art security and encryption features that are hard to hack even for government agencies. IoT will help you leverage this enhanced level of security in your car and improve theft prevention.

NFC door locks can relieve you of the nightmares linked to your car keys being lost or stolen. After registering the lock with your phone through its associated mobile app, you can unlock your car by tapping your phone against the handle. You can rest assured that only a person possessing your phone and being able to unlock it can unlock the door to your car. And in case you want to lend your car to a friend or family member, all you have to do is to grant access to their phone through your mobile app.

TapKey has implemented this concept successfully, creating a mobile app that turns the smartphone to a car key and enables car owners to securely and easily grant vehicle access to others.

And in case you lose your phone, having the lock registered with another phone will be a matter of logging into a cloud app and introducing your new phone.

Smart car alarms will quickly send an alert to your smartphone in case your car is being broken into, and in case your car does get stolen, your mobile app will help you find and track it through its GPS device. This can help report the theft and have it recovered much faster.

IoT provides improved control over vehicle status and driving

On-board Diagnostic (OBD). Telematics devices are smart cloud-connected IoT boxes installed on vehicles which provide insights and real-time information about vehicle health and driver habits. These devices function by communicating with a set of smart sensors installed on different vehicle parts including doors, windows, engine and tires, and constantly monitor and report the status of the vehicle.

A mobile app interacting with the telematics system can act as a digital assistant which alerts drivers in real-time about measurable events such as speeding, sharp cornering, seatbelt usage and over-acceleration. The app can also communicate with the cloud service where historical driving data is stored in order to enlighten drivers about bad habits they should correct, and their driving improvements over time.

EcoDrive is an interesting app that monitors your driving habits in real-time, including acceleration, deceleration, changing gears and speed variation, and gives you a score (or eco:Index) which helps you assess your safe driving skills.

More advanced use of IoT and telematics would be to keep tabs on and alert about maintenance issues that can compromise passenger safety, such as low tire pressure, malfunctioning engine, parts that need replacements and overdue services. Drivers would be able to get a complete report of their vehicles with a tap and swipe on their phone and without the need to look under the hood.

Chrysler’s UConnect app is an example of the efficient use of telematics and mobile technology. The app lets you remotely monitor and control your car’s maintenance, provides you with monthly health reports and alerts you about critical maintenance issues that need immediate attention.

The best part about telematics and on-board diagnostics is that they’re standardized across the industry and do not require vendor-specific integration, which means your mobile app and historical driving data can be migrated and ported when you switch vehicles.

IoT sensors improve vehicle safety

While the intersection of IoT and vehicles provides many opportunities, perhaps safety is the most prevalent. If there’s one thing that IoT should be praised for, it’s the fact that it’s promoting safe driving and assisting drivers in avoiding road incidents.

With more and more cities investing in smart infrastructures, IoT-powered vehicles are much better prepared to help drivers in commuting safely. Interacting with IoT sensors installed on roads, connected vehicles can detect when drivers are veering off the road as the result of distraction or fatigue, and alert them to steer back on the road. In the case of semi- and fully-autonomous vehicles, the car itself can take matters into its hands and correct the vehicle’s direction if the driver doesn’t react.

Smart sensors and smart cement can also gather information about road surface and bridge conditions. Connecting to cloud servers, mobile apps get real-time insights about road conditions and assist drivers in choosing safer roads and avoiding hazardous areas before heading out. In case a driver treks into a particularly dangerous zone, e.g. an ice-covered road, connected vehicles will directly communicate with local gateways and sensors, retrieve data about road conditions, and warn drivers about the dangers and instruct them to slow down.

In 2007, the collapse of the I-35W Mississippi Bridge in Minneapolis resulted in 13 casualties and hundreds of millions of dollars’ worth of damage. Today’s IoT technology could’ve detected the bridge’s failing structure and warned both maintenance authorities and drivers about the dangers, saving lives and preventing damage.

IoT helps avoid traffic and congestion

Few things are as frustrating as getting stuck in a traffic jam when you’re late for work or want to attend an important event. Being able to avoid congestion and plan in advance can save you time and also reduce fuel consumption.

Fortunately, IoT can help in this sector as well. IoT sensors in roadways track and report commuting in real-time, which can help drivers better plan their trip and avoid crowded areas while also assisting city authorities in distributing congestion and pushing traffic toward the less frequented areas.

Mobile apps gleaning information from traffic sensors can estimate time of arrival based on the level of traffic and also provide alternative routes to drivers which will cut down the time and stress of the trip.

The added benefit of controlling traffic through IoT technology will help reduce car accidents considerably, and will collectively reduce pollution and help us have greener cities.
IBM has a great post on how it’s using apps and its IoT platform to collect traffic data, generate insights and control congestion.

Caveats and requirements

All the benefits of connected, IoT-equipped and mobile controlled vehicles isn’t without its drawbacks. The vehicle industry is already dealing with several worries where vehicle IoT is concerned, chief among them being security and privacy issues. There have already been several cases where connected cars have been hacked through mobile apps, infotainment systems and other insecure connected gadgets that are installed on the car.

While none of these dismisses the importance and impact that IoT will have over the future of cars, it does highlight the need to pay more attention to the security of IoT, especially in the vehicle industry.

This can be achieved by making sure the developed software is built by experts that have the knowhow to deliver both functionality and security. Secure coding should be one of the main tenets of any software that will be installed in our cars and their related peripherals, lest we want to see them be exploited by malicious actors and used against us.

The future of IoT in vehicles

For the moment, you have your car in your pocket. But this is just a taste of how IoT is transforming the automotive industry. Cars that can be parked with a single tap of an app button, circular economies where automobiles are shared and rented as a service through mobile apps, and the era of completely autonomous vehicles are not far away. Every day, the Internet of Things is conquering new summits. Who knows what tomorrow holds?

See how Mokriya develops solutions for IoT problems

(Photo courtesy of Faraday Future)

By Rick Blaisdell. This article originally appeared here.

Unlike other industries, healthcare has been relatively conservative and slow in embracing innovations like cloud computing and the IoT, but that is starting to change, especially if we think about the past years. Innovative tech products and services are more and more part of our daily lives, making it harder for healthcare providers to ignore the potential advantages of connected medical devices.

Moreover, a new term is used more and more to describe this amazing connection between the Internet of Things and healthcare, and that is the Internet of Medical Things (IoMT). IoMT is the collection of medical devices and applications that connect to healthcare IT systems through online computer networks. Medical devices equipped with Wi-Fi allow the machine-to-machine communication, thus developing the basis of IoMT.

At the same time, healthcare companies are renewing their operative models through digital health technologies and are focusing more on prevention, personalization, consumer engagement and improved patient outcomes to remain competitive. Here are some great examples:

• An asthma inhaler with a built-in GPS-sensor – Propeller Health has released an FDA-approved asthma inhaler with a GPS-sensor. Basically, a tracking device is placed into an asthma inhaler, providing support and helping reduce the cost for health systems and thus for patients. Every time the inhaler is used, time and location are being saved, the GPS-data recorded and imported into a personal profile. This allows for tracking of the time and location of the use of the inhaler, allowing a user to even avoid those areas which may prompt his/her asthma attacks.

• New system for optimizing workflows in hospitals – In cooperation with Microsoft andHealthcast, The Henry Mayo Newhall hospital in Valencia, California implemented a smart system which provides the doctors with access to a wide range of data: from patient files to test results, prescriptions and much more. This was achieved by connecting 175 hospital devices, as well as the personal devices of the doctors, to the available computing offices and systems. Thanks to the new system, the doctors have secure access to examine laboratory tests, to write prescriptions, or to view the patient files at any time. As a result, the time for registration was reduced by 95% – from two minutes to six seconds.

• Digital contact lenses for diabetics – The contact lenses were jointly developed by Google and the Swiss health care group Novartis, and will help diabetics to measure their levels of blood sugar through tear liquid and to transfer it to a glucose monitor or a smart device like a mobile phone.
• Smart monitoring of medication – Vitality has been one of the pioneers in the medication area, developing a new system called GlowCap. Those drug containers use light and sounds to signal the patient when the time to take the medicine has come. They also remind the patient automatically through a call. Moreover, every week a report is being sent to customers, with information about how they should be taking their medication.

To drive adoption of IoMT systems and to achieve more end-to-end solutions, hospital administrators, vendors and manufacturers must cooperate to lead healthcare through this important change. The impact is clearly visible, as companies are developing a collaborative culture in embracing digital technology, and the next five to 10 years will be essential as they manage the data from patients and incorporate this into the physician’s workflow.

Photo source: freedigitalphotos.net

How Connected Healthcare is Becoming Vital

There is one word that describes the direction that the health care industry is heading, “connectivity”. This catch all term is used to describe using the internet to increase the reach of medicine. This is also known as the internet of things (IOT) and it is nothing new. It is however relatively new to healthcare.

The goal of connected healthcare is to empower both the providers and patients. Using connectivity, a provider can make use of remote patient monitoring, and consultations without the need to be face to face. This may seem like a moot point to some, but it would enable doctors to reach patients that they have never been able to before. Connected healthcare would also allow things like our cell phones and tablets to send real time medical information to our healthcare providers.

Taking it a step further the aim is going to involve using medical data in news ways. Rather than your medical file sitting unused in a cabinet somewhere the aim of connected healthcare is to compile the data in a way that lets your healthcare provider identify areas in which your day to day life may need improvement. Using this data, you and your provider would then be able to create novel solutions to the issue.

The question still remains though, why is connected healthcare becoming vital? We just explained what it is and some of the benefits but where is the “need”?

It is quite simple; out healthcare network would resemble that of a spider web if we connected all of the facilities with string. You have your imaging done at the hospital, your bloodwork done at a lab and your general check-ups done at your doctor’s office. Then there are outpatient procedures, specialists and countless pharmacies. In days past the only thing that connected these medical facilities were phone and fax (or you transporting your paperwork), which was in no way ideal. The margin for error was simply too great. What’s more it could take days for results of testing or procedures to make it where they needed to go.

What connected healthcare is allowing us to do is use the internet to digitally transmit records, prescriptions, files and test results almost instantaneously. For some this may not seem necessary, the fact is however that our providers are dealing with more and more patients every single day. One example of this would be the fact that the workload of a medical secretary has nearly doubled in the last decade, and where more volume is added the risk of mistakes also increases. Using a digital method for transport will eliminate a lot of the potential for human error within our healthcare network.

That is truly only the start though. Using connected healthcare doctors, specialists, surgeons, imaging techs and pharmacists can all have access to the most up to date and accurate information about their patients. Undoubtedly this will come to benefit us all in ways we cannot even imagine.  

We would like to hear your view of connected healthcare.  To schedule a quick call use the following link  

By Anirban Kundu. This post originally appeared here. 

Much has been said about the value at stake and new growth opportunities presented by the Internet of Things trend. A Cisco estimates puts this at $ 14.4 Trillion opportunity where as a new McKinsey survey values this around $ 6.2 Trillion by 2025. One thing which comes undisputed from various reports across analyst’s community is the significant addition to the global GDP, trade volumes and new opportunities which would be created across sectors and industries.  Most reports in unison claim the benefits of the Internet of Things and the far reaching consequences this would have for the city we live in, the buildings we work and live in to the vehicles we drive. Every aspect of our experience with the physical world would be re-imagined from the way we work, our shopping experience, our medical services to the purchase of the insurance and banking services.

In midst of all these far reaching consequences lies the biggest dilemma for the early adopters of Internet of Things. The promised value seems to be bit more elusive and early adopters still have not found the golden bullet to unlock all the treasure trove as has been outlined in the research. While we are confident about the promises of 2020, the IOT early adopters working in 2016 seems to be in for a “cognitive dissonance”. The journey to the value realization is still more distant and needs some fundamental restructuring of the existing business processes and industry structure as it exists today.

In this blog I intend to take a detailed look at the value realization dilemma with concepts from Economics and Analytics and chart a detailed path to the all elusive value realization. This would lay the foundation of a “Business Value Calculator” for the IOT scenarios which can be adopted by various entities to realize the potential of IOT.  At the onset we need to reexamine the aggregate consumer demand in the context of Internet of Things.

The Promise of the Infinity:

 The “Consumer Demand” curve needs to be revisited in the context of Internet of Things to bring fore the “Promise of the Infinity”.  Today our industry structure and the cost of production imply a physical limit on the profitable supply of the aggregate quantity demanded and is limited by the equilibrium quantity arrived at by the intersection of the supply and demand curve. As can be seen in the Figure below there are 2 major opportunities which has not been part of the revenue for the company namely – consumer surplus and the area of the curve beyond the equilibrium quantity.

Interestingly enough the area beyond the equilibrium does not even have a mention in economics literature due to constraints of profitability. However, in the context of Internet of Things this region which till now has not been accounted in any financial calculations would be critically examined and holds the key for the promise of the infinity.

Fig 1: Consumer Demand Curve/Equilibrium Pricing

Business’s today are based on this demand supply structure where we have spent elaborate efforts to reach the highest possible quantity demanded and continuously worked towards decreasing the price and bringing more customers into the fold. However as with the physical networks this limit is still a finite limit and as such we never had to explore the “fat tail” of the consumer demand curve.  This however is changing with the new business models where products are being offered as services. This fundamental transition has now liberated the current constraints on the product pricing and opens up the “Promise of the Infinity”.  This coupled with the power of the network has now made it possible for the ecosystem to drastically bring down the prices of the products by converting them into usage based services.

With new pricing structure and the offering of the products as services we need to reexamine the demand curve being the equilibrium previously set due to physical constraints.  The new value is now added by the large number of quantity demanded in the calculation of the value captured by the enterprise. While we see the prices of the services driven down we more than compensate this decrease by an exponential increase in the quantity demanded at the price.  This open up 2 interesting analytical scenarios first being the “price elasticity” analysis of the consumers and the second being resource usage analysis. 

The Power of Exponential

We are now in the era of transition where we are set to see that more and more products would be offered as services and as such we are moving to a completely new paradigm of computing the quantity demanded.  In the earlier figure where the limits to quantity demanded were also bound by the limits of affordability. There is a finite limit to the number of the people who could afford to “buy” a Ferrari or the most expensive jets. On the supply side we also would have the limits on to the units produced profitably. This has a fundamental change in the price elasticity of the products v/s service.

As the product purchase is bound by the physical limits there is considerably higher price elasticity than the price elasticity of the “products as a service”. This is a fundamental change which changes the slope of the demand curve and makes it much flatter in case of products as services and hence increasing the quantity demanded exponentially. 

Earlier the revenue recognized by the company was at the time of the purchase and additional services paid by the users. In case of the product as services we would convert one time product cost into usage based pricing and this would imply that the number of transactions in case of the “products as a service” is exponentially higher than the number of products sold.  

In a resource sharing paradigm the quantity defined would be based on the number of times the service is utilized at a reduced price as compared to the outright purchase price. This coupled with the net new users of the services takes the number of transactions as an exponential of the previous constrained quantity supplied.

Fig 2: An exponential increase in the number of transactions resulting from the new business model of products being offered as services

This is the foundation to start the definition of the IOT Value calculator. The final revenue increase is produced by the interaction of the increased quantity demanded and the reduced price of product when offered as a service.  In the next blog we would illustrate a more analytical treatment of the difference in the price elasticity between the two models. Also the usage metrics analysis based on the customer preferences. As in evident in the revenue calculation we have 2 exponential effects against the substantial decrease of the product price. Considering the nature of the inelastic demand curve for the “product as a service” we have the quantity effects far outweigh the effects of the price decrease. A mathematical treatment is available on request.

The analysis therefore lays the foundation for unlocking the elusive value of the IOT. Here we define this from an economic perspective and a follow up paper would be published where a company can simulate the usage behavior, price elasticity and increased number of transactions.

Finally the appeal of Consumer Surplus and Perfect Price Discovery

This is the sweet spot where advanced analytics meets the Economics to present the additional opportunities of mass personalization. We have seen the value which is captured moving down the “fat tail” of the demand curve.  Advanced analytics through segmentation, clustering and perfect price discovery helps us to transform the consumer surplus into economic value. 

While the demand for the product as a service would gather momentum, we would still see the need of mass personalization being driven by the ability of the enterprises to transform their manufacturing facility to enable lot size 1 production.  Harley Davidson had cut the lead time in the development of the customized production to less than 6 hours. This leads the fragmentation of the existing business models fracturing along two paths- one path to capture the high value consumer surplus through value added personalized offering and on the other side we would exponentially increase the number of transactions being offered at a lower price made possible orienting the product offering as services.

With advanced techniques in customer segmentation and the availability of personalized data availability per user we now are able to offer personalized products to translate the consumer surplus to economic value. While the traditional pricing strategies related to segmentation to offer group, channel or regional pricing have been employed successfully in the past to capture more of the consumer surplus, there were still potential to capture additional value specific to individual users. “Mass personalization” would help to transform more of the consumer surplus into economic value.

Bring in the additional value of the consumer surplus and combining it with the value based on the products as a service companies would be able to significantly extract the elusive of the IOT and set us on the path to create an Internet of Things “Value Calculator”. 

UK-based Machina Research is adding to the mix of predictions for IOT with a new Global IoT Market research report.

Their headline today: Global Internet of Things market to grow to 27 billion devices, generating USD3 trillion revenue in 2025

Key findings include:

In a prepared statement Machina Research CEO Matt Hatton commented: “Through our regular ongoing work in our IoT Forecasts Research Stream we are constantly monitoring hundreds of different constituent applications across every country and adjusting our outlook for each. Every year we take a snapshot of the IoT market, pulling our latest forecasts to examine how the overall market had developed in the year. This year the top line figures of 27 billion connections and USD3 trillion of revenue continue are eye-catching and the opportunity is substantial. However it's not just a case of rising tides lifting all boats. To take advantage of the opportunities in IoT, suppliers need to understand the key market dynamics and their competitive environment, and develop best practice. Most of what Machina Research does is focused on supporting various players understand and exploit the opportunities we outline in this study”.

Machina Research focuses on Internet of Things, M2M and Big Data markets. Their ‘IoT Global Forecast & Analysis 2015-2025’ provides an overview of the global IoT market from 2015 to 2025, featuring forecasts of connections, applications, technology, traffic and revenue. It is based on data extracted from Machina Research’s IoT Forecast Database in August 2016. The report is a summary snapshot of the detailed country-by-country and application-by-application forecasts contained within the IoT Forecast Database.

Smarter Cities and How They Can Serve Humanity

Communications technology is progressing at a phenomenal rate, especially when it comes to wireless communications and the ever growing Internet of Things. While many observers and media outlets focus on the benefits of devices and how they will impact consumers, producers, and service providers, there are also huge benefits to be gained by modernizing cities, and progressing towards a smart city model.

A smart city is any city where technology is used to improve public services, safety, and efficiency, and the development of such cities will have major economic and social benefits for individuals and organizations within them.

Major Benefits of Emerging Smart Cities

While many of the consumer technologies in the IoT industry have focused on consumer convenience and entertainment, smart city technologies are aimed more at improving quality of life and providing economic advantages within urban areas.

Transportation

One major area of focus for smart city developers, is transportation. Smart city planning requires that transportation is completely integrated, with mass automation. Big data plays a significant role, as connected sensors record data ranging from traffic statistics, to public transport vehicle location, or even the number of pedestrians who are using a major controlled crossing at any time of the day. A smart city will collect this data to aid urban planning, making it easier for cities to plan new infrastructure.

A smart city can also better manage its transportation infrastructure in real time. Sensor data can help to reroute traffic using electronic road signs, or could automatically adjust signal light timing at major intersections, depending on real time congestion and traffic flow. Rather than urban planners reacting to accumulated data over long time periods, smart cities will have immediate access to sensor data which can be interpreted by machines almost immediately, allowing for traffic management changes to occur within minutes, rather than days or months.

Safety

Safety in large cities has always been a major concern, and a significant area of expenditure for governments. Smart traffic management aids road safety, but other areas of personal safety can also be improved with smart cities. Automation can control lighting in public areas, allowing for increased security. Sensors can alert public services when maintenance needs to be performed on street lighting and traffic signals, and data can be used to increase efficiency of maintenance schedules, resulting in cost savings for large cities. Public cameras can deter and detect crime, and sensors can be used to detect gas leaks, fires, or air quality risks in public spaces. With the integration of location beacons in emergency vehicles, fire, police, and ambulance services can better coordinate coverage in high risk areas, and respond to incidents with increased speed.

Utilities

The benefits even extend into utilities. Sensors on electrical lines can detect faults and control electricity flow in real time. Water lines can also be monitored by IoT connected sensors, allowing for the real time detection of leaks and flow problems. Advanced sensors can even test for water quality along mains. Sensors on gas lines will also increase safety and reduce waste from inefficiency. According to data from the New Jersey Institute of Technology, wide scale smart energy sensors could save the United States up to $1.2 billion dollars per year, and efficiency improvements with other utilities would only add to the potential savings.

Significant Advantages for Stakeholders and Residents

The worldwide smart city technology market is expected to be worth almost $30 billion within the next seven years, a figure that illustrates the huge level of interest from cities and their technology partners.

Smart cities are not just about reducing the costs and resource requirements of the cities themselves, because the benefits will be directly felt by all who live and work within these urban areas. Convenience and quality of life can be improved, and city savings may translate to reduced local rates and taxes, while allowing for increased investment into key infrastructure and public services.

What do you see as the future of smarter cities.   Please call if you would like to discuss and see how we see them unfolding   Click here for a free Consultation 

By Javier Saade. This post originally appeared here.

IoT, smart devices, wearables, mobile technology and nanotech - yes, nanotech - are forcing financial services incumbents and challengers to rethink every aspect of their value chains.  Those value chains are getting to be exponentially more distributed and automated.   Increased digitization means more data being generated, from all kinds of places at an accelerating rate.   IoT, regardless of your perspective, promises to enable the development of new value-added services to improve and automate user engagement, customer acquisition and service delivery - everywhere at all times.  

In insurance for instance, user engagement is very low.  Customers like it that way because there are no incentives for them to interact other than once a year when a policy holder renews it.   But recasting the current low engagement environment with an IoT lens, insurers may be able to develop value-added services that give customers a reason to engage more frequently.  One way to do it is by providing discounts.  An example would be to give customers price breaks if they opt-in to apps that monitor perspiration levels, body temperature, and heart rate via smart clothing.   Sounds far fetched?  Think again.

Sounds far-fetched?  Think again.

My friend David Bray, the FCC’s CIO, once said this:  “...in 1977, 4.2 billion people lived on earth and the first Apple II went on sale running at 1MHz w 4 KB of RAM (note, that is the first half of a second of your favorite MP3 song).”   He continued, "today there are 7 billion people, about 850 million web servers online, and about 4 billion zetabytes of digital content worldwide.  By 2022 there will be 8 billion people, 75-300 billion networked devices globally and 96 zetabytes of digital content is estimated to exist”.

96 zetabytes, by the way is 96,000,000,000,000,000,000,000 bytes = 96 billion trillion bytes.  With this kind of exponential growth the opportunities are incalculable because data is the building block of the digitized economy.  Information its lifeblood and for that reason there are billions being deployed in IoT by players in almost every sector of the economy.   Real money to be sure yet for some products and services, like wearables and smart-home devices, the consumers themselves will bear the costs.  For other products, including but not limited to: automobile driving monitoring devices, smart city clouds, connected cars, smart farming, and industrial embedded data to name a few there is zero or very little incentive for consumers to bear the cost.  So in applications like these, companies are expected to seek partnerships with OEMs and OEDs to embed technologies (e.g., RFID tags) into their products.  Alternatively, innovators in the space may play a more integrated role designing and inventing applications for incumbents delivering IoT enabled services and products. 

RFID involves wireless communication that uses radio waves to identify and track objects.  It is analogous to a smart digital barcoding system that allows users to uniquely identify items without direct line-of-sight, identify thousands of items simultaneously and identify items within a defined proximity.  It can tell you what an object is, where it is, and how it is making the technology an indispensable IoT building block applicable in everything from supply chain and logistics finance to smart payments.   

Another interesting technology being used – telematics.  Telematics hardware uses GPS and wireless devices to collect real-time customer data.  Think about a car insurer adjusting a customers’ premiums based on a panoply of driving behavior and vehicle use.   These devices are now able to measure a number of additional behavioral factors, most notably hard braking (a decline of at least 10 MPH/second), which allows insurers to deeply refine risk models.  This refinement, if executed properly, could lead to potential pricing power and margins.

This refinement, if executed properly, could lead to potential pricing power and margins.    

Other technology evolutions are expected to make IoT even more viable.   One such evolution is miniaturization.  The number of transistors per chip has increased from thousands in the 1950s to over four billion in the present day.  One atom transistors are the natural limit of Moore’s Law.  This limit holds until a paradigm-shifting technology like quantum computing is able to perform at scale.  

Computing power is fundamentally and physically limited by the number of transistors that can fit on a chip.  In quantum computing there are magnitudes improvement in processing power because each quantum bit can theoretically be in an infinite number of states at one time.  In contrast, today there are two states, the well-known binary system which allows only “1s” or “0s”.   Increasing the amount of information conveyed per unit is the most realistic hope of extending Moore’s Law.  And extending Moore's law will give rise to whole new industries (e.g., everything becomes a computer) and super charging others more specifically (e.g., nanomechanics).   From a tech-enabled financial services perspective we can more effectively use real-time and uber-dynamic consumer data, perform individualized and highly contextualized analytics, and apply artificial intelligence to perform and deliver services across the entire value chain.

All of this potential raises security and privacy concerns.  Important everywhere but especially true when dealing people’s money or health.  The IoT’s infrastructure is vulnerable to hacking, almost by design.   Researchers recently claimed that they could access a plane’s satellite communications system during commercial flights via Wi-Fi or the plane’s entertainment console.  Other scary hack situations include thermostats, webcams, insulin pumps, automobiles, pacemakers and refrigerators.  As it applies to distributed ledger technologies, IoT-driven and blockchain-based systems require users to be both sophisticated and vigilant – not something to bet on.  Any systems used for the purpose of processing smart contracts, therefore, needs to be extremely robust and possess design redundancies to ensure the ability to withstand attacks.  This is not a surprise, but in a world where breaches can occur through an infinite number of entry points or nodes, cybersecurity becomes exponentially more important to maintain and difficult to manage.   

The internet of things is an exciting frontier where potentially hundreds of billions of devices will be able to talk to the network and to each other.   This efficiency should lead to goods and services most of us can’t even conceptualize.  This includes how we finance, price, transact and pay for those exact goods and services – B2B, B2C, B2B2C, C2B, P2P, P2C, C2C, O2O, B2G – and every other permutation of effecting commerce and creating or transferring value.  I look forward to keeping a very close eye on developments at this important and evolving intersection.

Note:  The idea for this piece was sparked by a research project our intern Matt completed for our firm, Fenway Summer Ventures. 

Here is the latest issue of the IoT Central Digest. This digest links you to a three part series entitled IoT 101, well worth a read. We also include articles about software tools for IoT device security, dive into fog computing and look at who holds the intellectual property in IoT.  If you're interested in being featured, we always welcome your contributions on all things IoT Infrastructure, IoT Application Development, IoT Data and IoT Security, and more. All members can post on IoT Central. Consider contributing today. Our guidelines are here.

IoT 101 – Everything You Need to Know to Start Your IoT Project

By Bill Vorhies

Summary: This is the first in a series of articles aimed at providing a complete foundation and broad understanding of the technical issues surrounding an IoT or streaming system so that the reader can make intelligent decisions and ask informed questions when planning their IoT system. Visit www.iotcentral.io to read the entire series.

Intellectual Property Held by the Top 100 IoT Startups

Posted by Mitchell Schwartz 

Using Mattermarks’s list of the Top 100 IoT startups in 2015 (ranked by funding, published in Forbes Oct 25, 2015) Ipqwery has looked behind the analytics to reveal the nature of the intellectual property (IP) behind these innovative companies. Our infographic presents a general summary of the IP within the group as a whole, and illustrates the trailing 5-year trends related to IP filing activity.

Automated Software Development Tools for Improving IoT Device Security

Posted by Bill Graham 

For IoT and M2M device security assurance, it's critical to introduce automated software development tools into the development lifecycle. Although software tools' roles in quality assurance is important, it becomes even more so when security becomes part of a new or existing product's requirements.

How IoT can benefit from fog computing

By Ben Dickson

What I’m mentioning a lot these days (and hearing about it as well) is the chaotic propagation and growth of the Internet of Things. With billions of devices slated to connect to the internet every year, we’re going to be facing some serious challenges. I’ve already discussed howblockchain technology might address connectivity issues for huge IoT ecosystems. But connectivity accounts for a small part of the problems we’ll be facing. Another challenge will be processing and making sense of the huge reams of data that IoT devices are generating. Close on its heels will be the issue of latency or how fast an IoT system can react to events. And as always, security and privacy issues will remain one of the top items in the IoT challenge list. Fog computing (aka edge computing) can help mitigate – if not overcome – these challenges

Additional Links

Follow us on Twitter | Join our LinkedIn group | Members Only | For Bloggers | Subscribe

By Bill Vorhies

Bill is Editorial Director for our sister site Data Science Central and has practiced as a data scientist and commercial predictive modeler since 2001. This article originally appeared here. 

Summary:  In this Lesson 3 we continue to provide a complete foundation and broad understanding of the technical issues surrounding an IoT or streaming system so that the reader can make intelligent decisions and ask informed questions when planning their IoT system. 

Continuing from Lesson 2, our intent is to provide a broad foundation for folks who are starting to think about streaming and IoT.  In this lesson we’ll explain how Spark and Storm handle data streams differently, discuss what real time analytics actually means, offer some alternatives for streaming beyond open source, and suggest some trends you should watch in this fast evolving space.

Three Data Handling Paradigms:  SPARK Versus Storm

When users compare SPARK versus Storm the conversation usually focuses on the difference in the way they handle the incoming data stream. 

• Storm processes incoming data one event at a time – called Atomic processing. 
• SPARK processes incoming data in very small batches – called Micro Batch.  A SPARK micro batch is typically between ½ second and 10 seconds depending on how often the sensors are transmitting.  You can define this value.
• A third method called Windowing allows for much longer windows of time and can be useful in some text or sentiment analysis applications, or systems in which signals only evolve over a relatively long period of time.

Atomic:  (aka one-tuple-at-a-time) Processes each inbound data event as a separate element.  This is the most intuitively obvious but also the most computationally expensive design.  For example, it’s used to guarantee fastest processing of individual events with least delay in transmitting the event to the subscriber.  Seen often for customer transactional inputs so that if some element of the event block fails the entire block is not deleted but moved to a bad record file that can later be processed further.  Apache Storm uses this paradigm.

Micro batching:  The critique of this approach is that it processes in batches (not atomic level streaming) but typically those batches are extremely small encompassing actions that occur within only a few seconds.  You can adjust the time window.  This makes the process somewhat more efficient.  SPARK Streaming uses this paradigm.

Windowing:  A hybrid of the two approaches, Windowing maintains the atomic processing of each data item but creates pseudo-batches (windows) to make processing more efficient.  This also allows for many more sophisticated interpretations such as sliding windows (e.g. everything that occurred in the last X period of time). 

All three of these approaches can guarantee that each data element is processed at least once.  Only the Atomic paradigm can guarantee that each data element is processed only once.

Consider this Example 

Your sensors are like FitBits and sample data every 10 seconds.  They transmit that in bursts whenever the sensor is cued to dump its data into a Wi-Fi stream.  One user may monitor the results of the stream many times during the day, valuing low latency and causing his sensor to upload via Wi-Fi frequently.  Another user may not be near a Wi-Fi connection or may simply not bother to download the data for several days.  Still a third user may have trouble with a network connection or the hardware itself that causes the sensor to transmit incomplete or missing packets that are then repeated later or are simply missing from the stream.

In this scenario, data from sensors originating at the same time may arrive at the stream processor with widely different delays and some of those packets that were disrupted may have been transmitted more than once or not at all.

You will need to carefully evaluate whether guaranteeing ‘only once’ processing, or the marginally faster response time of atomic processing warrant using this factor in your selection of the Stream Processor.

Streaming and Real Time Analytics

It’s common in IoT to find references to “real time analytics” or “in stream analytics” and these terms can be misleading.  Real time analytics does not mean discovering wholly new patterns in the data in real time while it is streaming by.  What it means is that previously developed predictive models that were deployed into the Stream Processor can score the streaming data and determine whether that signal is present, in real time.

It’s important to remember that the data science behind your sophisticated Stream Processor was developed in the classic two step data science process. First data scientists worked in batch with historical data with a known outcome (supervised learning) to develop an algorithm that uses the inputs to predict the likelihood of the targeted event.  The model, an algebraic formula, represented by a few lines of code (C, Python, Java, R, and others) is then exported into a program within the Stream Processor and goes to work evaluating the passing data to see if the signal is present.  If it is, some form of action alert is sent to the human or machine, or sent as a visual signal to a dashboard.

Recently the first indications that some new discoveries can be made in real time have been emerging but they are exceedingly rare.  See more in this article.

Beyond Open Source for Streaming

Why would you want to look beyond open source for your IoT system?  Largely because while open source tools and packages are practically free, this is the same as ‘free puppy’. 

Yes these packages can be downloaded for free from Apache but the most reasonable sources are the three primary distributors, Hortonworks, Cloudera, and MapR all of whom make sure the code is kept up to date and add certain features that make it easier to maintain.  Even from these distributors, your total investment should be in the low five figures.  This does not of course include implementation, consulting, or configuration support which is extra, either from the distributors, from other consultants, or from your own staff if they are qualified.

With open source what you also get is complexity.  Author Jim Scott writing about SPARK summed it up quite nicely.  “SPARK is like a fighter jet that you have to build yourself. The great thing about that is that after you are done building it, you have a fighter jet. Problem is, have you ever flown a fighter jet? There are more levers than could be imagined.”

In IT parlance, the configurations and initial programs you create in SPARK or other open source streaming platforms will be brittle.  That is every time your business rules change you will have to modify the SPARK code written in Scala, though Python is also available.

Similarly, standing up a SPARK or Hadoop storage cluster comes with programming and DBA overhead that you may not want to incur, or at least to minimize.  Using one of the major cloud providers and/or adding a SaaS service like Qubole will greatly reduce your labor with only a little incremental cost.

The same is true for the proprietary Stream Processors many of which are offered by major companies and are well tested and supported.  Many of these come with drag-and-drop visual interfaces eliminating the need for manual coding so that any reasonably dedicated programmer or analyst can configure and maintain the internal logic as your business changes.  (Keep your eye on NiFi, the new open source platform that also claims drag-and-drop).

Competitors to Consider

Forrester publishes a periodic rating and ranking of the competitor “Big Data Streaming Analytic Platforms” and as of the spring of 2016 listed 15 worthy of consideration.

Here are the seven Forrester regards as leaders in rank order:

1. IBM
2. Software AG
3. SAP
4. TIBCO Software
5. Oracle
6. DataTorrent,
7. SQLstream

There are eight additional ‘strong performers’ in rank order:

1. Impetus Technologies
2. SAS
3. Striim
4. Informatica
5. WSO2
6. Cisco Systems
7. data Artisans
8. EsperTech

Note that the ranking does not include the cloud-only offerings which should certainly be included in any competitive comparison:

1. Amazon Web Services’ Elastic MapReduce
2. Google Cloud Dataflow
3. Microsoft Azure Stream Analytics

Here’s the ranking chart:

It’s likely that you can get a copy of the full report from one of these competitors.  Be sure to pay attention to the detail.  For example here are some interesting observations from the numerical scoring table.

Stream Handling:  In this presumably core capability SoftwareAG got a perfect score while Impetus and WSO2 scored decidedly below average.

Stream Operators (Programs):  Another presumably core capability.  IBM Streams was given a perfect score.  Most other competitors had scores near 4.0 (out of 5.0) except for data Artisans given a noticeably weak score.

Implementation Support: data Artisans and EsperTech were decidedly weaker than others.

In all there are 12 scoring categories that you’ll want to examine closely.

What these 15 leaders and 3 cloud offerings have in common is that they greatly simplify the programming and configuration and hide the gory details.  That’s a value well worth considering.

Trends to Watch

IoT and streaming is a fast growth area with a high rate of change.  Witness the ascendance of SPARK in just the last year to become the go-to open source solution.  All of this development reflects the market demand for more and more tools and platforms to address the exploding market for data-in-motion applications.

All of this means you will need to keep your research up to date during your design and selection period.  However, don’t let the rate of change deter you from getting started.

• One direction of growth will be the further refinement of SPARK to become a single platform capable of all four architectural elements:  data capture, stream processing, storage, and query.
• We would expect many of the proprietary solutions to stake this claim also.
• When this is proven reliable you can abandon the separate components required by the Lambda architecture.
• We expect SPARK to move in the direction of simplifying set up and maintenance which is the same ground the proprietary solutions are claiming.  Watch particularly for integration of NiFi into SPARK, or at least the drag-and-drop interface elements creating a much friendlier UI.

By Bill Vorhies.

Bill is Editorial Director for our sister site Data Science Central and has practiced as a data scientist and commercial predictive modeler since 2001. This article originally appeared here. 

Summary:  In this Lesson 2 we continue to provide a complete foundation and broad understanding of the technical issues surrounding an IoT or streaming system so that the reader can make intelligent decisions and ask informed questions when planning their IoT system. 

Continuing from Lesson 1, our intent is to provide a broad foundation for folks who are starting to think about streaming and IoT.  In this lesson we’ll dive into Stream Processing the heart of IoT, then discuss Lambda architecture, whether you really need a Stream Processor, and offer a structure for thinking about what sensors can do.

Stream Processing – Open Source

Event Stream Processing platforms are the Swiss Army knives that can make data-in-motion do almost anything you want it to do.

The easiest way to understand ESP architecture is to see it as three layers or functions, input, processing, and output.

Input accepts virtually all types of time-based streaming data and multiple input streams are common.  In the main ESP processor occur a variety of actions called programs or operators.  And the results of those programs are passed to the subscriber interface which can send alerts via human interfaces or create machine automated actions, and also pass the data to Fast and Forever data stores.

It is true that Stream Processing platforms can directly receive data streams, but recall that they are not good at preserving accidentally lost data so you will still want a Data Capture front end like Kafka that can rewind and replay lost data.  It’s likely over the near future that many stream processors will resolve this problem and then you will need to revisit the need for a Kafka front end.

Stream Processing Requirements

The requirements for your stream processor are these:

• High Velocity:  Capable of ingesting and processing millions of events per seconds depending on your specific business need.
• Scales Easily:  These will all run on distributed clusters.
• Fault Tolerant:  This is different than guaranteeing no lost data.
• Guaranteed Processing:  This comes in two flavors: 1.) Process each event at least once, and 2. Process each event only once.  The ‘only-once’ criteria is harder to guarantee.  This is an advanced topic we will discuss a little later.
• Performs the Programs You Need for Your Application.

What Can ESP Programs Do

The real power is in the programs starting with the ability to do data cleansing on the front end (kind of a mini-MDM), then duplicate the stream of data multiple times so that each identical stream can be used in different analytic routines simultaneously without waiting for one to finish before the next begins.  Here’s a diagram from a healthcare example used in a previous article describing how this works that illustrates multiple streams being augmented by static data, and processed by different logic types at the same time.  Each block represents a separate program within the ESP that needs to be created by you.

There are a very large number of different logic types that can be applied through these ESP programs including:

• Compute
• Copy, to establish multiple processing paths – each with different retention periods of say 5 to 15 minutes
• Aggregate
• Count
• Filter – allows you to keep only the data from the stream that is useful and discard the rest, greatly reducing storage.
• Function (transform)
• Join
• Notification email, text, or multimedia
• Pattern (detection) (specify events of interest EOIs)
• Procedure (apply advanced predictive model)
• Text context – could detect for example Tweet patterns of interest
• Text Sentiment – can monitor for positive or negative sentiments in a social media stream

There is some variation in what open source and proprietary packages can do so check the details against what you need to accomplish.

Open Source Options for Stream Processing

The major open source options (all Apache) are these:

Samza:  A distributed stream processing framework. It uses Kafka for messaging, and YARN to provide fault tolerance, processor isolation, security, and resource management.

NiFi: This is a fairly new project still in incubation.  It is different because of its user-friendly drag-and-drop graphical user interface and the ease with which it can be customized on the fly for specific needs.

Storm:  A well tested event based stream processor originally developed by Twitter.

SPARK Streaming:  SPARK Streaming is one of the four components of SPARK which is the first to integrate batch and streaming in a single enterprise capable platform.

SPARK Streaming and Storm Are the Most Commonly Used Open Source Packages

SPARK has been around for several years but in the last year it’s had an amazing increase in adoption, now replacing Hadoop/MapReduce in most new projects and with many legacy Hadoop/MapReduce systems migrating to SPARK.  SPARK development is headed toward being the only stack you would need for an IoT application.

SPARK consists of five components all of which support Scala, Java, Python, and R.

1. SPARK:  The core application is a batch processing engine that is compatible with HDFS and other NoSQL DBs.  Its popularity is driven by the fact that it is 10X to 100X times faster than Hadoop/MapReduce.
2. ML.lib: A powerful on-board library of machine learning algorithms for data science.
3. SPARK SQL:  For direct support of SQL queries.
4. SPARK Streaming:  Its integrated stream processing engine.
5. GraphX:  A powerful graph database engine useful outside of streaming applications.

Storm by contrast is a pure event stream processor.  The differences between Storm and SPARK Streaming are minor except in the area of how they partition the incoming data.  This is an advanced topic discussed later.

If after you’ve absorbed the lesson about data partitioning and you determine this does not impact your application then in open source SPARK / SPARK Streaming is the most likely choice.

Lambda Architecture – Speed Plus Safety

The standard reference architecture for an IoT streaming application is known as the Lambda architecture which incorporates a Speed Layer and a Safety Layer. 

The inbound data stream is duplicated by the Data Capture app (Kafka) and sent in two directions, one to the safety of storage, and the other into the Stream Processing platform (SPARK Streaming or Storm).  This guarantees that any data lost can be replayed to ensure that all data is processed at least once.

The queries on the Stream Processing side may be extracting static data to add to the data stream in the Stream Processor or they may be used to send messages, alerts, and data to the consumers via any number of media including email, SMS, customer applications, or dashboards.  Alerts are also natively produced within the Stream Processor.

Queries on the Storage safety layer will be batch used for creating advanced analytics to be embedded in the Stream Processor or to answer ad hoc inquiries, for example to develop new predictive models.

Do You Really Need a Stream Processor?

As you plan your IoT platform you should consider whether a Stream Processor is actually required.  For certain scenarios where the message to the end user is required only infrequently or for certain sensor uses it may be possible to skip the added complexity of a Stream Processor altogether.

When Real Time is Long

When real time is fairly long, for example when notifying the end user of any new findings can occur only once a day or even less often it may be perfectly reasonable to process the sensor data in batch.

From an architecture standpoint the sensor data would arrive at the Data Capture app (Kafka) and be sent directly to storage.  Using regular batch processing routines today’s data would be analyzed overnight and any important signals sent to the user the following day.

Batch processing is a possibility where ‘real time’ is 24 hours or more and in some cases perhaps as short as 12 hours.  Shorter than this and Stream Processing becomes more attractive.

It is possible to configure Stream Processing to evaluate data over any time period including days, weeks, and even months but at some point the value of simplifying the system outweighs the value of Stream Processing.

Four Applications of Sensor Data

There are four broad applications of sensor data that may also impact your decision as to whether or not to incorporate Stream Processing as illustrated by these examples.

Sensor Direct:  For example, reading the GPS coordinates directly from the sensor and dropping them on to a map can readily create a ‘where’s my phone’ style app.  It may be necessary to join static data regarding the user (their home address in order to limit the map scale) and that could be accomplished external to a Stream Processor using a standard table join or it could be accomplished within a Stream Processor.

Expert Rules:  Without the use of data science, it may be possible to write rules that give meaning to the inbound stream of data.  For example, when combined with the patient’s static data an expert rule might be to summon medical assistance if the patient’s temperature reaches 103°.

Predictive Analytics: The next two applications are both within the realm of data science.  Predictive analytics are used by a data scientist to find meaningful information in the data.

Unsupervised Learning:  In predictive analytics unsupervised learning means applying techniques like clustering and segmentation that don’t require historical data that would indicate a specific outcome.  For example, an accelerometer in your FitBit can readily learn that you are now more or less active than you have been recently, or that you are more or less active than other FitBit users with whom you compare.  Joining with the customer’s static data is a likely requirement to give the reading some context. 

The advantage of unsupervised learning is that it can be deployed almost immediately after the sensor is placed since no long period of time is required to build up training data. 

Some unsupervised modeling will be required to determine the thresholds at which the alerts should be sent.  For example, a message might only be appropriate if the period of change was more than say 20% day-over-day, or more than one standard deviation greater than a similar group of users. 

These algorithms would be determined by data scientists working from batch data and exported into the Stream Processor as a formula to be applied to the data as it streams by.

Supervised Learning:  Predictive models are developed using training data in which the outcome is known.  This requires some examples of the behavior or state to be detected and some examples where that state is not present. 

For example we might record the temperature, vibration, and power consumption of a motor and also whether that motor failed within the next 12 hours following the measurement.  A predictive model could be developed that predicts motor failure 12 hours ahead of time if sufficient training data is available. 

The model in the form of an algebraic formula (a few lines of C, Java, Python, or R) is then exported to the Stream Processor to score data as it streams by, automatically sending alerts when the score indicates an impending failure. 

The benefits of sophisticated predictive models used in Stream Processing are very high.  The challenge may be in gathering sufficient training data if the event is rare as a percentage of all readings or rare over time meaning that much time may pass before adequate training data can be acquired.

Watch for our final installment, Lesson 3.

By Bill Vorhies.

Bill is Editorial Director for our sister site Data Science Central and has practiced as a data scientist and commercial predictive modeler since 2001. This article originally appeared here. 

Summary: This is the first in a series of articles aimed at providing a complete foundation and broad understanding of the technical issues surrounding an IoT or streaming system so that the reader can make intelligent decisions and ask informed questions when planning their IoT system. 

In talking to clients and prospects who are at the beginning of their IoT streaming projects it’s clear that there’s a lot of misunderstanding and gaps in their knowledge.  You can find hundreds of articles on IoT and inevitably they focus on some portion of the whole without an overall context or foundation.  This is understandable since the topic is big and far ranging not to mention changing fast. 

So our intent is to provide a broad foundation for folks who are starting to think about streaming and IoT.  We’ll start with the basics and move up through some of the more advanced topics, hopefully leaving you with enough information to then begin to start designing the details of your project or at least equipped to ask the right questions.

Since this is a large topic, we’ll spread it out over several articles with the goal of starting with the basics and adding detail in logical building blocks.

Is It IoT or Is It Streaming?

The very first thing we need to clear up for beginners is the nomenclature.  You will see the terms “IoT” and “Streaming” used to mean different things as well as parts of the same thing.  Here’s the core of the difference:  If the signal derives from sensors it’s IoT (Internet of Things).  The problem is that there are plenty of situations where the signal doesn’t come from sensors but are handled in essentially the same way.  Web logs, click streams, streams of text from social media, and streams of stock prices are examples of non-sensor streams that are therefore not “IoT”.

What they share however is that all are data-in-motion streams of data. Streaming is really the core concept and we could just as easily have called this “Event Stream Processing”, except that focusing on streaming leaves out several core elements of the architecture such as how we capture the signal, store the data, and query it.

In terms of the architecture, the streaming part is only one of the four main elements we’ll discuss here.  Later we’ll also talk about the fact that although the data may be streaming, you may not need to process it as a stream depending on what you think of as real time.  It’s a little confusing but we promise to clear that up below.

The architecture needed to handle all types of streaming data is essentially the same regardless of whether the source is specifically a sensor or not so throughout we’re going to refer to this as “IoT Architecture”.  And since this is going to be a discussion that focuses on architecture, if you’re still unclear about streaming in general you might start with these overviews: “Stream Processing – What Is It and Who Needs It” and “Stream Processing and Streaming Analytics – How It Works”.

Basics of IoT Architecture – Open Source

Open source in Big Data has become a huge driver of innovation.  So much so that probably 80% of the information available on-line deals with some element or package for data handling that is open source.  Open source is also almost completely synonymous with Apache Institute.  So to understand the basics of IoT architecture we’re going to start by focusing on open source tools and packages.

If you’re at all familiar with IoT you cannot have avoided learning something about SPARK and Storm, two of the primary Apache open source streaming projects but these are only part of the overall architecture.  Also, later in this series we’ll turn our attention to the emerging proprietary non-open source options and why you may want to consider them.

Your IoT architecture will consist of four components: Data Capture, Stream Processing, Storage, and Query.  Depending on the specific packages you choose some of these may be combined but for this open source discussion we’ll assume they’re separate.

Data Capture – Open Source

Think of the Data Capture component as the catchers mitt for all your incoming sources be they sensor, web streams, text, image, or social media.  The Data Capture application needs to:

1. Be able to capture all your data as fast as it’s coming from all sources at the same time.  In digital advertising bidding for example this can easily be 1 million events per second.  There are applications where the rate is even higher but it’s unlikely that yours will be this high.  However, if you have a million sensors each transmitting once per second you’re already there.
2. Must not lose events.  Sensor data is notoriously dirty.  This can be caused by malfunction, age, signal drift, connectivity issues, or a variety of other network, software and hardware issues.  Depending on your use case you may be able to stand some data loss but our assumption is that you don’t want to lose any.
3. Scale Easily:  As your data grows, your data capture app needs to keep up.  This means that it will be a distributed app running on a cluster as will all the other components discussed here.

Streaming data is time series so it arrives with at least three pieces of information: 1.) the time stamp from its moment of origination, 2.) sensor or source ID, and 3.) the value(s) being read at that moment.

Later you may combine your streaming data with static data, for example about your customer, but that happens in another component.

Why Do You Need a Message Collector At All?

Many of the Stream Processing apps including SPARK and Storm can directly ingest messages without a separate Message Collector front end.  However, if a node in the cluster fails they can’t guarantee that the data can be recovered.  Since we assume your business need demands that you be able to save all the incoming data, a front end Message Collector that can temporarily store and repeat data in the case of failure is considered a safe architecture.

Open Source Options for Message Collectors

In open source you have a number of options.  Here are some of the better known Data Collectors.  This is not an exhaustive list.

• FluentD – General purpose multi-source data collector.
• Flume – Large scale log aggregation framework.  Part of the Hadoop ecosystem.
• MQ (e.g. RabbitMQ) There are a number of these lightweight message brokers deriving from the original IBM MQTT (message queuing telemetry transport, shortened to MQ).
• AWS Kinesis – The other major cloud services also have open source Data Collectors.
• Kafka – Distributed queue publish-subscribe system for large amounts of streaming data.

Kafka is Currently the Most Popular Choice

Kafka is not your only choice but it is far and away today’s most common choice used by LinkedIn, Netflix, Spotify, Uber, and AirBNB among others.

Kafka is a distributed messaging system designed to tolerate hardware, software, and network failures and to allow segments of failed data to be essentially rewound and replayed, providing the needed safety in your system.  Kafka came out of LinkedIn in 2011 and is known for its ability to handle very high throughput rates and to scale out.

If your stream of data needed no other processing, it could be passed directly through Kafka to a data store.

Storage – Open Source

Here’s a quick way to do a back-of-envelope assessment of how much storage you’ll need.  For example:

Your system will need two types of storage, ‘Forever’ storage and ‘Fast’ storage.

Fast storage is for real time look up after the data has passed through your streaming platform or even while it is still resident there.  You might need to query Fast storage in just a few milliseconds to add data and context to the data stream flowing through your streaming platform, like what were the min and max or average readings for sensor X over the last 24 hours or the last month.  How long you hold data in Fast storage will depend on your specific business need.

Forever storage isn’t really forever but you’ll need to assess exactly how long you want to hold on to the data.  It could be forever or it could be a matter of months or years.  Forever storage will support your advanced analytics and the predictive models you’ll implement to create signals in your streaming platform, and for general ad hoc batch queries.

RDBMS is not going to work for either of these needs based on speed, cost, and scale limitations.  Both these are going to be some version of NoSQL.

Cost Considerations

In selecting your storage platforms you’ll be concerned about scalability and reliability, but you’ll also be concerned about cost.  Consider this comparison drawn from Hortonworks:

For on premise storage a Hadoop cluster will be both the low cost and best scalability/reliability option.  Cloud storage also based on Hadoop is now approaching 1¢ per GB per month from Google, Amazon, and Microsoft.

Open Source Options for Storage

Once again we have to pause to explain nomenclature, this time about “Hadoop”.  Many times, indeed most times that you read about “Hadoop” the author is speaking about the whole ecosystem of packages that are available to run on Hadoop. 

Technically however Hadoop consists of three elements that are the minimum requirements for it to operate as a database.  Those are: HDFS (Hadoop file system – how the data is stored), YARN (the scheduler), and Map/Reduce (the query system).  “Hadoop” (the three component database) is good for batch queries but has recently been largely overtaken in new projects by SPARK which runs on HDFS and has a much faster query method. 

What you should really focus on is the HDFS foundation.  There are other open source alternatives to HDFS such as S3 and Mongo, and these are viable options.  However almost universally what you will encounter are NoSQL database systems based on HDFS.  These options include:

• Hbase
• Cassandra
• Accumulo
• SPARK
• And many others.

We said earlier that RDBMS was non-competitive based on many factors, not the least of which is that the requirement for a schema-on-write is much less flexible than the NoSQL schema-on-read (late schema).  However, if you are committed to RDBMS you should examine the new entries in NewSQL which are RDBMS with most of the benefits of NoSQL.  If you’re not familiar, try one of these refresher articles here,here, or here.

Query – Open Source

The goal of your IoT streaming system is to be able to flag certain events in real time that your customer/user will find valuable.  At any given moment your system will contain two types of data, 1.) Data-in-motion, as it passes through your stream processing platform, and 2.) Data-at-rest, some of which will be in fast storage and some in forever storage.

There are two types of activity that will require you to query your data:

Real time outputs:  If your goal is to send an action message to a human or a machine, or if you are sending data to a dashboard for real time update you may need to enhance your streaming data with stored information.  One common type is static user information.  For example, adding static customer data to the data stream while it is passing through the stream processor can be used to enhance the predictive power of the signal.  A second type might be a signal enhancement.  For example if your sensor is telling you the current reading from a machine you might need to be able to compare that to the average, min, max, or other statistical variations from that same sensor over a variety of time periods ranging from say the last minute to the last month.

These data are going to be stored in your Fast storage and your query needs to be completed within a few milliseconds.

Analysis Queries:  It’s likely that your IoT system will contain some sophisticated predictive models that score the data as it passes by to predict human or machine behavior.  In IoT, developing predictive analytics remains the classic two step data science process: first analyze and model known data to create the predictive model, and second, export that code (or API) into your stream processing system so that it can score data as it passes through based on the model.  Your Forever data is the basis on which those predictive analytic models will be developed.  You will extract that data for analysis using a batch query that is much less time sensitive.

Open Source Options for Query

In the HDFS Apache ecosystem there are three broad categories of query options.

1. Map/Reduce:  This method is one of the three legs of a Hadoop Database implementation and has been around the longest.  It can be complex to code though updated Apache projects like Pig and Hive seek to make this easier.  In batch mode, for analytic queries where time is not an issue Map/Reduce on a traditional Hadoop cluster will work perfectly well and can return results from large scale queries in minutes or hours.
2. SPARK:  Based on HDFS, SPARK has started to replace Hadoop Map/Reduce because it is 10X to 100X faster at queries (depending on whether the data is on disc or in memory).  Particularly if you have used SPARK in your streaming platform it will make sense to also use it for your real time queries.  Latencies in the milliseconds range can be achieved depending on memory and other hardware factors.
3. SQL:  Traditionally the whole NoSQL movement was named after database designs like Hadoop that could not be queried by SQL.  However, so many people were fluent in SQL and not in the more obscure Map/Reduce queries that there has been a constant drumbeat of development aimed at allowing SQL queries.  Today, SQL is so common on these HDFS databases that it’s no longer accurate to say NoSQL.  However, all these SQL implementations require some sort of intermediate translator so they are generally not suited to millisecond queries.  They do however make your non-traditional data stores open to any analysts or data scientists with SQL skills.

Watch for Lessons 2 and 3 in the next weeks.