IoT and Big Data: Challenges and Applications

With the evolvement and development of IoT, the whole range of all imaginable things and industries becomes smarter: smart homes and cities, smart manufacturing machinery, connected cars, connected health and more. Countless things empowered to collect and exchange data are forming a totally new network – internet of things – the network of physical objects that can gather data in the cloud, transmit data and fulfill users’ tasks.

IoT and big data are right on the way to their hour of triumph. Still, there are some peculiarities and pitfalls to keep in mind to benefit from this innovation. In this article, we are happy to share the knowledge we’ve mined with the years in IoT consulting.

How IoT big data can be applied

First of all, there are various ways to get benefits from IoT big data: in some cases, it’s enough to get by with quick analysis, while some valuable outcomes are available only after deeper data processing.

Real-time monitoring. Big data gathered by connected devices can be used in real-time operations: measure temperature at home or in the office, track physical activities (count steps, monitor movements) and more. Real-time monitoring is highly used in healthcare (for example, to take heart rate, measure blood pressure, sugar). It’s also successfully applied in manufacturing (to control production machinery), agriculture (to monitor cattle and plants) and other industries.

Data analysis. Processing IoT-generated big data, there is the opportunity to go beyond monitoring and get valuable insights from these data: identify trends and tendencies, reveal unseen patterns and find hidden information and correlations.

Process control and optimization. Data that comes from sensors gives additional context to reveal non-trivial issues affecting performance and optimize processes.

• Traffic management: tracking traffic load in various dates and times to work out the recommendations aimed at traffic optimization (for example, increase the number of trains and buses at certain time periods, see if it’s profitable, advise on introducing new schemes of traffic lights and building new roads to make some streets less busy and manage traffic congestions).
• Retail: as some goods are almost over in a shopping place, supermarket’s personnel is informed about it, for example, to refill shelves with merchandise.
• Agriculture: water plants when it’s necessary according to sensors’ data.

Predictive maintenance. The data collected with connected devices can be a reliable source to predict risks, proactively identify potentially dangerous conditions, for example:

• Healthcare: monitoring patients’ state and identifying risks (for example, which patients are at risks of diabetes, heart attacks) to take timely measures.
• Manufacturing: predicting equipment failures.

Not all IoT solutions need big data. It should be also noted, that not all IoT solutions require big data (for example, if an owner of a smart home is going to switch off the light with the help of mobile phone, this operation may be performed without big data). It’s important to consider reducing efforts on processing dynamic data and avoid huge storages of the data, which will not be needed in the future.

Big data challenges in IoT

Huge volumes of data are totally useless, unless they are processed to get something valuable. Also, there are various challenges connected with data collecting, processing and storing.

Data reliability. Although big data is never 100% accurate, it’s important to be sure before analyzing data that the sensors function properly and the quality of the data coming for analysis is reliable and not spoiled with various factors (for example, unfavorable environment in which machinery operate, breakdowns in sensors).

Which data to store. Connected things generate terabytes of data, and it’s a demanding task to choose which data to store and which to drop. What is more, the value of some data is far not on the surface, but you may need this data in the future. And if you decide to store the data for the future, the challenge is to do it with minimal costs (as soon as data storing and processing are rather expensive).

Analysis depth. As soon as not all big data is important, another challenge appears: when is it enough to get by with quick analysis and when deeper analysis can bring more value.

Security. There is no doubt that connected things in various sectors can make our life better, but, at the same time, there are very important concerns about data security. Cyber criminals can get access to data centers and devices, connect to traffic systems, power plants, factories, steal personal data from telecom operators. IoT big data is a relatively new phenomenon for security specialists, and the lack of relevant experience increases security risks.

Big data processing in an IoT solution

In IoT systems, data processing components of an IoT architecture vary depending on the peculiarities of incoming data, expected outcomes and more. We’ve worked out our own approach to processing big data in IoT solutions.

Data comes from sensors connected to things. A “thing” can literally be any object: an oven, a car, a plane, a building, an industrial machine, rehabilitation equipment. Data comes either periodically or in streaming. The latter is essential for real-time data processing and managing things promptly.

Things send the data to gateways which ensure initial data filtering and preprocessing reducing the volume of data transferred to the next IoT system’s blocks.

Edge analytics. Before deep data analysis, it makes sense to conduct data filtering and preprocessing to select most relevant data needed for certain tasks. Also, this stage ensures real-time analytics to quickly recognize useful patterns found earlier by deep analysis in a cloud.

Cloud gateway is necessary for basic protocol translation and communication between different data protocols. It also enables data compression and secure data transmission between a field gateway and central IoT servers.

Data generated by connected devices is stored in its natural format in a data lake. Raw data comes to a data lake with “streams”. The data is kept in a data lake until it can be used for business purposes. Cleaned and structured data is stored in a data warehouse.

Machine learning. The machine learning module generates the models based on previously accumulated historical data. These models are regularly (for example, once in a month) updated with new data streams. Incoming data is accumulated and applied for training and creating new models. When these models are tested and approved by specialists, they can be used by control application which send commands or alerts in response to new sensor data.

To sum it up

IoT generates a lot of big data which can be used for real-time monitoring, analytics, process optimization and predictive maintenance, just to name a few. However, it should be kept in mind that getting valuable insights from huge volumes of data in various formats is not a trivial task: you need to be sure that sensors work properly, the data is securely transmitted and effectively processed. What is more, there is always a question: which data is worth storing and processing (as soon as both these processes are rather expensive).

Despite of potential problems listed above, it should be kept in mind that IoT development gains momentum and helps businesses across multiple industries open new digital opportunities.