Internet of things

Internet of Things Access Technologies

Internet of Things Access Technologies

The Internet of Things (IoT) has rapidly transformed how we interact with our surroundings. IoT has become integral to our daily lives, from smart homes to industrial automation. At the core of this technological revolution lies the concept of IoT access technologies, which enable devices to communicate and share data seamlessly. This blog post will delve into IoT access technologies, their key features, and their impact on various industries.

IoT access technologies encompass a range of methods that enable devices to connect and communicate within the IoT ecosystem. These technologies include Wi-Fi, Bluetooth, Zigbee, cellular networks, and more. Each technology offers unique features and suitability for specific IoT use cases.

Table of Contents

Highlights: IoT Access Technologies

Wi-Fi:

One of the most widely used IoT access technologies is Wi-Fi. With high data transfer speeds and widespread availability, Wi-Fi enables devices to connect effortlessly to the internet. Wi-Fi allows for convenient and reliable connectivity, whether controlling your thermostat remotely or monitoring security cameras. However, its range limitations and power consumption can be challenging in specific IoT applications.

Cellular Networks:

Cellular networks, such as 3G, 4G, and now the emerging 5G technology, play a vital role in IoT connectivity. These networks offer broad coverage areas, making them ideal for IoT deployments in remote or rural areas. With the advent of 5G, IoT devices can now benefit from ultra-low latency and high bandwidth, paving the way for real-time applications like autonomous vehicles and remote robotic surgery.

Bluetooth:

Bluetooth technology has long been synonymous with wireless audio streaming, but it also plays a significant role in IoT connectivity. Bluetooth Low Energy (BLE) is designed for IoT devices, offering low power consumption and short-range communication. This makes it perfect for applications like wearable devices, healthcare monitoring, and intelligent home automation, where battery life and proximity are crucial.

Zigbee:

Zigbee is a low-power wireless communication standard designed for IoT devices. It operates on the IEEE 802.15.4 standard and offers low data rates and long battery life. Zigbee is commonly used for home automation systems, such as smart lighting, temperature control, and security systems. Its mesh networking capabilities allow devices to form a network and communicate with each other, extending the overall range and reliability.

LoRaWAN:

LoRaWAN (Long Range Wide Area Network) is a low-power, wide-area network technology for long-range communication. It enables IoT devices to transmit data over long distances, making it suitable for applications like smart agriculture, asset tracking, and environmental monitoring. LoRaWAN operates on unlicensed frequency bands, enabling cost-effective and scalable IoT deployments.

Related: Before you proceed, you may find the following post helpful:

  1. 6LoWPAN Range
  2. Blockchain-based Applications
  3. Open Networking
  4. OpenFlow and SDN Adoption
  5. OpenStack Architecture



Internet of Things Access Technologies.

Key IoT Access Technologies Discussion points:


  • Introduction to data and analytics.

  • Fog, edge and cloud computing.

  • Access technology types.

  • Comments on TCP and UDP with IoT.

  • A final note on ethical challenges.

Back to Basics: Internet of Things (IoT).

The Internet of Things consists of a network of several devices, including a range of digital and mechanical objects, with separate access to transfer information over a network. The word “thing” can also represent an individual with a heart- monitor implant or even a pet with an IoT-based collar.

The term “thing” reflects the association of “internet” to devices previously disconnected with internet access. For example, the alarm clock was never meant to be internet-enabled, but now you can connect it to the Internet. With IoT, the options are endless.

Then, we have IoT access technologies. The three major network access technologies for IoT Connectivity are Standard Wireless Access – WiFi, 2G, 3G, and standard LTE. We also have Private Long Range – LoRA-based platforms, Zigbee and SigFox. Mobile IoT Technologies – LTE-M, NB-IoT, and EC-GSM-IoT

IoT Access Technologies

The origins of the Internet that started in the 1960s look entirely different from the map of what we have today. It is now no longer a luxury but more of a necessity. It started with areas such as basic messaging, which grew to hold the elasticity and dynamic nature of the cloud, to a significant technological shift into the world of the Internet of Things (IoT): Internet of Things theory. It’s not about buying and selling computers and connecting them anymore; it’s all about data, analytics, and new solutions, such as event stream processing.

Internet of Things Access Technologies: A New World

A World with the Right Connections

IoT represents a new world where previously unconnected devices now have new communication paths and reachability points. This marks IoT as the next evolutionary phase of the Internet, building better customer solutions.

The revolutionary phase is not just a technical phase; ethical challenges now face organizations, society, and governments. In the past, computers relied on input from humans. We entered keystrokes, and the machine would perform an action based on the input.

The computer had no sensors that could automatically detect the world around it and perform a specific action based on that. IoT ties these functions together. The function could be behavioral, making the object carry out a particular task or provide other information. IoT brings a human element to technology, connecting physically to logic.

 It’s all about data and analytics

The Internet of Things is not just about connectivity. The power of IoT comes from how all these objects are connected and the analytics they provide. New analytics lead to new use cases that will lead to revolutionary ideas enhancing our lives. Sensors are not just put on machines and objects but also on living things. Have you heard of the connected cow? Maybe we should start calling this “cow computing” instead of “cloud computing.” For example, an agricultural firm called Chitale Group uses IoT technologies to keep tabs on herds to monitor their thirst.

These solutions will formulate a new type of culture, intertwining various connected objects and dramatically shifting how we interact with our surroundings. This new type of connectivity will undoubtedly formulate how we live and form the base of a new culture of connectivity and communication.

 

IoT Access Technologies: Data management – Edge, fog, and cloud computing

In the traditional world of I.T. networks, data management is straightforward. It is based on I.P. with a client/server model, and the data is in a central location. IoT brings new data management concepts such as Edge, Cloud, and Fog computing. However, the sheer scale of IoT data management presents many challenges. Low bandwidth of the last mile leads to high latency, and new IoT architectural concepts such as Fog computing, where you analyze data close to where it’s connected, are needed. 

Like the cloud in Skype, Fog is on the ground and best suited for constrained networks that need contextual awareness and quick reaction. Edge computing is another term where the processing is carried out at the furthest point – the IoT device itself. Edge computing is often called Mist computing.

IoT Access Technologies
Diagram: IoT Access Technologies. Source Cisco.

Cloud computing is not everything.

IoT brings highlights the concept that “cloud computing is not everything.” IoT will require onsite data processing; some data must be analyzed in real-time. This form of edge computing is essential when you need near-time results and when there isn’t time to send visual, speed, and location information to the cloud for instructions on what to do next. For example, if a dog runs out in front of your car, the application does not have the time for several round trips to the cloud. 

Services like iCloud have had a rough few years. Businesses are worried about how secure their data will be when using one of the many cloud-based services available. This is because of the iCloud data breach in 2014. However, with the rise of cloud security solutions, many businesses are starting to see the benefits of cloud technology, as they no longer have to worry about their data security.

Internet is prone to latency.

The Internet is prone to latency, and we cannot fix it unless we shorten the links or change the speed of light. Connected cars require the capability to “think” and make decisions on the ground without additional hops to the cloud.

Fog computing is a distributed computing infrastructure located between the edge of a network and the cloud. It is a distributed computing architecture designed to address the challenges of latency and bandwidth constraints introduced by traditional cloud computing. F fog computing decentralizes the computing process rather than relying on a single, centralized data center to store and process data. It enables data to be processed closer to the network’s edge.

Fog computing

Fog computing performs better than cloud computing in terms of meeting the demands of emerging paradigms. However, batch processing is still preferred for high-end jobs in the business world, so it can only partially replace cloud computing. In conclusion, fog computing and cloud computing complement one another while having their advantages and disadvantages. Edge computing is crucial in the Internet of Things (IoT).

Security, confidentiality, and system reliability are research topics in the fog computing platform. Cloud computing will meet the needs of business communities with its lower cost based on a utility pricing model. In contrast, fog computing is expected to serve the emerging network paradigms that require faster processing with less delay and delay jitter. Fog computing will grow by supporting the emerging network paradigms that require faster processing with less delay and delay jitter.

Internet of Things Access Technologies: Architectural Standpoint

From an architectural point of view, one must determine the technologies used to allow these “things” to communicate with each other. And the technologies chosen are determined by how the object is classified. I.T. network architecture has matured over the last decade, but IoT architectures bring new paradigms and a fresh approach. For example, traditional security designs consist of physical devices with well-designed modules.

Although new technologies such as VM NIC Firewalls and other distributed firewalls other than IoT have dissolved the perimeter, IoT brings dispersed sensors outside the protected network, completely dissolving the perimeter to a new level. 

When evaluating the type of network needed to connect IoT smart objects, one needs to address transmission range, frequency bands, power consumption, topology, and constrained devices and networks. The technologies used for these topologies include IEEE 802.15.4, IEEE 802.15.4g and IEEE 802.15.4e, IEEE 1901.2a, IEEE 802.11ah, LoRaWAN, and NB-IoT. The majority of them are wireless. Similar to I.T. networks, IoT networks follow Layer 1 (PHY), Layer 2 (MAC), Layer 3 (I.P.), etc. layers. And some of these layers require optimizations to support IoT intelligent objects.

 

IP/TCP/UDP in an IoT world

The Internet Protocol (I.P.) is an integral part of IoT due to its versatility in dealing with the large array of changes in Layer 1 and Layer 2 to suit the last-mile IoT access technologies. I.P. is the Internet protocol that connects billions of networks with a well-understood knowledge base. Everyone understands I.P. and knows how to troubleshoot it.

It has proven robust and scalable, providing a solid framework for bi-directional or unidirectional communication between IoT devices. Sometimes, the full I.P. stack may not be necessary as protocol overhead may exceed device data. 

More importantly, IPv6. Using I.P. for the last mile of constrained networks requires introducing a new mechanism and routing protocols, such as RLP and adaptation layers, to handle the constrained environments. In addition, routing protocol optimizations must occur for constrained devices and networks. This is where we see the introduction of the IPv6 RPL protocol. IPv6 RPL protocol is a distance-vector routing protocol specifically designed for IoT intelligent objects.

Optimizations are needed at various levels, and control plane traffic must be kept to a minimum, leading to new algorithms such as On-Demand Distance Vector. Standard routing algorithms learn the paths and store the information for future use. This works compared to AODV, which does not send a message until a route is needed.

 

Both TCP and UDP have their place in IoT.

TCP and UDP will play a significant role in the IoT transport layer. TCP for guaranteed delivery or UDP to leave the handling to a higher layer. The additional activities TCP brings to make it a reliable transport protocol come at the overhead cost per packet and session.

On the other hand, UDP is connectionless and often used for performance, which is more critical than packet retransmission. Therefore, a low-power and Lossy Networks (LLN) network may be better suited than UDP & a more robust cellular network for TCP.

Session overhead may not be a problem on everyday I.T. infrastructures. Still, it can cause stress on IoT-constrained networks and devices, especially when a device needs only to send a few bytes per bytes of data per transaction. IoT Class 0 devices that only send a few bytes do not need to implement a full network protocol stack. For example, small payloads can be transported on top of the MAC layer without TCP or UDP.

 

Ethical challenges

What are the ethical ramifications of IoT? In the Cold War, everyone was freaked out about nuclear war; now, it’s all about data science. We are creating billions of connected “things,” and we don’t know what’s to come. The responsibilities around the ethical framework for IoT and the data it generates fall broadly on individual governments and society.

These are not technical choices; they are socially driven. This might scare people and hold them back with IoT, but if you look at technology, it’s been a fantastic force for good. One should not resist IoT and the use cases it will offer our lives. However, I don’t think it will work out well for you if you do.

New technologies will always have risks and challenges. But if you reflect on and look at original technologies, such as the wheel, they created new jobs rather than destroyed old ones. IoT is the same. It’s about being on the right side of history and accepting it.

Cisco and IoT

One technology that has played a significant role in the advancement of IoT is Cisco’s LoRaWAN. LoRaWAN, short for Long Range Wide Area Network, is a low-power, wide-area network protocol for long-range communication between IoT devices.

It operates in the sub-gigahertz frequency bands, providing an extended communication range while consuming minimal power. This makes it ideal for applications that require long-distance connectivity, such as smart cities, agriculture, asset tracking, and industrial automation.

Cisco’s Contribution:

Cisco, a global leader in networking solutions, has embraced LoRaWAN technology and has been at the forefront of driving its adoption. The company has developed a comprehensive suite of LoRaWAN solutions, including gateways, sensors, and network infrastructure, enabling businesses to leverage the power of IoT.

Example Use Case: Cisco’s LoRaWAN-compliant solution.

With Cisco’s LoRaWAN-compliant solution, IoT sensors and endpoints can be connected cheaply across cities and rural areas. Low power consumption also extends battery life up to several years. The solution operates in the 800–900 MHz ISM band around the globe as part of Cisco’s LoRaWAN (long-range wide-area network) solution.

The Cisco Industrial Asset Vision solution includes it and a stand-alone component. Monitoring equipment, people, and facilities with LoRaWAN sensors improve business resilience, safety, and efficiency.

Cisco IoT Solution
Diagram: Cisco IoT Solution. Source Cisco.

Benefits of Cisco’s LoRaWAN:

1. Extended Range: With its long-range capabilities, Cisco’s LoRaWAN enables devices to communicate over several kilometers, surpassing the limitations of traditional wireless networks.

2. Low Power Consumption: LoRaWAN devices consume minimal power, allowing them to operate on batteries for an extended period. This makes them ideal for applications where the power supply is limited or impractical to install.

3. Scalability: Cisco’s LoRaWAN solutions are highly scalable, accommodating thousands of devices and ensuring seamless communication between them. This scalability makes it suitable for large-scale deployments, such as smart cities or industrial IoT applications.

4. Secure Connectivity: Security is a top priority in any IoT deployment. Cisco’s LoRaWAN solutions incorporate robust security measures, ensuring data integrity and protection against unauthorized access.

Use Cases:

1. Smart Agriculture: LoRaWAN allows farmers to monitor soil moisture, temperature, and humidity, optimize irrigation, and reduce water consumption. Cisco’s LoRaWAN solutions provide reliable connectivity to enable efficient farming practices.

2. Asset Tracking: From logistics to supply chain management, tracking assets in real time is crucial. Cisco’s LoRaWAN solutions enable accurate and cost-effective asset tracking, enhancing operational efficiency.

3. Smart Cities: LoRaWAN is vital in building smart cities. It allows municipalities to monitor and manage various aspects, such as parking, waste management, and street lighting. Cisco’s LoRaWAN solutions provide the necessary infrastructure to support these initiatives.

As the IoT ecosystem expands, the choice of access technologies becomes critical to ensure seamless connectivity and efficient data exchange. Wi-Fi, cellular networks, Bluetooth, Zigbee, and LoRaWAN are examples of today’s diverse IoT access technologies. By understanding the strengths and limitations of each technology, businesses and individuals can make informed decisions about which access technology best suits their IoT applications. As we embrace the connected future, IoT access technologies will continue to evolve, enabling us to unlock the full potential of the Internet of Things.

Summary: IoT Access Technologies

The Internet of Things (IoT) has become a pervasive force in our ever-connected world, transforming how we live, work, and interact with technology. As the IoT continues to expand, it is crucial to understand the access technologies that enable its seamless integration. This blog post delved into IoT access technologies, highlighting their importance, benefits, and potential challenges.

Section 1: What are IoT Access Technologies?

IoT access technologies encompass the various means through which devices connect to the internet and communicate with each other. These technologies provide the foundation for IoT ecosystems, enabling devices to exchange data and perform complex tasks. From traditional Wi-Fi and cellular networks to emerging technologies like LPWAN (Low Power Wide Area Network) and 5G, the landscape of IoT access technologies is diverse and constantly evolving.

Section 2: Traditional Access Technologies

Wi-Fi and cellular networks have long been the go-to options for connecting IoT devices. Wi-Fi offers high bandwidth and reliable connectivity within a limited range, making it suitable for home and office environments. On the other hand, cellular networks provide wider coverage but may require a subscription and can be costlier. Both technologies have strengths and limitations, depending on the specific use case and requirements.

Section 3: The Rise of LPWAN

LPWAN technologies have emerged as a game-changer in IoT connectivity. These low-power, wide-area networks offer long-range coverage, low energy consumption, and cost-effective solutions. LPWAN technologies like LoRaWAN and NB-IoT are ideal for applications that require battery-powered devices and long-range connectivity, such as smart cities, agriculture, and asset tracking.

Section 4: The Promise of 5G

The advent of 5G technology is set to revolutionize IoT access. With its ultra-low latency, high bandwidth, and massive device connectivity, 5G opens up a world of possibilities for IoT applications. Supporting many devices in real-time with near-instantaneous response times unlocks new use cases like autonomous vehicles, remote healthcare, and smart industries. However, the deployment of 5G networks and the associated infrastructure pose challenges that must be addressed for widespread adoption.

Conclusion:

Internet of Things access technologies form the backbone of our interconnected world. From traditional options like Wi-Fi and cellular networks to emerging technologies like LPWAN and 5G, each has unique features and suitability for IoT applications. As the IoT expands, it is essential to leverage these technologies effectively, ensuring seamless connectivity and bridging the digital divide. By understanding and embracing IoT access technologies, we can unlock the full potential of the Internet of Things, creating a smarter and more connected future.

Event Stream Processing

Event Stream Processing

 

 

Event Stream Processing

In today’s fast-paced digital world, the ability to process and analyze data in real-time has become crucial for businesses across various industries. One technology that has gained significant attention and adoption is Event Stream Processing (ESP). In this blog post, we will explore what ESP is, its benefits, and its applications in different domains.

Event Stream Processing refers to the ability to process and analyze a continuous flow of events or data in real-time. These events can be generated from a variety of sources, such as sensors, social media feeds, financial transactions, clickstreams, and more. ESP systems are designed to handle high volumes of data and analyze it in real-time, allowing organizations to derive valuable insights and make data-driven decisions.

 

Highlights: Event Stream Processing

  • Massive Amounts of Data

It’s a common theme that the Internet of Things is all about data. IoT represents a massive increase in data rates from multiple sources that need to be processed and analyzed from various Internet of Things access technologies. In addition, various heterogeneous sensors exhibit a continuous stream of information back and forth, requiring real-time processing and intelligent data visualization with event stream processing (ESP) and IoT stream processing.

  • Data Flow

This data flow and volume shift may easily represent thousands to millions of events per second. It is the most significant kind of “big data” and will exhibit considerably more data than we have seen on the Internet of humans. Processing large amounts of data from multiple sources in real time is crucial for most IoT solutions. Making reliability in distributed system a pivotal factor to consider in the design process.

  • Data Transmission

Data transmitted between things instructs how to act and react to certain conditions and thresholds. Analysis of this data turns data streams into meaningful events, offering unique situational awareness and insight into the thing transmitting the data. This analysis allows engineers and data science specialists to track formerly immeasurable processes. 

 

Before you proceed, you may find the following helpful:

  1. Docker Container Security
  2. Network Functions
  3. IP Forwarding
  4. OpenContrail
  5. Internet of Things Theory
  6. 6LoWPAN Range

 



Event Stream Processing.

Key Event Stream Processing Discussion points:


  • Introduction to Analytics and Data handling.

  • Discussion on the IoT Stream Processing.

  • The challenges time series data.

  • Highlighting Event Steam Processing.

  • Dicussion on products that can be used.

 

Back to basics with Stream processing technology

Stream processing technology is increasingly prevalent because it provides superior solutions for many established use cases, such as data analytics, ETL, and transactional applications. It also enables novel applications, software architectures, and business opportunities. With traditional data infrastructures, data and data processing have been omnipresent in businesses for many decades.

Over the years, the collection and usage of data have grown consistently, and companies have designed and built infrastructures to manage that data. However, the traditional architecture that most businesses implement distinguishes two types of data processing: transactional processing and analytical processing.

 

  • A key point: Analytics and data handling are changing.

All this type of new device information enables valuable insights into what is happening on our planet, offering the ability to make accurate and quick decisions. However, analytics and data handling are challenging. Everything is now distributed to the edge, and new ways of handling data are emerging.

To combat this, IoT uses emerging technologies such as stream data processing with in-stream analytics, predictive analytics, and machine learning techniques. In addition, IoT devices generate vast amounts of data, putting pressure on the internet infrastructure. This is where the role of cloud computing comes in useful. Cloud computing assists in storing, processing, and transferring data in the cloud instead of connected devices.

 

Benefits of Event Stream Processing

One of the key benefits of Event Stream Processing is its ability to provide real-time insights. Traditional batch processing involves storing data and analyzing it in batches, which can lead to a delay in obtaining insights. ESP, on the other hand, enables organizations to react and respond to events as they happen, leading to faster decision-making and improved operational efficiency.

Another advantage of ESP is its ability to handle complex event patterns. ESP systems can detect and process complex event patterns in real-time, allowing organizations to identify and respond to critical situations promptly. For example, in the financial industry, ESP can be used to detect fraudulent transactions by analyzing patterns and anomalies in real-time, enabling immediate action to prevent financial loss.

Event Stream Processing Application

Event Stream Processing finds applications in various domains. In the retail industry, ESP can be used to analyze customer behavior and preferences in real-time, allowing retailers to personalize offers and improve customer experience.

In the healthcare sector, ESP can be leveraged to monitor patient data in real-time, enabling early detection of critical conditions and timely intervention. In the transportation industry, ESP can provide real-time insights into traffic patterns, helping to optimize routes and improve transportation efficiency.

To implement Event Stream Processing, organizations can utilize various technologies and tools. Some popular ESP frameworks include Apache Kafka, Apache Flink, and Apache Storm. These frameworks provide the necessary infrastructure and processing capabilities to handle high-speed data streams and perform real-time analytics.

 

IoT Stream Processing: Distributed to the Edge

IoT represents a distributed architecture. We have the distribution of analytics from the IoT platform, either cloud or on-premise, to network edges, making analytics more complicated. A lot of the filtering and analysis is carried out on the gateways and the actual things themselves. These types of edge devices process sensor event data locally.

Some can execute immediate local responses without contacting the gateway or remote IoT platform. A device with sufficient memory and processing power can run a lightweight version of an Event Stream Processing ( ESP ) platform.

For example, Raspberry PI supports complex-event processing ( CEP ). Gateways ingest event streams from sensors and usually carry out more sophisticated steam processing than the actual thing. Some can send an immediate response via a control signal to actuators, causing a state change.

 

Technicality is only one part of the puzzle; data ownership and governance are the other.

 

Time Series Data – Data in Motion

The reaction time must be immediate without delay in specific IoT solutions, such as traffic light monitoring in smart cities. This requires a different type of big data solution that processes data while it’s in motion. In some IoT solutions, there is too much data to store, so the analysis of data streams must be done on the fly while being transferred.

It’s not just about capturing and storing as much data as possible anymore. The essence of IoT is the ability to use the data while it is still in motion. Applying analytical models to data streams before they are forwarded enables accurate pattern and anomaly detection while they are occurring. This analysis offers immediate insight into events enabling quicker reaction times and business decisions. 

Traditional analytical models are applied to stored data offering analytics for historical events only. IoT requires the examination of patterns before data is stored, not after. The traditional store and process model does not have the characteristic to meet the real-time analysis of IoT data streams.

In response to new data handling requirements, new analytical architectures are emerging. The volume and handling of IoT traffic require a new type of platform known as Event Stream Processing ( ESP ) and Distributed Computing Platforms ( DCSP )

Event Stream Processing
Diagram: Event Stream Processing.

 

 

Event Stream Processing ( ESP ) 

ESP is an in-memory real-time process technique enabling the ability to analyze continuously flowing events in data streams. Assessing events in motion is known as “event streams.” This reveals what is happening now and can be used with historical data to predict future events accurately. To predict future events, predictive models are embedded into the data streams.

This type of processing represents a shift in how data is processed. Data is no longer stored and processed; it is analyzed while still being transferred, and models are applied.

ESP applies sophisticated predictive analytics models to data streams and then takes action based on those scores or business rules. It is becoming popular in IoT solutions with predictive asset maintenance and real-time detection of fault conditions.

For example, you can create models that signal a future unplanned condition. This can then be applied to ESP, quickly detecting upcoming failures and interruptions. ESP is also commonly used in network optimization of the power grid and traffic control systems.

ESP is in-memory, meaning all data is loaded into RAM. It does not use hard drives or substitutes, resulting in fast processing, enhanced scale, and analytics. In-memory can analyze terabytes of data in just a few seconds and can ingest from millions of sources in milliseconds. All the processing happens at the system’s edge before data is passed to storage.

How you define real-time depends on the context. Your time horizon will depict whether you need the full power of ESP. Events with ESP should happen close together in time and frequency. However, if your time horizon is over a relatively long period and events are not close together, your requirements might be fulfilled with Batch processing.

 

Batch vs Real-Time Processing

With Batch processing, files are gathered over time and sent together as a batch. It is commonly used when fast response times are not critical and for non-real-time processing. Batch jobs can be stored for an extended period and then executed; for example, an end-of-day report is suited for batch processing as it does not need to be done in real-time.

However, they can scale, but the batch orientation limits real-time decision-making and IoT stream requirements. Real-time processing involves a continual input, process, and output of data. Data is processed in a relatively small period. When your solution requires immediate action, real-time is the one for you. Examples of batch and real-time solutions include Hadoop for batch and Apache Spark focusing on real-time computation.

 

Hadoop vs Apache Spark 

Hadoop is a distributed data infrastructure that distributes data collections across nodes in a cluster. It includes a storage component called Hadoop Distributed File System ( HDFS ) and a processing component called MapReduce. However, with the new requirements for IoT, MapReduce is not the answer for everything.

MapReduce is fine if your data operation requirements are static, and you can wait for batch processing. But if your solution requires analytics from sensor streaming data, then you are better off using Apache Spark. Spark was created in response to the limitations of MapReduce.

 Apache Spark does not have a file system and may be integrated with HDFS or a cloud-based data platform such as Amazon S3 or OpenStack SwiftIt is much faster than MapReduce and operates in memory and real time. In addition, it has machine learning libraries to gain insights from the data and identify patterns. Machine learning can be as simple as a python event and anomaly detection script.