Pipelines Concept in IoT Analytics

Introduction

Companies must develop an open IoT architecture that adopts a holistic approach to data and analytics, allowing them to see a comprehensive picture of their whole manufacturing facility.

"You must combine data from several sources into a single holistic data platform."

"You also need an open, agnostic data pipeline that connects your devices to your platform," said Philipp Redlinger, IoT architect at System Vertrieb Alexander in Germany (SVA). "The pipeline must be able to incorporate data in various forms from numerous sources." As a result, it must be a protocol-agnostic pipeline that can be expanded to include additional protocols. In addition, the platform must be open on both the intake and output sides.

It should be able to share data and insights with end-users through desktop and mobile clients. It must also be able to communicate with other systems via a well-defined API."

"A data pipeline in combination with an open platform is vital to allow interoperability and data-driven apps and end-to-end scenarios," he added.

During last week's Splunk.conf21 event, Redlinger and his SVA colleague, IoT engineer Patrick Nieto Castro, co-presented the "From Sensor To Cloud" session.

The data pipeline of the IoT architecture, according to Redlinger, should not only be "a basic data funnel that just ingests all data" that passes through it.

"It should be able to execute data processing such as converting and filtering your data at an early stage to improve the quality of your data." You'll be able to give high-value insights for your business cases by transforming raw data into refined data that is ideally prepared for your analytical applications."

Create the IoT Analytics Pipeline

A pipeline accepts messages from a channel and allows them to be processed and filtered before being stored in a data storage. A pipeline is used to link a track to a data repository. Navigate to the AWS IoT Analytics console.

1. Navigate to Pipelines in the left navigation window.
2. Make a fresh pipeline:
3. Identifier: streampipeline
4. source of pipeline: streamchannel
5. When you click Next, IoT Analytics will analyze the data from your channel and display the properties of your simulated device. All messages are chosen by default.

source

1. Select Next.
2. You can transform the data in your pipeline and add or delete properties under 'Pipeline activities.'
3. Choose Calculate a message attribute as the type when you click Add Activity.
4. Cost is an attribute.
5. (sub metering 1 + sub metering 2 + sub metering 3) * 1.5 is the formula.
6. You may check your calculation by selecting Update preview, and the cost attribute will show in the message payload below.
7. By choosing Add activity and Remove characteristics from a message, you may add a second activity.
8. Click 'Next' on the id attribute. The id property is a Device Simulator-generated unique identifier. However, it adds noise to the data set.
9. When you click Update preview, the id property in the message payload will be removed.

source

1. Select 'Next.'
2. The output of pipeline: Select 'iotastore' from the 'Edit' menu.
3. Create a Pipeline by clicking the Create Pipeline button.