Bosch IoT Insights is the central place to collect data from different sources. Before device data can be examined, combined, and visualized, it needs to be processed. Raw device data is often encoded in a non-human-readable technical format. For example, hexadecimal byte strings or complex XML structures. Another challenge are device versions and variants that may send data in incompatible structures. Bosch IoT Insights addresses these project complications with data processing. Data processing allows projects to transform, normalize, enrich, and clean the incoming data.

images/confluence/download/attachments/1298258636/image2020-2-20_16-57-52-version-6-modificationdate-1644486211000-api-v2.png

Features

Decoding
Turns binary data into human-readable data. The decoding feature supports various industry-wide description standards, such as ODX, Fibex, A2L, MDF, DBC, and Google Protocol buffer.
Normalization
Unifies data by bringing together different data sources and unifying their data structures.
Duplicate removal
Removes duplicates to guarantee high data quality with reduced costs.
Filtering
Specific queries can be defined to filter data.
Aggregation
Different data sources can be aggregated together.
Forwarding
Data can be forwarded via the REST API to other cloud services.
Alarm
Specific alarm messages or triggers can be configured and sent via email, SMS, and webhooks.
Custom code
Custom processing scripts (Java, JavaScript, and Python) can be combined with other processors. For an example on how to configure the custom step, refer to Pipelines: Configuring the custom step.

images/confluence/download/attachments/1298258636/image2019-12-19_10-33-4-version-6-modificationdate-1644486211000-api-v2.png

Components

In Bosch IoT Insights, the Data Recorder acts as an endpoint for sent device data and stores the data in a raw format. The Processor component improves the raw data.

A processor contains defined improvement steps. The raw data is processed by those improvement steps until its final target format is reached. This chain of improvement steps is called Pipeline. In a Bosch IoT Insights project, pipelines can be configured with an editor. The editor provides several improvement/processing steps. The editor also allows you to upload your own project-specific processing logic.

File formats

The following file formats are supported:

File format	File extension	Description
ASAM MDF	mdf mf4	Binary file format that stores recorded or calculated data, refer to ASAM MDF
Comma-Separated Values	csv	Delimited file format that separates values with a comma
Tab-Separated Values	tsv	File format that stores values in a table
Vector DBC¹	dbc	File format that describes data over a CAN bus
ASAM ODX¹	odx pdx	File format for vehicle diagnostics, refer to ASAM MCD-2 D
ASAM-A2L¹	a2l	File format for internal ECU variables, refer to ASAM MCD-2 MC
ASAM-Fibex¹	dcm	File format for ECU network systems, refer to ASAM MCD-2 NET
Extensible Markup Language	xml	File format for storing semi-structured data
Protocol Buffers¹	proto	File format developed by Google to serialize data, refer to Protocol Buffers
GPS Exchange Format	gpx	File format to store location data
iTRAMS	bin	Bosch-specific binary file format that extends the Vector DBC file format

¹ The actual input for this file format are hex-encoded binary files, containing measurement information. The mentioned file format contains the meta information for decoding the given binary file.

The standard processing of Bosch IoT Insights also supports zipped content (zip, gzip, and inflate) to save storage space.