The custom code of your custom step can be either Java 17 or Python 3.9.1. This chapter provides a code example for each programming language.

Python versions 3.10 and higher cannot be used to configure the custom step.

Prerequisites

You have a paid plan.
To execute the following actions, you have to be assigned the Admin role or higher.
You use an IDE of your choice (e.g Eclipse) to develop the custom step for your pipeline.

The custom processing logic and its Intellectual Property (IP) belong to the project owner.
The project owner is liable for damages which are caused by the custom processing logic.
We advise you to perform security and validity scans (especially when using open source code).
We also advise you to verify open source license compliance of your custom processing logic.

Code examples

The code examples provided for Java or Python are explained in detail in the following. They can be customized according to your needs.

Proceed as follows

Download the Java or the Python code example and unzip the file.

Check the provided README file for more information on how to setup the required software.
Change the example according to your needs.
Build the code with your changes.
Zip the file again with the required files.

The executable-manifest.yaml file must be on top level.
Upload the zip file into a custom step of your Pipeline, refer to Configuring a pipeline.

Java example

File	Description	Required
`executable-manifest.yaml`	In the manifest, you configure which java class is used for the startup.	YES
`src/main/java/com/example/myfirstcustomstep/` `AsyncMyFirstCustomStepImpl.class`	This is the java class to start your processing step with. The java class has to listen for incoming data on `System.in` and print out processing results on `System.out`. The name of the class can be changed according to your needs in the `executable-manifest.yaml` file.	YES
`*.jar`	Unlike the Python example, all required dependencies are downloaded locally. When building the artifact with our maven plugin, the dependencies specified in the `pom.xml` file are automatically added to the jar.	YES

Python example

File	Description	Required
`executable-manifest.yaml`	In the manifest, you configure which script file is used for the startup.	YES
`src/step.py`	This is the Python script to start your processing step with. The Python script has to listen for incoming data on `sys.stdin` and print out processing results on `sys.stdout`. The name of the script can be changed according to your needs in the `executable-manifest.yaml` file.	YES
`requirements.txt`	This file is used to switch off the dependency auto detection and enforce specific versions of dependencies. All packages on the pypi.org website are available to list them here, as long as they run on a Linux Ubuntu 18.04 operating system.	NO

Manifest attributes

Attribute

Description

input

Defines how large input data is fetched an processed.

Supported data types:

insights+json: Injected input data is provided as JSON. All input data is embedded directly in the injected data.
insights-payloadRef+json: Injected input data is provided as JSON (small files) or with a payload reference (large files). Small input data is embedded directly in the injected data. Large input data has to be fetched from the S3 object storage by using the payload reference and needs to be processed by using streaming. This can prevent memory congestion.

There is an example in the code on how to decide if the input data is processed embedded or is fetched from the S3 object storage.