Integrating with Relay
There's a growing ecosystem of integrations that connect Relay with other tools and services, but it's a big world out there. There are always useful integrations that don't yet exist, but should. This guide will help you build an integration between Relay and an external service, so you can automate away your pain.
Decide on a goal
This may seem self-evident, but it's important to determine what your goal is before you begin coding. Relay has several useful points of extensibility, and understanding the purpose of each will help you find a starting point. Integrations consist of containers that are used in different parts of Relay, which have two flavors: steps and triggers.
- Steps - Relay runs steps, passing in parameters and secrets, as part of an automation workflow. Most of the work in Relay is done by steps, which use an SDK to retrieve and modify a workflow's state as it executes.
- Triggers - Relay supports several types of triggers, which are event handlers that initiate workflow runs. The Relay service handles
push
triggers natively, butwebhook
triggers work by executing a user-defined container to handle the payload of the webhook's incoming HTTP request. Therefore, integrations that connect to Relay using webhooks need to provide a trigger container.
Steps and triggers come together in workflows, YAML code written to accomplish the task you're faced with. Workflow authoring is covered in the Using workflows documentation, so here we'll focus on creating and using steps and triggers.
Creating containers for Relay
In its simplest form, a Relay container image runs, does some work, and then exits. It's possible to use any OCI-compliant container, as long as its entrypoint terminates with a zero exit code upon success and a non-zero code upon failure. However, beyond "hello world" examples, you'll likely want an image with more capabilities than just using alpine:latest
. There are two possible paths here, depending on your use case.
- If the work you're trying to do can be written in a shell or Python script, use the
relaysh/core
image. This is an Alpine-based image that the Relay team maintains, which comes pre-loaded with the Relay SDK. There arerelaysh/core:latest
andrelaysh/core:latest-python
flavors available. You can use these either as a base image in your own Dockerfile or in workflows directly, by specifying aninputFile
tag whose value is a URL to your script. - If there's an existing image that's made for your integration target, you can use it as the starting point for a custom container. Adding the Relay tools in at build time will allow you to use the metadata service via either the command-line interface or code SDK.
Using relaysh/core
Using the relaysh/core
image is the easiest option because it avoids having to build and push a custom container image, but it restricts you to providing your script in a single file. To use it, make your script accessible via HTTPS — GitHub repositories or gists both work fine — and provide the URL to it in the inputFile
attribute on a step
or trigger
definition. For very simple scripts, you can use the input
attribute and provide your commands as an array of shell commands.
There are two variants of relaysh/core
, indicated by their tag: relaysh/core:latest
accepts a Bash shell as input and relaysh/core:latest-python
expects a Python script. Specify which you want to use in the image
attribute of your definition.
relaysh/core shell example
Because webhook trigger containers need to handle a web request, the shell image isn't suitable for triggers, only steps.
steps:
- name: run-my-script
image: relaysh/core:latest
inputFile: https://git.io/JfiwE
relaysh/core Python example
The Python image is usable for both steps and triggers. The following workflow receives a pushed tag from Docker Hub and sends a notification with the name of the tag.
parameters:
dockerTagName:
description: The tag of the container which was pushed
triggers:
- name: my-python-trigger
source:
type: webhook
image: relaysh/core:latest-python
inputFile: https://git.io/JfiwD
binding:
parameters:
dockerTagName: ${event.tag}
steps:
- name: my-python-notification
image: relaysh/core:latest-python
inputFile: https://TODO/
spec:
tag: ${parameters.dockerTagName}
Using an upstream image
While it's possible to use an upstream image without modification, customizing it can greatly increase its usefulness. Adding the Relay SDK or ni
command-line utility to the container enables access to the Relay service APIs, which allow you to retrieve and set parameters, access secrets, and use Relay's persistent logging framework. Additionally, writing a custom entrypoint allows you to control the container's behavior to ensure it runs correctly in Relay.
For trigger containers, we recommend using the relaysh/core:latest-python
image as your base image and adding your own code as the entrypoint. See the Writing entrypoint code section for details.
This section assumes you're familiar with Dockerfiles and the container build/push process.
Adding the CLI
The ni
CLI utility is a small Go program meant to be run inside Relay containers. The most common use for it is to get and set variables from the Relay service API from entrypoint shell scripts. To add it to your container, use a snippet like this in your Dockerfile:
RUN set -eux ; \
mkdir -p /tmp/ni && \
cd /tmp/ni && \
wget https://packages.nebula.puppet.net/sdk/ni/v1/ni-v1-linux-amd64.tar.xz && \
wget https://packages.nebula.puppet.net/sdk/ni/v1/ni-v1-linux-amd64.tar.xz.sha256 && \
echo "$( cat ni-v1-linux-amd64.tar.xz.sha256 ) ni-v1-linux-amd64.tar.xz" | sha256sum -c - && \
tar -xvJf ni-v1-linux-amd64.tar.xz && \
mv ni-v1*-linux-amd64 /usr/local/bin/ni && \
cd - && \
rm -fr /tmp/ni
This will download, verify, and install the latest version of the CLI. To use it in your entrypoint, the ni get
subcommand will (with no arguments) print a JSON string containing all the parameters listed in the step's spec
section. Given a parameter to retrieve, it'll print that in string form (without an enclosing JSON object).
The ni output set
subcommand will write values back to the metadata service for use by later steps, and ni log
will write log messages which can be helpful when debugging.
parameters:
message:
description: "The message to output from the final step"
steps:
- name: generated-output
image: relaysh/core
input:
- ni output set --key dynamic --value "$(/bin/date)"
- name: hello-world
image: relaysh/core
spec:
message: ${parameters.message}
dynamic: ${outputs.'generated-output'.dynamic}
supersecret: ${secrets.mysecret}
input:
- echo "Hello world. Your message was $(ni get -p {.message}), and the generated output was $(ni get -p {.dynamic})."
- echo "Everything I know about your run is contained in $(ni get)"
- ni log info "This worked out wonderfully."
Adding the SDK
If you're using Python in your container, the latest version of the Relay SDK can be installed via pip
in your Dockerfile:
RUN pip --no-cache-dir install relay-sdk
The SDK itself has a top-level README and inline API documentation; check out the latest docs at readthedocs.io.
Writing entrypoint code
Whichever route you've chosen, at this point you've got the setup for a Relay-friendly container. The last part is probably the most interesting, because it's where you need to write enough code to make the container do your bidding. We'll handle the examples for steps and triggers slightly differently, since there are different APIs you'll be working with.
Trigger entrypoints and webhook API
Under the hood, when a workflow references a webhook trigger, the Relay app will prompt the user to register the webhook on the remote service. Once it's set up, Relay associates all incoming requests to their associated trigger containers. When a request comes in, Knative launches the appropriate container and connects the incoming payload to the entrypoint, an HTTP handler.
The container does the following:
- Starts a webserver on the port specified by the environment variable
$PORT
(defaults to8080
). - Handles a POST request and decode the incoming payload.
- Decides whether to run the rest of the workflow.
- Maps values from the request payload onto event data.
- Sends this mapping back into the Relay service as an event.
The Relay Python SDK is by far the easiest way to do this. The Integration template repository has a simple starting point, and there are full-featured examples for Dockerhub push events and new Pagerduty incidents. Check out the Relay integrations on Github for more ideas.
Step entrypoints
Steps have a relatively easy lot in life. They run, do some work to advance the state of the workflow, and exit. In many cases, a step can use existing scripts which you modify only enough to take advantage of the Relay service API. The Getting started shows an example workflow that uses the input
map to access this API for retrieving and setting keys and values to pass data through the workflow. For more advanced use cases, you'll probably need a script that uses the ni utility or its python SDK equivalent to get and set data.
Formally, a step container running in Relay can expect the following:
- It will be executed without persistent storage.
- Only keys declared in the step's
spec
map in the workflow will be available in the metadata service. - It can set output values through the metadata service which will be available to later steps.
- Exiting with a zero exit code will cause the workflow run to continue.
- Exiting with a non-zero exit code will cause the workflow run to be terminated and no dependent steps will run.
For examples of using the ni
utility in shell commands, see the kustomize integration; for examples of using the Python SDK, the AWS EC2 integration has several steps of varying complexity.
Building and publishing containers
Once you've got your base image and custom code together, you can proceed with building and publishing the containers. Relay has conventions for container and integration metadata which aren't required but will increase consistency and help with future compatibility. These conventions are encoded in the template integration repo, specifically, Relay containers should be built with LABEL
directives that indicate their title and description.
The following Dockerfile shows this in action:
FROM relaysh:core-python
COPY "./handler.py" "/step-github-trigger-pull-request-merged.py"
ENTRYPOINT []
CMD ["python3", "/step-github-trigger-pull-request-merged.py"]
LABEL "org.opencontainers.image.title"="GitHub pull request merged"
LABEL "org.opencontainers.image.description"="This trigger fires when a GitHub pull request is merged."
You can publish images to any publicly-accessible container registry. The service defaults to Docker Hub for workflow image
attributes that are not fully-qualified with a hostname, so these are equivalent:
steps:
- name: default-registry
image: relaysh/core
- name: explicit-registry
image: index.docker.io/relaysh/core
Designing a complete integration
While Relay supports running any Linux container, it is most powerful when metadata accompanies the container. The metadata helps the service describe and validate the image's requirements. We call the combination of metadata and image definitions a Relay integration.
You may have already taken a look at the integrations we publish in our GitHub organization. Each of these is based on an initial template that provides a starting point for the required parts of an integration:
- A file called
integration.yaml
at the root of the integration that defines the integration name and version - For each step image, a file called
step.yaml
in the respective directory (for a step namedfoo
, the path would besteps/foo/step.yaml
) - For each trigger image, a file called
trigger.yaml
in the respective directory (for a trigger namedbar
, the path would betriggers/bar/trigger.yaml
)
For a deeper look at Relay integrations, take a look at the integration layout RFC.
Image metadata
In addition to providing a name and version, the information in a step.yaml
or trigger.yaml
enables Relay to give users more context about the image when they use it.
Specifying the following fields will help users get the best experience when they use your images:
summary
: A human-readable title for the image, like "List EC2 instances"description
: A paragraph or two describing the purpose of the image and how to work with it in Markdown formatexamples
: A list of examples or templates that a user can start fromschemas
: JSON Schema validation schemas for yourspec
s, outputs, and events
Working with examples
Any step or trigger can provide zero or more usage examples, either in the form of a single step or trigger or an entire workflow. You must indicate whether an example is for a step, trigger, or workflow by using an API version and kind. For example,
examples:
- summary: Create a new GitHub issue
content:
apiVersion: v1
kind: Step
name: create-issue-on-github
image: relaysh/github-step-issue-create
spec:
title: My first issue
- summary: Use a user-specified parameter to create a new GitHub issue
content:
apiVersion: v1
kind: Workflow
parameters:
title:
description: The GitHub issue title
steps:
- name: create-issue-on-github
image: relaysh/github-step-issue-create
spec:
title: ${parameters.title}
Working with schemas
Relay generates documentation and performs validation when you provide schemas for your steps and triggers. In step metadata, schemas look like this:
schemas:
spec:
$schema: 'http://json-schema.org/draft-07/schema#'
type: object
properties:
title:
type: string
required: [title]
outputs:
$schema: 'http://json-schema.org/draft-07/schema#'
type: object
properties:
issueNumber:
type: integer
required: [issueNumber]
In trigger metadata, the schemas are:
schemas:
spec:
$schema: 'http://json-schema.org/draft-07/schema#'
type: object
properties: {} # ...
event:
$schema: 'http://json-schema.org/draft-07/schema#'
type: object
properties: {} # ...
For a particular schema, you can also write it in a separate file and then reference it. This is useful if your editor provides JSON Schema meta-validation.
schemas:
spec:
source: file
file: spec.schema.json
Our JSON Schema documentation renderer supports one extension, x-relay-connectionType
, that you can add to an object schema to indicate that the value must come from a Relay connection. For example, to indicate that a step requires an AWS connection, you might write:
# ...
connection:
type: object
x-relay-connectionType: aws
You can also use the writeOnly
schema property to hint to a user that a value typically comes from a secret:
# ...
password:
type: string
writeOnly: true