​Docker ​Task ​Runner

Run tasks as Docker containers.

How to use the Docker task runner

The following example shows how to use the Docker task runner to execute commands in a Docker container:

id: docker_task_runner
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      cpu:
        cpus: 1
    commands:
      - echo "Hello World!"

Once you specify the taskRunner type, you get autocompletion and validation for the runner-specific properties. In the example above, the task allocates one CPU to the container.

Docker task runner properties

The only property required by the taskRunner is the containerImage property, which must be set on the script task. The image can be from a public or private registry.

Additionally, when using the Docker task runner, you can configure memory allocation, volumes, environment variables, and more. For a full list of available properties, refer to the Docker plugin documentation or explore them in the built-in Code Editor in the Kestra UI.

Task runner behavior in a failure scenario

In general, each task runner container initiated by Kestra will continue running until the task completes, even if the Kestra worker is terminated (for example, due to a crash). However, there are a few caveats depending on how Kestra and the task runner are deployed.

Kestra running in a Docker container, task runner running in DinD

When Kestra runs in a Docker container and uses DinD for task runners, terminating the Kestra container will also terminate the DinD container and any running task containers inside it. No container is automatically restarted.

Kestra running in Kubernetes, task runner running in DinD

When Kestra and DinD are deployed in the same pod in a Kubernetes environment, the pod will restart if the Kestra Worker fails. This ensures that both the DinD container and any task runner containers are restarted.

Kestra deployed with Docker Compose, task runner running in DinD

When using Docker Compose, Kestra and DinD containers can be managed independently. Restarting the Kestra container does not automatically restart the DinD container. Therefore, task runners running inside DinD may continue running even if Kestra is restarted.

Insecure registry

The Docker task runner supports insecure registries. Before using this feature, ensure the insecure registry is configured on the host machine where your Kestra server is running.

For example, if you have the insecure registry 10.10.1.5:5000, add the following configuration to /etc/docker/daemon.json, then restart your Docker daemon:

{
  "insecure-registries": ["10.10.1.5:5000"]
}

You can then use this configuration in a flow with the following YAML:

id: docker_example
namespace: demo

tasks:
  - id: my_command
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      config: |
        {
          "insecure-registries": ["10.10.1.5:5000"]
        }
    containerImage: 10.10.1.5:5000/my-image
    commands:
      - echo "Hello World!"