markov

Writing Workflows

This guide walks through building a markov workflow step by step, from a minimal example to a multi-workflow pipeline.

Anatomy of a Workflow File

Every markov workflow file is a YAML document with these top-level sections:

entrypoint: main          # Which workflow runs first (required)
namespace: markov-test    # K8s namespace for k8s_job steps
forks: 5                  # Default concurrency for for_each

vars:                     # Global variables
  greeting: "hello"

step_types:               # Reusable step type definitions
  echo_job:
    base: k8s_job
    job:
      image: alpine:3.19

rules:                    # Rule definitions for gate steps
  - name: my_rule
    when: "score > 8"
    action: continue

workflows:                # One or more workflow definitions
  - name: main
    steps:
      - name: first-step
        type: shell_exec
        params:
          command: "echo hello"

Only entrypoint and workflows are required.

Directory Workflows

Single-file workflows are still the simplest place to start. When a workflow grows, you can split it into a directory and pass the directory path to markov run or markov validate.

my-pipeline/
  meta.yaml
  vars.yaml
  rules.yaml
  step_types.yaml
  workflows/
    main.yaml
    per-item.yaml

For larger custom type libraries, step_types.yaml can be replaced by or combined with a step_types/ directory containing one or more YAML maps.

The category files contain their category directly:

# meta.yaml
entrypoint: main
forks: 3

# vars.yaml
items: ["alpha", "bravo", "charlie"]

# workflows/main.yaml
name: main
steps:
  - name: process
    for_each: items
    as: item
    type: shell_exec
    params:
      command: "echo '{{ item }}'"

Run it with:

markov validate my-pipeline
markov run my-pipeline

See examples/dir-based-hello-world/ for a minimal runnable directory workflow.

Starting Simple

The simplest workflow runs a shell command:

entrypoint: hello

workflows:
  - name: hello
    steps:
      - name: say-hello
        type: shell_exec
        params:
          command: "echo 'hello from markov'"

Run it:

markov run hello.yaml

Using Variables

Define variables in the vars block and reference them with {{ }} syntax:

entrypoint: hello

vars:
  greeting: "hello from markov"

workflows:
  - name: hello
    steps:
      - name: say-hello
        type: shell_exec
        params:
          command: "echo '{{ greeting }}'"

Override variables from the command line:

markov run hello.yaml --var greeting="hey there"

Capturing Output with register

Use register to store a step’s output and reference it in later steps:

entrypoint: hello

vars:
  greeting: "hello from markov"

workflows:
  - name: hello
    steps:
      - name: say-hello
        type: shell_exec
        params:
          command: "echo '{{ greeting }}'"
        register: hello_result

      - name: show-date
        type: shell_exec
        params:
          command: "date +%Y-%m-%d"
        register: date_result

      - name: combine
        type: shell_exec
        params:
          command: "echo 'Message: {{ hello_result.stdout }} on {{ date_result.stdout }}'"

Each step type produces different output fields. shell_exec produces stdout, stderr, and exit_code.

Conditional Steps with when

Skip steps based on conditions:

- name: step-two
  type: shell_exec
  when: "value != 'skip-me'"
  params:
    command: "echo 'continued: {{ value }}'"

When the expression evaluates to false, the step is marked as “skipped” (not failed) and execution continues.

Computing Values with set_fact

Use set_fact to compute and store variables:

- name: compute
  type: set_fact
  vars:
    item_count: "{{ items | length }}"
    is_ready: "status == 'ready'"
    data: "{{ raw_json | from_json }}"

Values are merged directly into the workflow context and available to all downstream steps.

Validating State with assert

Use assert to fail the workflow if conditions aren’t met:

- name: check_preconditions
  type: assert
  that:
    - "status == 'ready'"
    - "score >= 4"
  msg: "Preconditions not met"

The workflow fails on the first false expression. If msg is omitted, the default message includes the failed expression.

Multiple Workflows and Sub-Workflows

A file can define multiple workflows. Use the workflow field on a step to invoke another workflow:

entrypoint: main

vars:
  items: ["alpha", "bravo", "charlie"]

workflows:
  - name: main
    steps:
      - name: fan-out
        for_each: "items"
        as: item
        workflow: per-item
        vars:
          value: "{{ item }}"

      - name: done
        type: shell_exec
        params:
          command: "echo 'all done'"

  - name: per-item
    vars:
      value: null
    steps:
      - name: process
        type: shell_exec
        params:
          command: "echo 'processing: {{ value }}'"

Sub-workflow vars with null values are parameter declarations — they document what the workflow expects to receive.

Custom Step Types

Define reusable step types to avoid repeating configuration:

step_types:
  echo_job:
    base: k8s_job
    job:
      image: alpine:3.19
      command: ["/bin/sh", "-c"]
    defaults:
      backoff_limit: 0
      ttl_seconds: 300

workflows:
  - name: main
    steps:
      - name: hello
        type: echo_job
        params:
          args: ["echo 'hello world'"]

The base field specifies which primitive type to wrap. Parameters are merged in order: job < defaults < params < step-level params.

Validating Before Running

Use markov validate to check syntax without executing:

markov validate my-pipeline.yaml

This checks that the entrypoint exists, workflow/step names are unique, types resolve correctly, for_each has the required as field, and unsupported structured YAML fields are rejected. Executor-specific keys remain valid under params.

Common Mistakes

What’s Next