Multi-Service Monorepo

Pilum is built for monorepos with many services. This guide covers project layout, service selection, dependency ordering, failure handling, parallelism and build cache warming.

Recommended Structure

my-project/
├── .gitignore                # also read by Pilum for discovery
├── .pilumignore              # exclude examples, fixtures, etc.
├── .pilum.yml                # optional project config (webhooks, compose)
├── go.mod                    # or go.work, package.json + lockfile, ...
├── _templates/               # Dockerfiles, shared by services
│   └── go
├── services/
│   ├── api-gateway/
│   │   ├── pilum.yaml        # template: go
│   │   └── main.go
│   ├── user-service/
│   │   ├── pilum.yaml
│   │   └── main.go
│   └── worker/
│       ├── pilum.yaml
│       └── main.go
├── shared/                   # shared libraries (no pilum.yaml)
│   ├── auth/
│   └── database/
└── tools/
    └── my-cli/
        └── pilum.yaml        # type: homebrew

Dockerfiles live in _templates/ at the project root, not in each service directory. A service picks one with template: <file>. Pilum runs docker build from the project root with -f ./_templates/<template> and passes the service’s directory as --build-arg SERVICE_NAME=<path>, so one Dockerfile serves every service in that language. pilum init scaffolds _templates/<language> for you.

services/api-gateway/pilum.yaml

name: api-gateway
type: gcp-cloud-run
project: my-project
region: us-central1
registry_name: services
template: go          # → ./_templates/go

build:
  language: go
  cmd: go build -o ./dist/app .
  env_vars:
    CGO_ENABLED: "0"
    GOOS: linux
    GOARCH: amd64

Always set type:, because it picks the recipe. With only provider: gcp, Pilum falls back to the first GCP recipe, which builds and pushes but never deploys.

Service Discovery

Pilum finds every pilum.yaml under the current directory, up to 4 levels deep, skipping paths matched by .gitignore and .pilumignore:

$ pilum list
Found 4 services:

  • api-gateway
      Provider: gcp
      Project:  my-project
      Region:   us-central1
      Service:

  • user-service
      Provider: gcp
      Project:  my-project
      Region:   us-central1
      Service:

  • worker
      ...

Both flat (api/pilum.yaml) and nested (services/workers/email/pilum.yaml) layouts work, as long as each pilum.yaml is within the depth limit. Service names must be unique across the repo.

Selecting Services

Every pipeline command (deploy, build, publish, push, dry-run) runs on all services unless you narrow it:

# By name: space- or comma-separated
pilum deploy api-gateway worker --tag=v1.0.0
pilum deploy api-gateway,worker --tag=v1.0.0

# By glob (quote it so the shell doesn't expand it)
pilum deploy 'api-*' --tag=v1.0.0

# By provider
pilum deploy --provider=gcp --tag=v1.0.0

# Only services with git changes, plus their dependents
pilum deploy --only-changed --tag=v1.0.0
pilum deploy --only-changed --since=v0.9.0 --tag=v1.0.0

Filters are applied in order: names, then --provider, then --only-changed. A name that matches nothing prints a warning with a suggestion. If you named services and none matched, the command fails.

Only changed services

--only-changed keeps the services affected by files changed since --since, plus uncommitted and untracked files, then adds every service that depends on one of those, transitively. The default --since is the default branch, preferring origin/<default branch> (what a CI checkout has) over a local main or master. The comparison starts from the merge base, so commits made on the base branch after yours branched off don’t count.

A service is affected by a change to:

What changed	Notes
Its own directory	Including its pilum.yaml
A path listed under watch:	Relative to where you run Pilum. For dependencies Pilum can’t trace, such as a shared Node workspace package
A Go package it imports, directly or transitively	Found with one go list -e -deps per module root and build configuration (GOOS, GOARCH, CGO_ENABLED, -tags). If go list fails, those services count as changed
The lockfile or manifest at its workspace root	go.mod, go.sum, go.work, package.json, pnpm-lock.yaml, yarn.lock, Cargo.lock, uv.lock, pom.xml and similar, found the same way as for cache warming
Its Dockerfile template	_templates/<template>

Each selected service is printed with the change that selected it:

$ pilum deploy --only-changed --since=v1.4.0 --tag=v1.5.0
● 3 of 20 service(s) affected by changes since v1.4.0:
  api: pkg/auth/token.go changed
  gateway: depends on a changed service
  worker: pkg/auth/token.go changed

Go imports are traced, other languages are not: a Node app that imports a workspace package elsewhere in the repo only redeploys for that package if you list it under watch::

apps/web/pilum.yaml

name: web
watch:
  - packages/ui

When git can’t compare (a shallow clone without the base commit, or a --since ref that doesn’t exist), Pilum warns and selects every service, suggesting fetch-depth: 0. Deploying too much is recoverable; silently skipping a changed service is not. Outside a git repository, --only-changed has no effect.

When nothing changed (since 0.10), Pilum prints No services changed; nothing to deploy (or build, publish, …) and exits 0, even if you named services on the command line. A release that only touched docs or CI files is a successful no-op. Naming a service that doesn’t exist still fails, because names are checked before the change filter. In 0.9, a run that named services and had nothing to deploy failed with exit code 3.

For the right --since value in CI, see Environments & CI.

Service Dependencies

Use depends_on to order services:

services/api/pilum.yaml

name: api
type: gcp-cloud-run
project: my-project
region: us-central1
registry_name: services
template: go
build:
  language: go
  cmd: go build -o ./dist/app .
depends_on:
  - database-migrations

services/worker/pilum.yaml

name: worker
# ...
depends_on:
  - api
  - database-migrations

Pilum builds a dependency graph and groups services into waves by depth. Services in a wave run in parallel, and waves run one after another. Waves apply to every step tagged build, deploy or execute. Push steps (publish to registry) always run flat in parallel, because pushing an image doesn’t depend on another service.

━━━ Step 4/5: deploy to cloud run ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Wave 1/3:
  ✓ database-migrations   3.2s
  Wave 2/3:
  ✓ api                   2.9s
  Wave 3/3:
  ✓ worker                3.1s

Waves are used only when at least one selected service declares depends_on. --no-deps turns them off and runs every step flat.

If you deploy a subset whose dependencies aren’t selected, those dependencies are not built or updated. The summary says so:

  ⚠ 1 dependency NOT built or updated (not part of this run):
     worker → api
     Include them explicitly if you expected a fresh build.

Failure handling

Failures are handled at two levels:

1. Within a step, a failure skips only that service’s dependents in later waves. Independent services in the same step run to completion:

   ━━━ Step 1/5: build binary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
     Wave 1/2:
     ✗ database-migrations
     ✓ billing
     Wave 2/2:
     ○ api      (dependency database-migrations failed)
     ○ worker   (dependency database-migrations failed)

2. Across steps, once any step has failed, Pilum skips all remaining steps for every service. Only steps marked always_run in the recipe still run, such as the npm recipe’s manifest restore. In the example above, billing built successfully but is not pushed or deployed in this run.

So a monorepo deploy is all-or-nothing per step: if one service’s build fails, nothing reaches the deploy step. The command exits non-zero and reports the error from the failed step.

Circular dependencies

pilum check does not detect cycles. At deploy time Pilum prints:

✗ Dependency order could not be resolved: circular dependency detected involving: [api worker]
✗ Deploying in declaration order — dependent services may run before what they depend on.

Any step that uses waves then fails with the same cycle error. Run pilum dry-run after changing depends_on to catch this before a real deploy.

Parallel Execution

Pilum sizes the worker pool per step:

• Build steps are limited by CPU and memory. Workers = cores ÷ cores per build (default 2), capped by 75% of RAM ÷ the heaviest language’s build memory. Each build gets GOMAXPROCS = cores ÷ workers. Docker build steps are sized from the Docker daemon (on Docker Desktop, the VM).
• Push, deploy and execute steps mostly wait on the network and get one worker per service, up to 16.

pilum deploy --tag=v1.0.0                  # automatic sizing
pilum deploy --max-workers=2 --tag=v1.0.0  # at most 2 at a time, every step
pilum deploy --max-workers=1 --tag=v1.0.0  # one at a time

Set build.resources.cpu and build.resources.memory on services that are heavier than their language default. Details are in Build Cache Warming → Parallelism.

Build Cache Warming

When many services share dependencies, parallel builds on a cold cache each compile the same packages. Before the first build step, Pilum runs one warm command per toolchain and workspace, for example go build over all the Go services’ packages, or pnpm fetch in a temporary copy of the lockfile. The builds then only compile their own code.

━━━ Warming build caches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ go (19 services, auto)    60.1s

On Torch’s 19 Go services, this cut a cold-cache build from 490s to 69s. Warming is automatic for Go, Node, Python, Rust, JVM, .NET and PHP services that have a build.cmd, when two or more share a workspace. It is skipped when no build step runs. Turn it off with --no-warm or build.warm: false. See Build Cache Warming.

Mixed Providers

A single monorepo can deploy to different providers. Each service declares its own type:

services/api/pilum.yaml

name: api
type: gcp-cloud-run
project: my-gcp-project
region: us-central1
registry_name: services
template: go
build:
  language: go
  cmd: go build -o ./dist/app .

services/lambda-handler/pilum.yaml

name: lambda-handler      # also the CloudFormation stack name
type: aws-lambda
project: my-project
region: us-east-1
# Runtime, memory and timeout live in the SAM template (template.yaml) in this directory

tools/cli/pilum.yaml

name: my-cli
type: homebrew
description: My command-line tool
license: Apache-2.0
homebrew:
  project_url: https://github.com/my-org/my-cli
  tap_url: https://github.com/my-org/homebrew-tap
  token_env: GH_TOKEN
build:
  language: go
  version_var: github.com/my-org/my-cli/cmd.version

Deploy them all together:

pilum deploy --tag=v1.0.0

Pilum runs step 1 of every service’s recipe, then step 2, and so on. A service whose recipe has fewer steps has nothing to do in the later ones.

Multi-Region Services

A service can deploy to several regions:

services/global-api/pilum.yaml

name: global-api
type: gcp-cloud-run
project: my-project
registry_name: services
template: go
regions:
  - us-central1
  - europe-west1
  - asia-east1
build:
  language: go
  cmd: go build -o ./dist/app .

Pilum expands this into one instance per region:

$ pilum list
Found 3 services:

  • global-api (us-central1)
      Provider: gcp
      Project:  my-project
      Service:

  • global-api (europe-west1)
      ...

# All regions
pilum deploy global-api --tag=v1.0.0

# One region
pilum deploy "global-api (us-central1)" --tag=v1.0.0

Excluding Services

Use .pilumignore to exclude directories:

.pilumignore

# Development only
examples
fixtures
test-*

# Archived services
archived/
*-deprecated

.gitignore is read too (disable that with --no-gitignore). See Ignore Files for the full syntax.

Naming Conventions

Consistent names make globs and filters useful:

Pattern	Example	Select with
{domain}-{type}	user-api, user-worker	'user-*'
{type}-{name}	api-gateway, worker-email	'worker-*'

Don’t put environments in names (prod-api, staging-api). Use one service with an environments block and --env instead. See Environments & CI.

CI/CD Tips

# Push to main: deploy what changed since the previous push
- uses: actions/checkout@v4
  with:
    fetch-depth: 0
- uses: SID-Technologies/pilum-action@v1
  with:
    command: deploy
    tag: ${{ github.sha }}
    flags: --only-changed --since=${{ github.event.before }}

On a push to main, the default --since (the default branch) is the commit being deployed, so it finds nothing. Always pass --since: the previous push, or the last released tag when deploying tags.

# Pull requests: validate and preview
- run: pilum check
- run: pilum dry-run --only-changed --since=origin/${{ github.base_ref }} --tag=pr-${{ github.event.pull_request.number }}

Restore the language cache (for example actions/setup-go with caching on) so the warm phase is near-instant on CI runners. A complete workflow is in Environments & CI.

Next Steps

• Build Cache Warming — warming and worker sizing in detail
• Environments & CI — --env, locks and GitHub Actions
• Ignore Files — exclude directories
• Service Configuration — every pilum.yaml field
• CLI Commands — all commands and flags