Skip to content

feat: native multi-platform builds via runner matrix (no QEMU emulation)#435

Open
tgenov wants to merge 24 commits intodevcontainers:mainfrom
tgenov:main
Open

feat: native multi-platform builds via runner matrix (no QEMU emulation)#435
tgenov wants to merge 24 commits intodevcontainers:mainfrom
tgenov:main

Conversation

@tgenov
Copy link
Copy Markdown

@tgenov tgenov commented Feb 26, 2026

Problem

The current multi-platform build support (platform input) relies on QEMU emulation through Docker buildx. This works but has significant drawbacks:

  • Performance: Cross-architecture emulation is 5-10x slower than native builds. A linux/arm64 build on an amd64 runner can take 30+ minutes for large images.
  • Reliability: QEMU emulation can produce subtle runtime differences and occasionally fails on complex build steps (e.g., compiling native extensions).
  • Cost: The long build times consume more CI minutes than necessary.

GitHub Actions and Azure DevOps both offer native ARM runners (ubuntu-24.04-arm, ARM64 pool), making emulation unnecessary if the action supports a matrix-based workflow where each platform builds natively on its own runner.

Proposed Solution

Add two new inputs that enable a split build/merge workflow:

platformTag (per-platform build phase)

Used in matrix jobs. Each job runs on a native runner for its target platform and sets platformTag to a suffix (e.g., linux-amd64, linux-arm64). The action:

  1. Appends -{platformTag} to each image tag (e.g., myimage:latest-linux-amd64)
  2. Builds using the native runner architecture (no --platform flag needed)
  3. Pushes the platform-specific image

mergeTag (manifest merge phase)

Used in a final job after all matrix builds complete. The action:

  1. Skips the build entirely
  2. Calls docker buildx imagetools create to combine the per-platform images into a multi-arch manifest under the original tag (e.g., myimage:latest)

Example Workflow (GitHub Actions)

jobs:
  build:
    strategy:
      matrix:
        include:
          - runner: ubuntu-latest
            platform: linux/amd64
            platformTag: linux-amd64
          - runner: ubuntu-24.04-arm
            platform: linux/arm64
            platformTag: linux-arm64
    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/setup-buildx-action@v3
      - uses: devcontainers/ci@v0.3
        with:
          imageName: ghcr.io/org/repo/devcontainer
          imageTag: latest
          platform: ${{ matrix.platform }}
          platformTag: ${{ matrix.platformTag }}
          push: always

  manifest:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/setup-buildx-action@v3
      - uses: devcontainers/ci@v0.3
        with:
          imageName: ghcr.io/org/repo/devcontainer
          imageTag: latest
          mergeTag: linux-amd64,linux-arm64

Scope of Work

Per CONTRIBUTING.md, changes must maintain feature parity between GitHub Actions and Azure DevOps, include tests, and compile via ./scripts/build-local.sh.

GitHub Action

  • Add platformTag and mergeTag inputs to action.yml
  • Update runMain() in github-action/src/main.ts:
    • Early return when mergeTag is set (save state for post step)
    • Skip --platform and --output flags when platformTag is set (native runner builds correct arch)
    • Append -{platformTag} suffix to image tags
    • Skip skopeo requirement when platformTag is set
  • Update runPost() in github-action/src/main.ts:
    • Manifest merge via docker buildx imagetools create when mergeTag state is present
    • Direct docker push with platform-suffixed tags when platformTag state is present
  • Add createManifest wrapper to github-action/src/docker.ts

Azure DevOps Task

  • Add platformTag and mergeTag inputs to azdo-task/DevcontainersCi/task.json
  • Mirror the runMain() / runPost() logic in azdo-task/DevcontainersCi/src/main.ts
  • Add createManifest wrapper to azdo-task/DevcontainersCi/src/docker.ts

Common

  • Add createManifest function to common/src/docker.ts using docker buildx imagetools create

Tests

  • Unit test for createManifest in common/__tests__/docker.test.ts
  • Unit tests for platformTag / mergeTag input handling
  • Integration test workflow demonstrating the matrix build pattern

Documentation

  • Update docs/github-action.md with platformTag and mergeTag input descriptions
  • Update docs/azure-devops-task.md with equivalent input descriptions
  • Add native multi-platform example to docs/multi-platform-builds.md alongside the existing QEMU approach

Design Notes

Why two inputs instead of extending platform?

The existing platform input (e.g., linux/amd64,linux/arm64) triggers a single QEMU-emulated build. The new inputs are orthogonal — platformTag tags the output of a single-platform native build, and mergeTag merges tagged outputs. This avoids breaking the existing QEMU-based workflow.

Why docker buildx imagetools create instead of docker manifest?

imagetools create works with remote registry images without pulling them locally, and supports OCI image indexes natively. It is the recommended approach for combining multi-platform images that are already pushed to a registry.

Runner polymorphism

This design makes the action polymorphic across runner types. Without the new flags, the action behaves as before — a single QEMU-emulated build on one runner. With platformTag and mergeTag, the same action drives a matrix of native runners where each builds its own architecture natively, then a final job merges the results.

Mode Flags Runner Build strategy
QEMU (existing) platform only Single runner Cross-arch emulation via buildx
Native (new) platform + platformTag + mergeTag Matrix of native runners Each runner builds its own arch natively

Backwards compatibility

When neither platformTag nor mergeTag is set, the action behaves exactly as before. The existing platform input with QEMU emulation continues to work unchanged.

Open Questions

Polymorphism vs. separation of concerns

The current design overloads a single action with two distinct multi-platform strategies (QEMU vs. native runners) controlled by input flags. An alternative would be to ship the native runner workflow as a separate action (e.g., devcontainers/ci/native-multiplatform) with a dedicated interface, keeping the original action focused on single-runner builds.

Polymorphism keeps the surface area small and avoids forcing users to switch actions when migrating from QEMU to native runners. Separation would make each action's contract simpler and avoid the conditional logic around platformTag/mergeTag that touches both runMain and runPost. Which trade-off does the project prefer?

Note

We are already using the fork internally with GitHub Actions. The DevOps task implementation follows the same patterns but has not been tested in an AzDO pipeline.

Add platformTag and mergeTag inputs to support building on native
ARM runners in a matrix strategy, then merging per-platform images
into a multi-arch manifest via docker buildx imagetools create.

This avoids slow QEMU emulation for multi-platform builds by allowing
each matrix job to build natively for its own platform.
The devcontainer CLI rejects --platform without --output. For native
single-platform builds (platformTag set), use type=docker to load
the image into the local daemon for subsequent docker push.
The devcontainer CLI rejects --platform for docker-compose-based
devcontainers. When platformTag is set, the runner is already the
correct native architecture, so --platform is unnecessary.
- Mirror platformTag/mergeTag logic in azdo-task (task.json inputs,
  runMain/runPost in main.ts, createManifest wrapper in docker.ts)
- Add unit tests for createManifest in common/__tests__/docker.test.ts
- Update docs/github-action.md and docs/azure-devops-task.md input tables
- Add native multi-platform builds section to docs/multi-platform-builds.md
  with examples for both GitHub Actions and Azure DevOps Pipelines
@tgenov
Copy link
Copy Markdown
Author

tgenov commented Feb 26, 2026

@microsoft-github-policy-service agree

@tgenov tgenov marked this pull request as ready for review February 26, 2026 07:41
@tgenov tgenov requested review from a team and stuartleeks as code owners February 26, 2026 07:41
@abdurriq abdurriq requested a review from Copilot March 27, 2026 14:36
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a native (non-QEMU) multi-platform build strategy by splitting per-architecture builds into matrix jobs that push platform-suffixed tags, followed by a final manifest-merge job that publishes a multi-arch tag.

Changes:

  • Introduces platformTag (per-platform tagging/push) and mergeTag (manifest merge) flows for both GitHub Actions and Azure DevOps.
  • Adds a createManifest implementation using docker buildx imagetools create, plus unit tests.
  • Updates docs and action/task metadata to document and expose the new inputs.

Reviewed changes

Copilot reviewed 11 out of 15 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
action.yml Exposes new platformTag / mergeTag inputs for the GitHub Action.
github-action/src/main.ts Implements platform-suffixed tagging, merge-only early return, and post-step manifest creation/push behavior.
github-action/src/docker.ts Adds a GitHub Action wrapper for createManifest.
github-action/dist/sourcemap-register.js Updated build artifact.
github-action/dist/licenses.txt Updated build artifact.
common/src/docker.ts Adds shared createManifest implementation via docker buildx imagetools create.
common/__tests__/docker.test.ts Adds unit tests for createManifest.
azdo-task/DevcontainersCi/task.json Exposes new platformTag / mergeTag inputs for the AzDO task.
azdo-task/DevcontainersCi/src/main.ts Mirrors the platform-suffixed tagging and manifest merge flow in AzDO.
azdo-task/DevcontainersCi/src/docker.ts Adds an AzDO wrapper for createManifest.
docs/multi-platform-builds.md Documents the new native matrix strategy and examples.
docs/github-action.md Documents new GitHub Action inputs.
docs/azure-devops-task.md Documents new AzDO task inputs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

"required": false
},
{
"name": "mergeTag",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Naming could be more clear

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify what naming you'd prefer? Happy to rename.

@abdurriq
Copy link
Copy Markdown

@tgenov Thanks for kickstarting this work, it would be very useful to have native builds. I'll be happy to re-review once the above comments are addressed.

tgenov added 11 commits March 28, 2026 21:07
Move the mergeTag block after the push option filtering logic in both
GitHub Action and Azure DevOps implementations. Previously, mergeTag
would bypass all push gating and could publish manifests on PRs or
when push was set to 'never'.
…orm.ts

Extract buildImageNames and mergeMultiPlatformImages helpers to eliminate
duplicated logic between GitHub Action and Azure DevOps implementations.
… platformTag

- Fail early with a clear error if mergeTag is set without push: always,
  preventing silent no-ops when default push filtering skips the manifest.
- Fail early if both mergeTag and platformTag are set on the same step.
- Simplify redundant return logic in mergeTag runPost blocks.
- Add push: always to manifest job examples in docs.
- Remove duplicate "Creating multi-arch manifest" log from AzDO wrapper
  (mergeMultiPlatformImages already logs this).
- Shorten GH Action group header to avoid repeating the log message.
- Update General Notes to reflect GitHub's hosted ARM runners and link
  to the native matrix strategy.
- Add missing Docker login and buildx setup steps to the AzDO native
  multi-platform example.
@tgenov
Copy link
Copy Markdown
Author

tgenov commented Mar 28, 2026

Thanks for the review. All (but one) comments addressed. Beyond the requested changes, I also added validation that mergeTag and platformTag cannot be set together, and updated the docs General Notes section to reflect GitHub's hosted ARM runners.

@tgenov tgenov requested a review from Copilot March 28, 2026 20:34
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 17 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ageName

- Move isDockerBuildXInstalled() before the mergeTag early return so
  missing buildx fails immediately with a clear error instead of at
  runtime in the post step.
- Validate imageName is set when mergeTag is used.
- Guard buildImageNames calls so empty imageName produces [] instead
  of invalid image refs like ':latest'.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 17 changed files in this pull request and generated 5 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tgenov and others added 3 commits March 28, 2026 22:57
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…xample

- Trim platformTag input and reject values containing whitespace or
  commas with a clear error message.
- Remove platform from the native matrix example since it is ignored
  when platformTag is set.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 17 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

| noCache | false | Builds the image with `--no-cache` (takes precedence over `cacheFrom`) |
| cacheTo | false | Specify the image to cache the built image to |
| platform | false | Platforms for which the image should be built. If omitted, defaults to the platform of the GitHub Actions Runner. Multiple platforms should be comma separated. |
| platform | false | Platforms for which the image should be built. If omitted, defaults to the platform of the GitHub Actions Runner. Multiple platforms should be comma separated. Ignored when `platformTag` is set (matrix mode). |
Copy link

Copilot AI Mar 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The platform input description says it “defaults to the platform of the GitHub Actions Runner”, but this document is for the Azure DevOps task. This should reference the Azure Pipelines agent (or simply “the build agent”) to avoid confusing users.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot apply changes based on this feedback

tgenov and others added 2 commits March 28, 2026 23:18
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants