Skip to content

feat: declared, programmable and observable sources for toolchains, payloads and plugin tools #755

Closed
#758
Closed
feat: declared, programmable and observable sources for toolchains, payloads and plugin tools#755
#758

Description

@speak-agent
speak-agent
opened on Oct 1, 2026

Motivation

A build uses three kinds of things whose source a project may want to state: the toolchain, the xlings payloads a plugin declares ([feature-xlings.<f>]), and the tools a plugin runs. Today:

  • A payload a plugin declares is provisioned before any build.mcpp runs, whether or not the build uses it. A build.mcpp that names its own tool (o.cmake = "/usr/bin/cmake") still downloads xim:cmake, and with MCPP_NO_AUTO_INSTALL=1 the build is refused before build.mcpp runs (measured on 2026.9.30.2 with a stand-in plugin; [tools.overrides] does not apply to payloads). Only manifest gates (feature, cfg, when) avoid the download.
  • A payload is installed when it is declared, not when it is used (bundletool for every Android build, appimagetool for every Linux build, emit build-database downloads).
  • The toolchain can only be a managed payload: a toolchain a developer already has (a self-built LLVM, a vendor cross toolchain) cannot be named by path, and a build program cannot configure one.
  • Output and records do not say which of these came from the ecosystem default and which a project or machine chose, so a failure cannot be attributed.

Design

Recorded in mcpp-community/mcpp-plugins .agents/docs/2026-10-01-ecosystem-build-plugin-framework-design.md (v3) and in this repository's .agents/docs with the implementing PR. In short:

  1. A source model: each toolchain, payload and plugin tool has one source of class managed, pinned, custom, program or host, with its origin (file:line, environment variable, global config). One decision record (in resolution.json) drives the output, mcpp why, machine output, the build information and the pack record.
  2. Output: a build whose sources are all managed/pinned prints exactly what it prints today. A non-default source gets one Using ... [class · origin] line; the Finished line summarises them; a failure names the source of the tool that failed. --managed-only / MCPP_MANAGED_ONLY=1 refuses any non-default source.
  3. Payload overrides: [xlings.overrides] in the root manifest (also under [target.'cfg(..)']), MCPP_XLINGS_OVERRIDE_<NS>_<NAME>, and ~/.mcpp/config.toml. An overridden payload is not provisioned; it still takes part in version unification; xpkg_dir answers the override; xpkg_program/xpkg_source are added.
  4. On-request provisioning: a payload entry may state provision = "on-request"; a build program asks with xpkg_request, the engine installs every request in one batch and re-runs only the programs that asked.
  5. Custom toolchains: [toolchain] <key> = { path = ..., prefix, sysroot, family, launcher, tools } (a normalized layout, a path is enough), MCPP_TOOLCHAIN=path:<dir>, a bootstrap key, and { configure = "build.mcpp" }: the root build program, compiled with the bootstrap toolchain, runs once in a toolchain phase before the dependency graph is resolved and states the build toolchain.
  6. Specifications and documentation of the plugin framework (SPEC build-plugins, SPEC-004, docs/20, 23, 30, 31, 50, English and Chinese).

Modules involved

modules/manifest, modules/buildmcpp, src/build/prepare/*, src/build/build_program.cppm, src/build/hostprogram.cppm, src/toolchain/*, src/ui.cppm, src/cli*, src/config.cppm, docs and specs.

Activity

  1. speak-agent
    closed this as completedin #758on Oct 1, 2026
  2. speak-agent
    added 2 commits that reference this issue on Oct 1, 2026

    2026.10.1.3: a source for every tool a build uses, declared, programm…

    4d81d06

    0.19.0: one resolver for a member's tool, payloads installed on reque…

    7f5f0ed

  3. speak-agent commented on Oct 1, 2026

    @speak-agent
    speak-agent
    on Oct 1, 2026
    MemberAuthor

    Landed in mcpp 2026.10.1.3

    Every tool a build uses now has a source that can be declared, decided by a build program, and read back. The design record is .agents/docs/2026-10-01-tool-and-toolchain-sources-design.md; the ecosystem side is mcpp-plugins 0.19.0.

    What the engine gained

    • [xlings.overrides] states where a declared payload comes from — in the root manifest (also under [target.'cfg(..)']), as MCPP_XLINGS_OVERRIDE_<NS>_<NAME>, or in ~/.mcpp/config.toml. An overridden payload is not provisioned and does not reach the offline gate. A stated version is checked against every requirement a package of the graph made; without one, a note names the requirement that went unchecked. A dependency that writes the table is refused.
    • provision = "on-request" installs a payload when a build program asks for it (mcpp::xpkg_request). Every request of one invocation is installed together and only the programs that asked run again, so a build whose program names its own tool downloads nothing. mcpp emit build-database installs nothing and records MCPP_BUILD_DATABASE_PAYLOAD_DEFERRED.
    • A toolchain named by path: [toolchain] <key> = { path, prefix, sysroot, family, launcher, tools }, or MCPP_TOOLCHAIN=path:<dir>. mcpp probes the drivers, drives them with its own link model, and writes nothing into the tree. The driver and each stated tool enter the fingerprint by content, and the fast paths decline once one changed.
    • [toolchain] bootstrap, and [toolchain] <key> = { configure = "build.mcpp" } — a toolchain phase where mcpp::phase() is "toolchain" and the root build program states the toolchain with mcpp::toolchain(key, value).
    • A build reports its sources: a line of its own for anything that is not the ecosystem's (Using … [custom · mcpp.toml:22]), a summary on Finished, a record in resolution.json, and mcpp why sources | tool <name> | payload <ns:name>, including as mcpp.why.sources under --format json. --managed-only refuses a build whose sources are not the ecosystem's, naming each.
    • Protocol 15: xpkg_source, xpkg_program, xpkg_request, xpkg_pending, phase, decision, toolchain.

    A project that writes none of the new keys builds exactly as before, and its output is unchanged.

    Measured, not assumed. The behaviour this issue asked for is MCPP_NO_AUTO_INSTALL=1 plus a build program that names its own cmake: mcpp-plugins' tests/cmake-consumer builds, reports Using cmake (mcpp.deps.cmake) ← … [program · build.mcpp:10], and the Finished line carries program: cmake (mcpp.deps.cmake). A validation lab ran eight such cases on ubuntu-24.04, macos-15 and windows-2022.

    What the other platforms found, each with its measurement, is in the PR: a compile command that outgrew the Windows shell limit (and a response file that has to be written in each driver's grammar — LLVM's GNU tokenizer escapes backslashes inside quotes too), a stated linker that reached the link on Linux only, a manifest key that said nothing about the engine floor, which() missing a name that is also a shell builtin, and an executable suffix a host appends itself.

    Released and verified in the ecosystem

    mcpp 2026.10.1.3
mirror: CN (MCPP_HOME=/tmp/…/mcpphome)
warm: a toolchain is installed in this home (toolchain: gcc 16.1.0 (x86_64-linux-gnu))
PASS s755_override      an override skips provisioning, reaches the build program and is reported
PASS s755_toolchain     a toolchain named by path builds, runs and is reported; nothing written into the tree
PASS s755_plugins       plugins 0.19.0: a named cmake is used and the payload is not asked for
summary: 3 passed, 0 failed, 0 skipped

    Three validation labs carry the measurements: speak-agent/mcpp-framework-lab (10 cases on three platforms; naming the host's cmake instead of provisioning saves 23.2 s, 14.8 s and 14.2 s cold, and the saving is the one-time provisioning cost rather than a per-build one), speak-agent/mcpp-toolchain-lab (8 cases; it found that a stated linker reached the link on Linux only), and speak-agent/llvm-macos27-lab (the upstream macOS 27 evidence recorded on #669).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions