CLI test failure #4510

Activity

1. github-actions commented on Aug 13, 2026

   

   github-actionsbot

   on Aug 13, 2026 – with GitHub Actions

   ContributorAuthor

   More actions

   CLI test 31654300574 failed on ref refs/heads/main
2. github-actions

   added

   cli-test-failure

   on Aug 13, 2026
3. cklin

   assigned

   Copilot

   and

   cklin

   on Aug 13, 2026
4. Copilot

   linked a pull request that will close this issuePin tutorial pack in CLI authentication tests #4514on Aug 13, 2026
5. cklin commented on Aug 13, 2026

   

   cklin

   on Aug 13, 2026 – with Copilot SWE Agent

   Contributor

   More actions

   Investigation results for run 31654300574:

   • 8/10 matrix jobs failed: CLI v2.23.9, v2.24.3, v2.25.6, and v2.26.2 on both Linux and Windows. Both nightly jobs passed.
   • The common failure is in pack downloads, not extension logic. The stable CLIs reject OCI manifest config fields such as digest/size with UnrecognizedPropertyException (for example, Unrecognized field "digest" (class com.semmle.cli2.pack.docker.DockerManifest$Config)). The remaining packaging, code-scanning-pack, and quick-query failures are downstream of pack setup/download failing.
   • The authentication tests downloaded the floating codeql/tutorial latest version even though they only verify credential prompting. A newly published pack therefore made every supported stable CLI fail this unrelated test.
   • CLI v2.26.3 was released with a fix making pack uploads OCI-compliant, and main has already advanced the newest tested CLI from v2.26.2 to v2.26.3.

   Recommended repairs:

   1. Pin the authentication-test fixture to a known compatible version (codeql/tutorial@0.0.11) so this test does not depend on an unrelated latest pack. This is implemented in Pin tutorial pack in CLI authentication tests #4514; its Linux and Windows CLI checks pass.
   2. Keep the v2.26.3 matrix bump, then rerun the full scheduled matrix. If older supported CLIs still encounter affected manifests, the pack(s) should be republished with the corrected upload format (or registry compatibility restored); retries will not fix malformed/incompatible manifests.
   3. Longer term, retain one explicit live-registry compatibility smoke test, but use versioned packs/controlled fixtures for tests whose subject is extension behavior. That preserves detection of registry regressions without turning unrelated suites into floating integration tests.
6. cklin

   closed this as completedin #4514on Aug 13, 2026
7. cklin

   reopened this on Aug 13, 2026
8. cklin

   marked CLI test failure #4515 as a duplicate of this issue on Aug 13, 2026
9. cklin commented on Aug 13, 2026

   

   cklin

   on Aug 13, 2026

   Contributor

   More actions

   CLI tests are still failing on main: https://github.com/github/vscode-codeql/actions/runs/31717159216
10. github-actions commented on Aug 14, 2026

    

    github-actionsbot

    on Aug 14, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    CLI test 31757281137 failed on ref refs/heads/main
11. cklin commented on Aug 14, 2026

    

    cklin

    on Aug 14, 2026

    Contributor

    More actions

    There is a fix PR pending approval:

    • Pin CLI integration test pack fixtures #4516
12. github-actions commented on Aug 15, 2026

    

    github-actionsbot

    on Aug 15, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    CLI test 31852929364 failed on ref refs/heads/main
13. github-actions commented on Aug 16, 2026

    

    github-actionsbot

    on Aug 16, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    CLI test 31916702462 failed on ref refs/heads/main
14. github-actions commented on Aug 17, 2026

    

    github-actionsbot

    on Aug 17, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    CLI test 31981334184 failed on ref refs/heads/main
15. cklin

    closed this as completedin #4516on Aug 17, 2026