Skip to content

feat(presets): support regex selectors for command declarations - #4776

Open
lmtyy wants to merge 5 commits into
github:mainfrom
lmtyy:feat/4659-regex-preset-selectors
Open

lmtyy wants to merge 5 commits into
github:mainfrom
lmtyy:feat/4659-regex-preset-selectors

Conversation

@lmtyy

@lmtyy lmtyy commented Sep 28, 2026 •

Copy link
Copy Markdown

feat(presets): support regex selectors for command declarations

Closes #4659.

This replaces the previous (much larger) version of this PR with the focused change the
maintainer asked for: regex:<pattern> support for preset command declarations only.
Everything else that had accumulated here is split out into subsequent targeted PRs; the
full earlier work is preserved on the branch feat/4659-regex-preset-selectors-full
(2aa5e892) as the source for those follow-ups.

What this does

A preset may use a bounded selector in a provides.templates entry of type command:

provides:
  templates:
    - type: command
      name: 'regex:^speckit\\.(plan|tasks|implement)$'
      file: commands/workflow-guidance.md
      strategy: append
  • The selector is expanded, in declaration order, into the concrete lower-layer commands it
    matches. Those concrete names then flow through the existing command resolution,
    registration, composition and removal paths.
  • Matching is against the full logical command name only — never a path, file name or
    substring — and only against commands that actually exist in the layers below (enabled
    presets, enabled extensions, core, including built-in core). The declaring preset itself,
    higher-priority presets and project-local overrides are not used as evidence of existence.
  • A literal selector string is never used as an output path, a .composed file name or a
    registry key: the registry only ever sees concrete names.
  • A regex: name on a non-command declaration is a validation error, not a silent
    no-op.
  • Entries without the prefix keep exact-name matching and are completely unaffected.
  • Diagnostics: preset info reports the commands a selector currently matches;
    preset resolve <name> names the declaration that matched when a layer came from a
    selector.
  • Matching is deterministic (declaration order preserved, overlapping declarations are not
    de-duplicated away) and uses the bounded matcher with its existing limits.

Scope: what is deliberately not here

Split into follow-up PRs, each based on its own point in upstream/main:

  • template and script selectors, and selector support on the extension side;
  • automatic reconciliation after provider/integration changes (see the known limitation);
  • historical recovery sweeps and the ownership/ledger redesign beyond the minimal guard;
  • the install-transaction rework, snapshot/journal machinery and CLI boundary refactors
    (design/cli.md, design/shared.md).

Per the review: the ownership subsystem (ownership.py) and the related overwrite-policy
changes have been removed — they are not required to deliver command selectors and are
better handled separately. Coverage now lives in the existing surface-owned test suites, with
one new focused module for the selector engine (tests/specify_cli/presets/test_selector_engine.py).

Known limitations (please review with these in mind)

  1. No automatic broadcast on provider change. A selector is expanded at install/enable
    time and the existing reconciliation of concrete names still runs, but there is no new
    mechanism that re-expands selectors when a new provider/integration appears. This is the
    main user-visible limitation of the reduced scope and the reason the change is small.
  2. Bounded regex subset. The matcher intentionally implements a bounded subset rather
    than full Python regex compatibility; the limits and the rejection behaviour are covered
    by tests.

Tests / evidence

  • The only new test module is tests/specify_cli/presets/test_selector_engine.py, the single
    focused module for the selector engine (parsing and bounded matching).
  • Everything else is asserted in the existing surface-owned suites: test_resolver.py,
    test_manager_commands.py, test_manager.py, test_manager_skills.py, test_manifest.py,
    test_command_info.py, test_command_resolve.py.
  • Preset suite on this revision: 890 passed. Full suite: 12,461 passed, 98 skipped, 0 failed.
  • uvx ruff@0.15.0 check src tests passes; compileall and git diff --check are clean.
  • No tests are skipped, weakened, or marked expected-failure to reach these numbers.
  • Every review finding fixed in this revision has regression coverage in an existing suite that
    fails before the change and passes after it, including both command-file and skill
    reconciliation where the shared filter applies.

AI disclosure

Authored with AI assistance (Hermes Agent); reviewed and verified against the test suite by
a human before submission.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Command resolution, skill registration, diagnostics, warnings, and lifecycle reconciliation have unresolved functional gaps.

Review effort: Balanced
Findings: 3 High severity · 2 Medium severity

Open (5)
What changed in this PR

Adds regex: resource selectors to preset resolution and command materialization.

Changes:

  • Validates and full-matches regex selectors.
  • Expands command selectors and adds lifecycle reconciliation.
  • Adds resolver and selector coverage.
File Description
tests/​specify_cli/​presets/​test_regex_selectors.py Tests selector matching and resolution.
src/​specify_cli/​presets/​command_set_priority.py Reconciles commands after priority changes.
src/​specify_cli/​presets/​command_info.py Displays selector matches.
src/​specify_cli/​presets/​command_enable.py Reconciles enabled preset commands.
src/​specify_cli/​presets/​_selectors.py Implements selector helpers.
src/​specify_cli/​presets/​_resolver.py Resolves regex template and script layers.
src/​specify_cli/​presets/​_manifest.py Validates regex expressions.
src/​specify_cli/​presets/​_manager.py Integrates expansion into lifecycle handling.
src/​specify_cli/​presets/​_manager_commands.py Expands command selectors.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/specify_cli/presets/_resolver.py Outdated
Comment thread src/specify_cli/presets/_resolver.py Outdated
Comment thread src/specify_cli/presets/command_info.py Outdated
Comment thread src/specify_cli/presets/_manager.py Outdated
Comment thread src/specify_cli/presets/_manager_commands.py Outdated
@mnriem

mnriem commented Sep 28, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

@lmtyy

lmtyy commented Sep 28, 2026

Copy link
Copy Markdown
Author

Addressed all five Copilot review findings:

  • Added command regex selectors to normal layer resolution and composition.
  • Preserved all matching regex declarations in manifest order.
  • Fixed the invalid _selectors imports used by diagnostics.
  • Reused selector expansion for AI-skill registration.
  • Added selector-aware reconciliation for relevant preset/extension lifecycle changes.

Also added regression coverage for overlapping selectors, command composition strategies, diagnostics, AI-skill registration, and lifecycle state changes.

Validation:

  • 8624 passed, 17 skipped
  • 183 targeted regression tests passed
  • Ruff passed
  • Python compilation passed
  • git diff --check passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@mnriem mnriem added the triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate label Sep 28, 2026
@lmtyy

lmtyy commented Sep 29, 2026

Copy link
Copy Markdown
Author

All previous Copilot review findings have been addressed and the updated test suite is passing. The latest Copilot re-review appears to have failed due to a review error, so the PR is ready for another review when convenient.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Selector lifecycle paths can leave stale commands or skills and several acceptance criteria remain incomplete.

Review effort: Balanced
Findings: 6 Medium severity

Open (6)

Comment thread src/specify_cli/presets/_manager.py Outdated
Comment thread src/specify_cli/presets/_manager.py Outdated
Comment thread src/specify_cli/presets/_manager_skills.py
Comment thread src/specify_cli/presets/command_info.py Outdated
Comment thread src/specify_cli/presets/command_set_priority.py Outdated
Comment thread tests/specify_cli/presets/test_regex_selector_lifecycle.py Outdated
@mnriem

mnriem commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Selector lifecycle transitions can leave stale or untracked command and skill artifacts.

Review effort: Balanced
Findings: 3 Medium severity · 2 Low severity

Open (5)
Resolved since last review (6)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Stale artifacts remain when selectors lose all matches

src/​specify_cli/​extensions/​_commands.py:121

This refresh runs only after the extension state has changed, so a selector whose sole lower layer was disabled or removed now expands to nothing. register_enabled_presets_for_agent() never sees the previously materialized concrete name, and its reconciler also skips names with no remaining layers, leaving the old command/SKILL artifact active. Capture the pre-mutation matches (or persisted registrations), reconcile the union with post-mutation matches, and explicitly unregister names that no longer resolve.

Comment thread src/specify_cli/presets/_manager_commands.py Outdated
Comment thread src/specify_cli/presets/_manager_commands.py Outdated
Comment thread src/specify_cli/presets/_manager_commands.py
Comment thread src/specify_cli/presets/command_info.py
Comment thread tests/specify_cli/presets/test_regex_selector_lifecycle.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Selector cleanup, disabled-extension resolution, and installation rollback still have correctness gaps.

Review effort: Balanced
Findings: 1 High severity · 3 Medium severity

Open (4)
Resolved since last review (5)
Previously missed (2)

In code that hasn't changed since last review

Medium severity Selector expansion failure leaves partial installation

src/​specify_cli/​presets/​_manager.py:456

Selector diagnostics and expansion run after the preset directory and registry entry are created but before the existing rollback guard. _expand_command_selectors() deliberately raises when artifact enumeration fails, so such a failure leaves an enabled, partially installed preset behind. Move both calls into the guarded installation phase so the established cleanup executes.

Medium severity Re-enumeration causes false failure after registration

src/​specify_cli/​presets/​_manager.py:507

This repeats artifact enumeration outside the transactional registration block. If inventory changes or becomes unreadable after registration, the command reports installation failure even though the preset and artifacts are already committed. Reuse the concrete command_templates computed for registration.

Comment thread src/specify_cli/presets/_resolver.py Outdated
Comment thread src/specify_cli/integrations/_command_upgrade_layout.py Outdated
Comment thread src/specify_cli/presets/_manager_commands.py
Comment thread src/specify_cli/presets/command_disable.py Outdated
@mnriem

mnriem commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Please address Copilot feedback

@lmtyy

lmtyy commented Sep 30, 2026

Copy link
Copy Markdown
Author

@mnriem I've addressed the latest Copilot feedback in commit 1ce59c1, including the disabled-extension fallback, include_disabled handling, stale selector tracking/cleanup, and install rollback coverage.

When convenient, could you please re-request the Copilot review? Thanks!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Selector rollback, disabled-state handling, and extension-layer deduplication contain unresolved correctness issues.

Review effort: Balanced
Findings: 1 High severity · 5 Medium severity

Open (6)
Resolved since last review (3)

Comment thread src/specify_cli/extensions/__init__.py Outdated
Comment thread src/specify_cli/integrations/_command_upgrade_layout.py Outdated
Comment thread src/specify_cli/presets/_manager.py Outdated
Comment thread src/specify_cli/presets/_resolver.py Outdated
Comment thread src/specify_cli/presets/command_disable.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/specify_cli/extensions/__init__.py Outdated
Comment thread src/specify_cli/presets/_resolver.py Outdated
Comment thread src/specify_cli/presets/command_info.py Outdated
Comment thread src/specify_cli/presets/command_set_priority.py Outdated
@mnriem
mnriem requested a balanced review from Copilot September 30, 2026 16:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Comment thread src/specify_cli/ownership.py Outdated
Comment thread src/specify_cli/presets/_manager.py Outdated
@mnriem

mnriem commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Please keep this PR focused on command selector parsing, bounded matching, expansion, resolution, diagnostics, and the reconciliation strictly required for those selectors. Remove ownership.py and the related overwrite-policy changes unless you can demonstrate that they are necessary to deliver this feature and cannot be handled separately.

Please also place coverage in the existing surface-owned test suites, with at most one new focused module for the selector engine. Add a negative test for an unmatched selector alongside an exact declaration with the same target name. If the ownership changes remain, they additionally re

A preset may declare `regex:<pattern>` in a `provides.templates` entry of type
`command`. The selector expands, in declaration order, into the concrete
lower-layer commands it matches; those concrete names then flow through the
existing command resolution, registration, composition and removal paths.
Matching is against the full logical command name only, never a path, file name
or substring, and a literal selector is never used as an output path, a
`.composed` file name or a registry key. A `regex:` name on a non-command
declaration is a validation error. Diagnostics: `preset info` reports current
matches and `preset resolve <name>` attributes selector-contributed layers.

Review follow-up (this revision):
- Removed the ownership subsystem (`ownership.py`) and all related
  overwrite-policy changes, which had grown into this branch and were not
  required by the selector feature.
- Kept the change to command selector parsing, bounded matching, expansion,
  resolution, diagnostics, and the reconciliation strictly required by those
  selectors.
- Moved coverage into the existing surface-owned test suites; no new test
  modules are added.
- Added the requested negative coverage: an unmatched selector alongside an
  exact declaration for the same target name must not warn.

AI disclosure: this change was authored with AI assistance (Hermes Agent);
behaviour was reviewed and verified against the test suite by a human.
@lmtyy
lmtyy force-pushed the feat/4659-regex-preset-selectors branch from 44c1990 to 4b4347a Compare October 14, 2026 14:52
@lmtyy

lmtyy commented Oct 14, 2026

Copy link
Copy Markdown
Author

Hi — thank you for the clear guidance.

I have reduced the PR to what you asked for: ownership.py and all related overwrite-policy changes are removed, and the change is now limited to command selector parsing, bounded matching, expansion, resolution, diagnostics and the reconciliation those selectors require. Coverage has moved into the existing surface-owned suites — only one new module remains, test_selector_engine.py, for the parser and matcher — and I added the requested negative test (unmatched selector alongside an exact declaration for the same name no longer warns), plus coverage for the two re-anchored findings.

It is now a single commit on current main, 18 files, +3,258/−117. Locally: 12,436 passed, 0 failed; preset suite 865 passed; ruff clean.

The workflow runs need approval again after this push — would you mind approving them? Thanks again.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Comment thread src/specify_cli/presets/_manager_skills.py
Comment thread src/specify_cli/presets/_resolver.py
Comment thread src/specify_cli/presets/_resolver.py Outdated
Comment thread src/specify_cli/presets/_resolver.py Outdated
@mnriem

mnriem commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

After the scope-reduction force push, these are the findings that should be addressed in this PR:

The other review threads were resolved because they target code removed by 4b4347a8, contradict the requested exact-declaration warning behavior, or concern provider-change reconciliation that was explicitly deferred from this reduced PR. Please keep the follow-up changes and regression coverage limited to the four findings above.

AI disclosure: Posted for @mnriem by GitHub Copilot using GPT-5.6 Sol in human-directed agent mode; review analysis, scope triage, and comment drafting.

- Skill rendering no longer prefers a stale `.composed` artifact left behind by
  a retired selector match: composed content is reused only while the current
  winning declaration stack still composes and belongs to this preset, so an
  exact `replace` fallback is rendered instead.
- Every selector compilation cause in `REGEX_COMPILE_ERRORS` is reported for
  persisted manifests, not only `SelectorSafetyError`; unrelated invalid
  manifests stay silent.
- Manifests whose command declarations are all exact no longer trigger the
  lower-layer extension scan: exact declarations are collected first and
  eligibility is calculated only when a regex declaration needs it, so a corrupt
  extension registry cannot affect exact resolution.
- Removed the duplicate conventional lookup that admitted the undocumented
  `templates/templates/commands/` location; exact resolution and selector
  candidates now share the documented locations.

Regression coverage for each finding lives in the existing surface-owned suites
(`test_manager_skills.py`, `test_resolver.py`); all four tests fail before the
fix and pass after it.

AI disclosure: authored with AI assistance (Hermes Agent); reviewed and verified
by a human against the test suite.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Resolver fallbacks can let unmatched selectors create resources or satisfy higher selectors, and some diagnostics remain incomplete.

0 open findings

14 resolved since last review
Previously missed (4)

In code that hasn't changed since last review

Medium severity Preserve ineligible selector declarations to suppress convention fallback

src/​specify_cli/​presets/​_resolver.py:650

An ineligible matching selector is collapsed to [], which callers interpret as “this command is undeclared” and then fall back to convention lookup. If its payload is commands/foo.md, resolve("foo", "command") therefore returns that file even with no lower foo resource, so the selector creates the resource it was meant only to augment. Preserve the matching declaration with a None candidate to suppress fallback, and add a conventional-path regression case.

Medium severity Prevent unmatched regex declarations from chaining via conventional payloads

src/​specify_cli/​presets/​_resolver.py:678

This helper still treats an unmatched regex declaration's conventional payload as a concrete lower resource. Thus a lower preset containing name: regex:^foo$ and file: commands/foo.md can make a higher selector eligible, allowing regex declarations to chain even though the contract forbids using another selector as evidence. Consult _preset_declarations_for_resource first and only use legacy convention fallback when no declaration matched.

Medium severity Resolve dotless command names as commands for selector attribution

src/​specify_cli/​presets/​command_resolve.py:67

The new selector attribution remains unreachable for valid dotless command names. Command manifests allow names such as new, and a selector can match a lower preset's exact new command, but preset resolve new classifies it as a template because line 25 uses the presence of . as the type discriminator. The command is then reported as missing instead of naming the selector declaration; provide an explicit or unambiguous command-type resolution path and cover this accepted name shape.

Low severity Clarify validation versus warning behavior for oversized resource names

docs/​reference/​presets.md:341

This says every listed limit is enforced during manifest validation with PresetValidationError, but the 1,024-character resource-name limit is only checked at match time; selector_matches catches that error, warns, and returns False. Clarify that only pattern/state/edge limits invalidate the manifest, while oversized concrete names become warned non-matches.

🧠 Review effort: Balanced

@lmtyy

lmtyy commented Oct 14, 2026

Copy link
Copy Markdown
Author

Hi — all four findings are addressed in 0a0b024:

  • Stale composed files: composed content is now reused only when the current winning declaration still composes and belongs to this preset, so a retired selector match can no longer override the exact fallback.
  • Malformed selectors: every compilation cause in REGEX_COMPILE_ERRORS is reported for persisted manifests; unrelated invalid manifests stay silent.
  • Extension scans: exact declarations are collected first, and lower-layer eligibility is computed only when a regex declaration needs it, so exact resolution no longer touches the extension registry.
  • Duplicate lookup: the second conventional lookup is removed; exact resolution and selector candidates now share the documented locations.

Each fix has a regression test in the existing suites that fails before the change and passes after it. Full suite is green locally (12,444 passed, 0 failed), and CI is green across Ubuntu, macOS and Windows on 3.13 and 3.14. Copilot's latest pass reports 0 open findings.

Ready for another look when you have a moment — thanks again.

@mnriem

mnriem commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Please address the previously missed ones. If not relevant please explain why. If already done, please resolve the conversation for it and mention that in the summary. Almost there!

- An eligible-by-pattern selector that matches no lower-layer resource is no
  longer collapsed to nothing. The matching declaration is preserved with no
  candidate, so callers suppress their convention fallback instead of reading
  "undeclared" and resolving a conventional file the selector was only meant to
  augment (`resolve()` and `collect_all_layers()` both follow this).
- A selector declaration's own conventional payload no longer counts as a
  concrete lower resource. `_has_concrete_resource()` now treats a manifest that
  declares the command - via an exact entry or a matching selector - as
  authoritative before consulting the convention lookup, so regex declarations
  cannot chain through each other.
- `specify preset resolve <name>` resolves dotless command names as commands.
  Command manifests accept names such as `new`, so the presence of a dot can no
  longer decide the resource type: when the name resolves as a command, the
  selector attribution path is used instead of reporting it missing.
- Documented that only the pattern/state/edge limits invalidate a manifest, while
  an oversized concrete resource name is a warned non-match at match time.

Each fix has regression coverage in the existing suites
(`test_resolver.py`, `test_command_resolve.py`); all three new tests fail before
the fix and pass after it.

AI disclosure: authored with AI assistance (Hermes Agent); reviewed and verified
by a human against the test suite.
@lmtyy

lmtyy commented Oct 14, 2026

Copy link
Copy Markdown
Author

All four items in the "Previously missed" list are fixed in ce13b9b:

  • Ineligible selector collapsed to nothing — the matching declaration is now preserved with no candidate, so resolve() and collect_all_layers() suppress their convention fallback instead of resolving a conventional file the selector was only meant to augment.
  • Selector payload treated as lower-layer evidence — _has_concrete_resource() treats a manifest that declares the command (an exact entry or a matching selector) as authoritative before the convention lookup, so regex declarations can no longer chain through each other.
  • Dotless command names — preset resolve new now resolves the name as a command, since a dot can no longer decide the type, so selector attribution is reachable for accepted dotless command names.
  • Limits documentation — clarified that only the pattern/state/edge limits invalidate a manifest, while an oversized concrete resource name is a warned non-match at match time.

Each fix has a regression test in the existing suites (test_resolver.py, test_command_resolve.py); all three new tests fail before the change and pass after it. Full suite: 12,447 passed, 0 failed, surface unchanged at 18 files.

The four from your earlier list were fixed in 0a0b024, each with its own regression coverage.

One note on your second request: the review threads on this PR currently show 0 unresolved, so there is nothing left for me to resolve on my side — flagging that here as you asked. Thanks again for the careful triage.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Selector eligibility and refresh logic can still create or retain artifacts without a valid lower resource.

1 open finding
Previously missed (4)

In code that hasn't changed since last review

Medium severity Do not suppress warnings for unusable exact fallbacks

src/​specify_cli/​presets/​_manager.py:525

An unusable exact declaration currently suppresses the unmatched-selector warning. If regex:^new$ has no lower match and the same preset declares exact new with a missing payload, expansion retains the exact entry, this predicate succeeds, but no command is registered. Preserve the intentional suppression only when the exact fallback is usable.

Medium severity Require an effective replace base for selector eligibility

src/​specify_cli/​presets/​_resolver.py:630

This existence check treats any file-backed lower declaration as a concrete command, regardless of its strategy. A lower preset containing only append/prepend/wrap for a name and no replace base cannot produce that command (resolve_content returns None), yet it makes an upper replace selector eligible, allowing the selector to create a name that did not actually exist below. Determine eligibility from a lower stack that has an effective replace base, and add a negative case for a composition-only lower declaration.

Low severity Clarify hard validation behavior for preset info

docs/​reference/​presets.md:364

This says only installation fails hard, but preset info explicitly revalidates an installed unreadable manifest and exits with status 1 (command_info.py:69-81, covered by test_persisted_unsafe_manifest_is_bounded). Clarify that normal resolution warns and ignores the preset, while installation and preset info surface a hard validation error.

Low severity Update test evidence to reflect suites in this revision

tests/​specify_cli/​presets/​test_selector_engine.py:1

The PR's Tests / evidence section cites five selector test modules that are not present in this revision (test_regex_selectors.py, test_selector_safety.py, test_selector_resolution_contract.py, test_resolver_candidates.py, and test_regex_selector_diagnostics.py). Update the evidence to name the actual suites in this diff and report results from the current revision so reviewers can reproduce the claimed coverage.

🧠 Review effort: Balanced

Comment thread src/specify_cli/presets/_manager_commands.py
@mnriem

mnriem commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Please address the open finding and all four previously missed findings from the latest Copilot review. They are within the reduced scope of this PR:

  • Filter declarations with missing or unusable payloads before calculating both current selector matches and the matches persisted after reconciliation. When an existing refresh path runs, a retired selector artifact must not remain merely because an unusable same-name declaration exists.

  • Suppress an unmatched-selector warning only when the same preset provides a usable exact fallback. An exact declaration whose payload is missing does not satisfy the command.

  • Require selector eligibility to be backed by an effective lower resolution stack containing a usable replace base. A lower stack containing only append, prepend, or wrap declarations must not allow a selector to create the command name.

  • Update the documentation to explain that normal resolution warns and ignores an unsafe persisted manifest, while installation and preset info surface hard validation errors.

  • Replace the stale Tests / evidence section with the actual test modules and exact results from the current revision. The description should also acknowledge the one new focused test_selector_engine.py module rather than saying that no new test modules were added.

Please add focused regression coverage in the existing surface-owned suites for the first three behavioral findings, including both command-file and skill reconciliation where the shared behavior applies. Keep the implementation limited to correctness when the existing reconciliation paths run; this does not require adding the explicitly deferred provider-change broadcast or historical recovery machinery.

… ones

- Selector reconciliation filters unusable declarations before computing the
  current and tracked match sets. A declaration whose payload has disappeared
  materializes nothing, so treating it as current left a previously materialized
  selector command (or skill) on disk after the selector lost its provider and its
  exact fallback pointed at a missing file.
- The unmatched-selector warning is suppressed only by a usable exact fallback: a
  declared-but-missing payload registers nothing, so staying silent about the
  unmatched selector would hide that nothing was written.
- Selector eligibility requires an effective `replace` base below. A lower layer
  that only composes (append/prepend/wrap) cannot produce the command
  (`resolve_content` returns None), so it no longer makes an upper `replace`
  selector eligible and cannot let a selector create a name that never existed
  below. Undeclared conventional files, legacy extension namespace files and core
  resources still count as bases, matching the existing resolution behaviour.
- Documented that normal resolution warns and ignores an unusable selector while
  installation and `preset info` surface the hard validation error.

Regression coverage lives in the existing surface-owned suites (`test_resolver.py`,
`test_manager_commands.py`) and covers both reconciliation paths: the shared filter is
asserted for command files and for skills in the same test. All three new tests fail
before the change and pass after it.

AI disclosure: authored with AI assistance (Hermes Agent); reviewed and verified
by a human against the test suite.
@lmtyy

lmtyy commented Oct 14, 2026

Copy link
Copy Markdown
Author

All five items from your latest review are addressed in 52bcc33:

  • Reconciliation filter — declarations whose payload has disappeared are filtered before both the current and the persisted match sets are computed, so a retired selector artifact cannot survive merely because an unusable same-name declaration exists. The regression test asserts this for command files and for skills, since both reconciliation paths share the filter.
  • Warning suppression — the unmatched-selector warning is suppressed only by a usable exact fallback; a declared-but-missing payload registers nothing, so the warning is emitted.
  • Eligibility — selector eligibility now requires an effective lower resolution stack with a usable replace base. A lower stack containing only append/prepend/wrap no longer lets a selector create the command name, while undeclared conventional files, legacy extension namespace files and core resources still count as bases, so existing resolution behaviour is unchanged.
  • Documentation — now states that normal resolution warns and ignores an unsafe persisted manifest, while installation and preset info surface hard validation errors (status 1).
  • Tests / evidence — the description now names the suites actually present in this revision, acknowledges the single new focused test_selector_engine.py module, and reports the current numbers: preset suite 879 passed; full suite 12,450 passed, 0 failed.

Each of the first three behavioural findings has regression coverage in an existing suite that fails before the change and passes after it. The implementation stays limited to correctness, with no redesign of the areas already resolved in earlier rounds.

The workflow runs need approving again after this push.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Layer collection and eligibility inconsistencies can omit valid bases or admit selectors without an effective lower resource.

2 open findings
1 resolved since last review
Previously missed (2)

In code that hasn't changed since last review

Medium severity Composition creates aliases for unusable declarations

src/​specify_cli/​presets/​_manager_commands.py:310

For overlapping declarations, the composition loop can synthesize .composed/<name>.md for a declaration whose own payload is missing merely because another declaration from the same preset is the top layer. This first-wins dedup then selects that unusable declaration and registers its aliases; later reconciliation writes the usable declaration but never removes those selector-only alias files. Only compose when the top layer path equals the current declaration's payload, or retain the original payload usability when choosing the winner.

Medium severity Command surface incorrectly overrides exact template resolution

src/​specify_cli/​presets/​command_resolve.py:49

This unconditionally switches a dotless name to the command surface whenever a command exists, even if a template with the same name also resolves. Previously preset resolve foo always inspected the template, so adding selector diagnostics makes existing exact-template diagnostics silently show an unrelated command instead, contrary to the stated exact-name compatibility. Keep template precedence and fall back to command only when no template layer exists (or expose an explicit type selector).

🧠 Review effort: Balanced

Comment on lines +162 to +167
{
name
for name in concrete_candidates
if _PresetCommandMethods._command_name_has_lower_layer(
resolver, name, lower_preset_ids, lower_extension_ids
)
Comment on lines +767 to +780
for pack_id in pack_ids[start:]:
base_dir = self.presets_dir / pack_id
if self._declared_replace_resources(base_dir, name, template_type):
return True
# An undeclared conventional file is that layer's own content and
# defaults to ``replace``, so it is a valid base as well. Only a
# layer that declares the name purely for composition is not.
entry, _candidate = self._manifest_declared_template(
base_dir, name, template_type
)
if entry is None and self._conventional_resource(
base_dir, name, template_type
) is not None:
return True
…ith resolution

- Expansion now applies the same eligibility rule as resolution: a candidate name
  counts only when the lower stack has an effective `replace` base. A lower
  command declared only with append/prepend/wrap no longer appears in `preset
  info`, suppresses the unmatched warning, or is tracked as a selector match.
- Replace-base detection requires a *usable* declaration: it goes through the
  same declaration lookup as eligibility, so an ineligible middle selector is no
  longer reported as a base for an upper selector, and the conventional fallback
  is suppressed when the manifest declares the command through a selector. The
  lookups stay bounded to layers strictly below the one being evaluated, so the
  recursive check terminates.
- Pre-composition and reconciliation only write `.composed` content when the top
  layer is the declaration's own payload, so an unusable declaration can no
  longer be selected by first-wins deduplication and leave selector-only alias
  files behind.
- `preset resolve` keeps exact-template precedence for dotless names and only
  falls back to the command surface when no template layer exists, restoring the
  previous exact-template diagnostics while keeping selector attribution for
  command-only names.

Regression coverage in the existing surface-owned suites; every new case fails
before the change and passes after it.

AI disclosure: authored with AI assistance (Hermes Agent); reviewed and verified
by a human against the test suite.
@lmtyy

lmtyy commented Oct 14, 2026

Copy link
Copy Markdown
Author

All four findings from the latest review are addressed in b906e1c:

  • Expansion eligibility — expansion now applies the same rule as resolution: a candidate name counts only when the lower stack has an effective replace base, so a lower command declared only with append/prepend/wrap no longer shows up in preset info, no longer suppresses the unmatched warning, and is no longer tracked as a selector match.
  • Replace-base usability — the replace base is now determined through the same declaration lookup as eligibility, so an ineligible middle selector can no longer serve as the base for an upper selector, and the conventional fallback is suppressed when the manifest declares the command through a selector (which is exactly the trap case in the finding). The lookups are bounded to layers strictly below the one being evaluated, so the recursive check terminates: a nine-preset chain resolves in ~3 ms.
  • Composition aliases — pre-composition and reconciliation now write .composed content only when the top layer is the declaration's own payload, so an unusable declaration can no longer win the first-wins deduplication and leave selector-only alias files behind.
  • preset resolve precedence — exact-template precedence is preserved for dotless names; the command surface is used only when no template layer exists. Both directions are covered by a parametrised test (template present → template wins; absent → selector attribution still shown).

Coverage lives in the existing surface-owned suites and every new case fails before the change and passes after it. Preset suite: 890 passed; full suite: 12,461 passed / 0 failed. I also checked that the three non-target resolution scenarios (exact declaration, selector over core, composition) produce byte-identical results before and after this change.

The workflow runs need approving again after this push.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

triage-can-wait Verdict: valid and in-scope but deprioritized; held behind the evidence gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support regex selectors for preset commands, templates, and scripts

3 participants