Skip to content

Commit 1a108b5

Browse files
codexByron
authored andcommitted
test(perf): benchmark repository journeys across CLI and GixPython
Full-suite timings mix repository operations with repeated fixture setup and coverage. Add a benchmark-only `pyperf` harness against a pinned existing GitPython checkout, with one warm `Repo` per worker and fresh high-level wrappers per invocation. Measure the complete read-only journey and eight named public-API operations, including a patch operation that uses fallback. Separately time direct `Repo` construction and discovery from `git/objects`, creating and closing a repository per invocation. Keep those lifecycle costs outside the already-open journey so future native-handle reuse has a visible effect on operation timings. `MEASUREMENTS` makes additions join individual timings, the journey and result parity checks. Retain environment/revision metadata, result digests and one invocation's native/fallback decisions in raw `pyperf` results. The comparison reports both backend means and standard deviations and rejects mismatched results, installations or workload metadata. Document reproducible local setup without downloading another interpreter or modifying the measured repository. All eleven workload paths passed GixPython first and CLI second on the fixed fixture, with matching result digests. Four comparison regression cases passed with both installations. Ruff lint/format and `git diff --check` passed. Install GixPython in both benchmark environments for `gix.testtools`, and select the backend with `GIT_PYTHON_BACKEND`. Preserve that selection across `pyperf` worker environment filtering; record the actual GixPython version in both result files. Move the fixed repository's preparation into a cached read-only Bash fixture, and ignore generated fixture data and archives. Ruff lint/format, ShellCheck and whitespace checks passed here. On the completed stack, both backend worker smoke runs completed, all 14 result digests matched and existing Gix CLI-process budgets passed. Single-value samples do not establish a performance change.
1 parent ce53306 commit 1a108b5

7 files changed

Lines changed: 352 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,8 @@ __pycache__/
66
.cache/
77
.mypy_cache/
88
.pytest_cache/
9+
tests/fixtures/generated-do-not-edit/
10+
tests/fixtures/generated-archives/*.tar
911

1012
# Transient editor files
1113
*.swp

‎test/performance/README.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Backend benchmark
2+
3+
`bench_repository.py` uses `pyperf` to measure a complete read-only journey and
4+
eight named operations against an existing repository, plus separate direct
5+
opening and nested-directory discovery measurements. It keeps one `Repo`
6+
instance open per worker, but creates fresh commit/tree/index wrappers per
7+
invocation so their cached properties cannot turn the measurement into a no-op.
8+
The CLI's persistent `cat-file` processes and the filesystem cache are warmed.
9+
Repository creation, imports, preflight and result hashing are outside timing.
10+
Native repository reopening and compatibility checks remain inside timing.
11+
This exposes any future benefit of retaining a native Gix repository handle.
12+
`open_repository` and `discover_repository` instead create and close a new
13+
`Repo` in each timed invocation; they do not join the already-open journey.
14+
15+
Use two installations with **the same existing CPython 3.11+ interpreter**.
16+
Both install GixPython for fixture tooling; `GIT_PYTHON_BACKEND` selects the
17+
backend in the manager and its workers. Install the benchmark dependencies in
18+
both environments:
19+
20+
```sh
21+
uv pip install --python .tox/gix/bin/python -r test/performance/benchmark-requirements.txt
22+
uv pip install --python .venv/bin/python -r test/performance/benchmark-requirements.txt
23+
```
24+
25+
Prepare the same fixed repository used by CI (GitPython 3.1.45) through
26+
`gix.testtools`, outside the measured operations. Its Bash script lives in
27+
`tests/fixtures/benchmark.sh`; no remote access is needed if that revision
28+
already exists locally:
29+
30+
```sh
31+
BENCH_REPO=$(.venv/bin/python -c 'from pathlib import Path; from gix import testtools; print((testtools.scripted_fixture_read_only_with_args("benchmark.sh", [str(Path.cwd())]) / "repo").resolve())')
32+
```
33+
34+
Run from the source root, sequentially to avoid competing for CPU and disk:
35+
36+
```sh
37+
GIT_PYTHON_BACKEND=gix .tox/gix/bin/python -m test.performance.bench_repository \
38+
--repo "$BENCH_REPO" --expect-backend gix -o /tmp/gix.json
39+
GIT_PYTHON_BACKEND=cli .venv/bin/python -m test.performance.bench_repository \
40+
--repo "$BENCH_REPO" --expect-backend cli -o /tmp/cli.json
41+
.tox/gix/bin/python -m test.performance.compare_backends /tmp/cli.json /tmp/gix.json
42+
.tox/gix/bin/python -m pyperf compare_to /tmp/cli.json /tmp/gix.json --table --table-format md
43+
```
44+
45+
The default uses six worker processes, three measured values per process and
46+
one warmup, with calibrated loop counts and a minimum value duration of 100 ms.
47+
Pass standard `pyperf` options such as `--rigorous` for more samples or
48+
`--debug-single-value` for a smoke check. Raw JSON retains sample distributions,
49+
environment details, fixture/source revisions, result digests, and one untimed
50+
invocation's native/fallback decisions. The comparison rejects differing
51+
results, revisions, Python/Git versions and storage formats. The harness also
52+
rejects the wrong backend rather than silently comparing CLI against itself.
53+
54+
The journey inventories refs, reads metadata for 25 first-parent commits,
55+
browses the root tree and README blob, reads index entries, counts reachable
56+
commits and checks ancestry, inspects the latest diff/statistics and patch, and
57+
checks dirty/untracked/ignored paths. An alternative `--repo` must have `HEAD`,
58+
at least one parent, a root `README.md` and a `git/objects` directory for the
59+
nested discovery measurement. The harness never prepares or
60+
mutates that worktree, and ignores ambient Git environment/config overrides.
61+
62+
Add a function returning JSON-compatible results to `MEASUREMENTS` in
63+
`bench_repository.py`. It automatically becomes a separate benchmark and part
64+
of the journey and parity checks. Include supported and fallback operations;
65+
the current patch measurement intentionally exercises fallback. Backend
66+
decisions are adapter events, not complete subprocess counts.
67+
68+
Timing is observational: shared CI runners and local activity introduce noise.
69+
The CI job publishes all means, standard deviations and ratios, retains raw
70+
artifacts, and uses `pyperf compare_to` for significance reporting. It fails on
71+
execution/parity errors, without a noisy speed threshold. Compare like-for-like
72+
fixture and source revisions when tracking performance across backend changes.
Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
"""Read-only GitPython journeys, measured with pyperf against an existing checkout.
2+
3+
Run as a module from the source root so pyperf workers import this checkout:
4+
python -m test.performance.bench_repository --repo /path/to/repo --expect-backend gix -o gix.json
5+
"""
6+
7+
from hashlib import sha256
8+
from importlib.metadata import version
9+
import json
10+
import os
11+
from pathlib import Path
12+
import subprocess
13+
14+
import pyperf
15+
16+
from git import Repo, _backend
17+
18+
19+
def inventory(repo):
20+
return {
21+
"head": repo.head.commit.hexsha if repo.head.is_detached else repo.head.reference.path,
22+
"refs": sorted(ref.path for ref in repo.references),
23+
}
24+
25+
26+
def history(repo):
27+
return [
28+
[commit.hexsha, str(commit.author), commit.authored_date, commit.message]
29+
for commit in repo.iter_commits("HEAD", max_count=25, first_parent=True)
30+
]
31+
32+
33+
def browse(repo):
34+
tree = repo.commit("HEAD").tree
35+
blob = tree / "README.md"
36+
content = blob.data_stream.read()
37+
return {
38+
"entries": [[entry.path, entry.mode, entry.hexsha] for entry in tree],
39+
"readme": sha256(content).hexdigest(),
40+
}
41+
42+
43+
def index(repo):
44+
return sorted([path, stage, entry.mode, entry.hexsha] for (path, stage), entry in repo.index.entries.items())
45+
46+
47+
def graph(repo):
48+
tip, parent = repo.commit("HEAD"), repo.commit("HEAD~1")
49+
return {
50+
"count": tip.count(),
51+
"merge_base": [commit.hexsha for commit in repo.merge_base(parent, tip)],
52+
"is_ancestor": repo.is_ancestor(parent, tip),
53+
}
54+
55+
56+
def changes(repo):
57+
tip = repo.commit("HEAD")
58+
return {
59+
"diff": [[diff.change_type, diff.a_path, diff.b_path] for diff in tip.parents[0].diff(tip)],
60+
"stats": tip.stats.total,
61+
}
62+
63+
64+
def patch(repo):
65+
tip = repo.commit("HEAD")
66+
return [sha256(diff.diff).hexdigest() for diff in tip.parents[0].diff(tip, create_patch=True)]
67+
68+
69+
def worktree(repo):
70+
return {
71+
"dirty": repo.is_dirty(untracked_files=True),
72+
"untracked": sorted(repo.untracked_files),
73+
"ignored": repo.ignored(".benchmark-cache/probe"),
74+
}
75+
76+
77+
# Add a named public-API operation here: it is measured separately and joins the
78+
# complete journey automatically. Return JSON-compatible data for parity checks.
79+
MEASUREMENTS = {
80+
"inventory": inventory,
81+
"history_25": history,
82+
"browse_tree_and_blob": browse,
83+
"read_index": index,
84+
"revision_graph": graph,
85+
"diff_and_stats": changes,
86+
"patch_diff": patch,
87+
"worktree_status": worktree,
88+
}
89+
90+
91+
def journey(repo):
92+
return {name: operation(repo) for name, operation in MEASUREMENTS.items()}
93+
94+
95+
def open_repository(path):
96+
with Repo(path) as opened:
97+
return [opened.bare, opened.object_format, opened.ref_format]
98+
99+
100+
def discover_repository(path):
101+
with Repo(path, search_parent_directories=True) as opened:
102+
return [opened.bare, opened.object_format, opened.ref_format]
103+
104+
105+
def worker_arguments(cmd, args):
106+
cmd.extend(["--repo", str(args.repo), "--expect-backend", args.expect_backend])
107+
108+
109+
def main():
110+
runner = pyperf.Runner(
111+
processes=6,
112+
values=3,
113+
program_args=("-m", "test.performance.bench_repository"),
114+
add_cmdline_args=worker_arguments,
115+
)
116+
runner.argparser.add_argument("--repo", type=Path, required=True, help="Existing worktree; never modified")
117+
runner.argparser.add_argument("--expect-backend", choices=("cli", "gix"), required=True)
118+
args = runner.parse_args()
119+
args.repo = args.repo.resolve()
120+
if _backend.name != args.expect_backend:
121+
raise RuntimeError(f"Expected {args.expect_backend}, imported {_backend.name}; set GIT_PYTHON_BACKEND")
122+
# The manager and every worker use the same isolated Git environment.
123+
for key in list(os.environ):
124+
if key.startswith("GIT_"):
125+
del os.environ[key]
126+
os.environ.update(
127+
GIT_CONFIG_NOSYSTEM="1",
128+
GIT_CONFIG_GLOBAL=os.devnull,
129+
GIT_OPTIONAL_LOCKS="0",
130+
GIT_PYTHON_BACKEND=args.expect_backend,
131+
)
132+
args.inherit_environ = [*(args.inherit_environ or []), "GIT_PYTHON_BACKEND"]
133+
134+
def git(*arguments, cwd=args.repo):
135+
return subprocess.check_output(["git", *arguments], cwd=cwd, text=True).strip()
136+
137+
runner.metadata.update(
138+
backend=_backend.name,
139+
gixpython=version("GixPython"),
140+
git_version=git("version"),
141+
gitpython_revision=git("rev-parse", "HEAD", cwd=Path.cwd()),
142+
fixture_revision=git("rev-parse", "HEAD"),
143+
fixture_tree=git("rev-parse", "HEAD^{tree}"),
144+
fixture_status=git("status", "--porcelain") or "clean",
145+
fixture_object_format=git("rev-parse", "--show-object-format"),
146+
fixture_ref_format=git("rev-parse", "--show-ref-format"),
147+
)
148+
with Repo(args.repo) as repo:
149+
measurements = [("journey", journey, repo)]
150+
measurements.extend((name, operation, repo) for name, operation in MEASUREMENTS.items())
151+
measurements.extend(
152+
[
153+
("open_repository", open_repository, args.repo),
154+
("discover_repository", discover_repository, args.repo / "git" / "objects"),
155+
]
156+
)
157+
for name, operation, target in measurements:
158+
# Untimed preflight validates parity and records native/fallback
159+
# decisions for ONE invocation, rather than calibrated loop counts.
160+
before = _backend.statistics()
161+
result = operation(target)
162+
decisions = [
163+
{"method": method, "outcome": outcome, "count": count - before.get((method, outcome), 0)}
164+
for (method, outcome), count in sorted(_backend.statistics().items())
165+
if count > before.get((method, outcome), 0)
166+
]
167+
metadata = {
168+
"result_digest": sha256(json.dumps(result, sort_keys=True).encode()).hexdigest(),
169+
"backend_decisions": json.dumps(decisions, sort_keys=True),
170+
}
171+
runner.bench_func(name, operation, target, metadata=metadata)
172+
173+
174+
if __name__ == "__main__":
175+
main()
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
pyperf==2.10.0
2+
-r ../../gix-requirements.txt
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
"""Validate benchmark parity and print every measurement, including slowdowns."""
2+
3+
import json
4+
import sys
5+
6+
import pyperf
7+
8+
9+
def compare(cli_path, gix_path):
10+
cli, gix = pyperf.BenchmarkSuite.load(str(cli_path)), pyperf.BenchmarkSuite.load(str(gix_path))
11+
if cli.get_benchmark_names() != gix.get_benchmark_names():
12+
raise ValueError("The backends measured different journeys")
13+
rows = []
14+
for name in cli.get_benchmark_names():
15+
left, right = cli.get_benchmark(name), gix.get_benchmark(name)
16+
lm, rm = left.get_metadata(), right.get_metadata()
17+
if (lm["backend"], rm["backend"]) != ("cli", "gix"):
18+
raise ValueError("Expected CLI results followed by GixPython results")
19+
for key in (
20+
"gitpython_revision",
21+
"fixture_revision",
22+
"fixture_tree",
23+
"fixture_status",
24+
"fixture_object_format",
25+
"fixture_ref_format",
26+
"git_version",
27+
"python_version",
28+
"result_digest",
29+
):
30+
if lm[key] != rm[key]:
31+
raise ValueError(f"{name}: backends differ in {key}: {lm[key]!r} != {rm[key]!r}")
32+
decisions = json.loads(rm["backend_decisions"])
33+
native = sum(item["count"] for item in decisions if item["outcome"] == "native")
34+
fallback = sum(item["count"] for item in decisions if item["outcome"].startswith("CLI:"))
35+
rows.append(
36+
f"| {name} | {left.mean() * 1000:.2f} ± {left.stdev() * 1000:.2f} "
37+
f"| {right.mean() * 1000:.2f} ± {right.stdev() * 1000:.2f} "
38+
f"| {left.mean() / right.mean():.2f}× | {native} / {fallback} |"
39+
)
40+
print("| Measurement | CLI mean ± stdev (ms) | GixPython mean ± stdev (ms) | CLI / Gix | Native / fallback |")
41+
print("| --- | ---: | ---: | ---: | ---: |")
42+
print("\n".join(rows))
43+
print("\nRatios above 1 favor GixPython. Decisions are per invocation, not subprocess counts.")
44+
print("\nWarm-cache, read-only measurements; checkout, imports and preflight are outside timing.")
45+
print("The journey reuses one open Repo; opening and nested discovery are measured separately.")
46+
47+
48+
if __name__ == "__main__":
49+
compare(*sys.argv[1:])
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
"""The benchmark comparison must reject incompatible workloads and results."""
2+
3+
import pytest
4+
5+
pyperf = pytest.importorskip("pyperf")
6+
7+
8+
@pytest.mark.parametrize("changed", [None, "result_digest", "fixture_revision", "backend"])
9+
def test_comparison_requires_parity(tmp_path, capsys, changed):
10+
from test.performance.compare_backends import compare
11+
12+
metadata = {
13+
"gitpython_revision": "source",
14+
"fixture_revision": "fixture",
15+
"fixture_tree": "tree",
16+
"fixture_status": "clean",
17+
"fixture_object_format": "sha1",
18+
"fixture_ref_format": "files",
19+
"git_version": "git version 2.52.0",
20+
"python_version": "3.12.14",
21+
"result_digest": "equal",
22+
"backend_decisions": "[]",
23+
"name": "journey",
24+
}
25+
paths = [tmp_path / "cli.json", tmp_path / "gix.json"]
26+
for backend, path in zip(("cli", "gix"), paths):
27+
values = {**metadata, "backend": backend}
28+
if backend == "gix" and changed:
29+
values[changed] = "different"
30+
run = pyperf.Run([0.1, 0.2], metadata=values, collect_metadata=False)
31+
pyperf.BenchmarkSuite([pyperf.Benchmark([run])]).dump(str(path))
32+
if changed:
33+
with pytest.raises(ValueError, match="differ|Expected"):
34+
compare(*paths)
35+
else:
36+
compare(*paths)
37+
assert "journey | 150.00" in capsys.readouterr().out

‎tests/fixtures/benchmark.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env bash
2+
set -eu
3+
4+
git init --object-format=sha1 --ref-format=files repo
5+
git -C repo fetch --no-tags "$1" 6ba2c0a2f9ee7feffd7e079621c4845820180c9a
6+
git -C repo checkout -b benchmark FETCH_HEAD
7+
git -C repo config core.excludesFile /dev/null
8+
git -C repo config core.attributesFile /dev/null
9+
git -C repo config core.fsmonitor false
10+
git -C repo config core.untrackedCache false
11+
git -C repo config gc.auto 0
12+
git -C repo config maintenance.auto false
13+
mkdir repo/.benchmark-cache
14+
touch repo/.benchmark-cache/probe repo/benchmark-untracked.txt
15+
echo '.benchmark-cache/' >> repo/.git/info/exclude

0 commit comments

Comments
 (0)