Skip to content

Commit 6d3fec3

Browse files
codexByron
authored andcommitted
ci: compare CLI and GixPython repository benchmarks
Add a separate `Backend benchmark` job using one CPython 3.12 interpreter and two installations with released GixPython 0.2.0. Explicitly select the backend for each manager and its workers. Prepare the fixed SHA-1/files GitPython 3.1.45 checkout through a read-only `gix.testtools` Bash fixture before timing; its initial state includes one branch, an untracked file and an ignored directory. Prepare Rust and cache Cargo outputs for source builds on Linux. Keep Git configuration and optional locks from changing the measured workload. Run the calibrated `pyperf` suite with GixPython first, then CLI on the same runner. Include separate direct repository opening and nested discovery while keeping the operation journey on a retained `Repo`. Validate result digests and workload metadata, publish all means and standard deviations plus `pyperf` significance reporting in the job summary, and retain raw JSON and comparison artifacts even on failure. Timings are observational on shared runners; execution and parity errors fail the job, without a noisy performance threshold. New `MEASUREMENTS` entries automatically participate in this job. The completed stack passed all 14 worker smoke measurements on both backends with matching result digests and existing CLI-process budgets. The workflow passes actionlint and whitespace checks. Hosted execution remains unverified; single-value local samples establish no performance conclusion.
1 parent 1a108b5 commit 6d3fec3

1 file changed

Lines changed: 80 additions & 0 deletions

File tree

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
name: Backend benchmark
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
workflow_dispatch:
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
compare-backends:
14+
runs-on: ubuntu-latest
15+
timeout-minutes: 30
16+
steps:
17+
- uses: actions/checkout@v7
18+
with:
19+
fetch-depth: 0
20+
persist-credentials: false
21+
- uses: actions/setup-python@v7
22+
with:
23+
python-version: "3.12"
24+
- name: Ensure Git 2.52 or newer
25+
run: |
26+
if ! dpkg --compare-versions "$(git version | awk '{print $3}')" ge 2.52; then
27+
sudo add-apt-repository --yes ppa:git-core/ppa
28+
sudo apt-get update
29+
sudo apt-get install --yes git
30+
fi
31+
- name: Prepare Rust for GixPython
32+
run: |
33+
rustup update stable
34+
rustup default stable
35+
echo "CARGO_TARGET_DIR=$RUNNER_TEMP/gix-target" >> "$GITHUB_ENV"
36+
- name: Cache GixPython builds
37+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
38+
with:
39+
path: |
40+
~/.cargo/registry
41+
~/.cargo/git
42+
${{ runner.temp }}/gix-target
43+
~/.cache/pip
44+
key: benchmark-gix-${{ runner.os }}-${{ hashFiles('gix-requirements.txt') }}
45+
- name: Install separate backends with the same interpreter
46+
run: |
47+
python -m venv .bench-cli
48+
python -m venv .bench-gix
49+
.bench-cli/bin/python -m pip install . -r test/performance/benchmark-requirements.txt
50+
.bench-gix/bin/python -m pip install '.[gix]' -r test/performance/benchmark-requirements.txt
51+
- name: Prepare the shared read-only benchmark fixture
52+
run: |
53+
.bench-cli/bin/python -c 'from pathlib import Path; from gix import testtools; print("BENCH_REPO=" + str((testtools.scripted_fixture_read_only_with_args("benchmark.sh", [str(Path.cwd())]) / "repo").resolve()))' >> "$GITHUB_ENV"
54+
mkdir benchmark-results
55+
- name: Measure GixPython operations, opening and discovery first
56+
env:
57+
GIT_PYTHON_BACKEND: gix
58+
run: |
59+
.bench-gix/bin/python -m test.performance.bench_repository \
60+
--repo "$BENCH_REPO" --expect-backend gix -o benchmark-results/gix.json
61+
- name: Measure CLI operations, opening and discovery
62+
env:
63+
GIT_PYTHON_BACKEND: cli
64+
run: |
65+
.bench-cli/bin/python -m test.performance.bench_repository \
66+
--repo "$BENCH_REPO" --expect-backend cli -o benchmark-results/cli.json
67+
- name: Check parity and publish comparison
68+
run: |
69+
.bench-gix/bin/python -m test.performance.compare_backends \
70+
benchmark-results/cli.json benchmark-results/gix.json > benchmark-results/comparison.md
71+
cat benchmark-results/comparison.md >> "$GITHUB_STEP_SUMMARY"
72+
.bench-gix/bin/python -m pyperf compare_to \
73+
benchmark-results/cli.json benchmark-results/gix.json --table --table-format md \
74+
>> "$GITHUB_STEP_SUMMARY"
75+
- name: Retain raw timings and comparison
76+
if: always()
77+
uses: actions/upload-artifact@v7
78+
with:
79+
name: backend-benchmark
80+
path: benchmark-results/

0 commit comments

Comments
 (0)