Skip to content

Testing and coverage

Unit tests (fast)

Run the test suite in tests/ (excludes slow and integration tests by default in CI):

python -m pytest tests/ -v --tb=long

Examples-are-running (smoke) tests: Check that all gradiend.examples scripts run to completion (no metric checks). Run from the project root (examples are in the repository, not in the pip package):

pytest test_bench/examples/ -v -s

These tests are marked slow and integration; they run each example as a subprocess and assert exit code 0. On failure, full logs are written to test_bench/results/last_failure_<module>.log. See test_bench/README.md for smoke-only options (e.g. -m integration) and details.

For an offline GPU job, first prepare every dataset, model, spaCy package, and the complete raw Wikipedia dataset snapshot using the same shared cache as the job:

python scripts/prefetch_example_assets.py --cache-dir /shared/drechsel/hf-cache
python scripts/prefetch_example_assets.py --cache-dir /shared/drechsel/hf-cache --verify-only

Set HF_HOME=/shared/drechsel/hf-cache in the offline test job as well. The explicit path prevents models from being prefetched into a login node's default ~/.cache/huggingface while the job reads the shared cache.

scripts/prefetch_hf_datasets.py is only the legacy dataset/export helper; it does not fetch models such as dbmdz/german-gpt2 and is not sufficient for the offline example suite.

The main test suite exposes the same smoke runner through tests/test_examples_smoke_integration.py. Therefore the complete heavyweight selection includes both the slow/integration tests and all configured example runs:

pytest tests/ -v -s -m "slow or integration"

Do not run this command as a routine unit-test check: it loads real models and executes training examples. The bridge delegates to the test-bench implementation, so the example list and failure logging remain defined in one place.

Exclude slow/integration tests explicitly:

python -m pytest tests/ -v -m "not slow and not integration"

Run a single test file or test:

pytest tests/test_data_generation.py -v
pytest tests/test_img_format.py::TestImgFormatTrainerForwarding::test_plot_encoder_distributions_receives_img_format_from_trainer -v

Test coverage

Install dev dependencies (includes pytest-cov):

pip install -r requirements-dev.txt
# or: pip install -e ".[dev]"

Run tests with coverage report (terminal + optional HTML):

# Coverage for package code only (exclude tests and examples)
pytest tests/ -v --cov=gradiend --cov-report=term-missing --cov-report=html -m "not slow and not integration"
  • Terminal: --cov-report=term-missing shows missed lines per file.
  • HTML report: --cov-report=html writes htmlcov/index.html; open it in a browser to see line-by-line coverage and find untested code.

Useful options:

  • --cov-fail-under=60 — fail the run if total coverage is below 60%.
  • Omit -m "not slow and not integration" to include all tests (slower).

Test bench (integration, GPU)

The test bench runs real training and verification; it is slower and typically needs a GPU. You do not need reference scores: verified tests use built-in thresholds (e.g. correlation ≥ 0.35); run the bench the same way every time. See test_bench/README.md for first-time run and:

  • Running locally: python test_bench/run_bench.py
  • Docker (recommended): Build and run the test-bench image; it runs the full test bench including verified runs that assert scores (e.g. correlation ≥ threshold) and required files. Use --build-arg BASE_IMAGE=continuumio/miniconda3:py311 for a different Python.
  • Unit tests in Docker (no GPU): docker build -f test_bench/Dockerfile.unit-tests --build-arg PYTHON_VERSION=3.11 -t gradiend-unit-tests . then docker run --rm gradiend-unit-tests.

The test bench is not run in standard CI; run it manually or in a nightly job.