GitHub Actions CI/CD (10/10): Going Professional — Split Jobs, Caching, and Nightly Builds

Summary: Optimize your CI/CD pipeline for speed and cost. Split lint and test into independent jobs, cache pip dependencies so installs take seconds instead of minutes, and add a nightly scheduled run for heavy integration tests. This is the final part of the series.

Key	Value
Package name	helloci
Working directory	~/projects/helloci
CI workflow	.github/workflows/ci.yml
Nightly workflow	.github/workflows/nightly.yml
Cache key	pip dependencies
Schedule	nightly at 03:00 UTC

0. Prerequisites

• The complete helloci project with CI, CD, branch protection, and artifacts from Parts 1–9
• All workflows passing on the main branch

---

1. Why Optimize

Your current CI workflow works, but as your project grows, you will notice:

• Slow feedback — waiting 3-5 minutes for a lint check that takes 2 seconds
• Wasted compute — installing the same pip packages on every run
• Blocked PRs — integration tests holding up merges for code that only changed a docstring

Professional CI pipelines solve these problems by splitting work into independent stages, caching expensive operations, and moving slow tests to a separate schedule.

---

2. Design the Pipeline

Here is the target pipeline design.

Job	Trigger	Depends on	Purpose
lint	Every push/PR	nothing	Fast static checks (~15s)
unit-tests	Every push/PR	lint	Core correctness (~30s)
integration-tests	Every push/PR	lint	Database tests (~60s)
nightly-integration	Schedule (03:00 UTC)	nothing	Full integration suite

Key decisions:

• Lint runs first — if formatting is wrong, there is no point running tests. Unit tests and integration tests both depend on lint passing.
• Unit and integration tests run in parallel — after lint passes, both start simultaneously.
• Nightly runs cover slow, expensive tests that should not block every PR.

---

3. Add Dependency Caching

Caching stores your pip downloads between runs. The first run downloads everything. Subsequent runs with the same dependencies skip the download entirely.

The actions/setup-python action has built-in caching support. Update every job that installs Python to include the cache parameter.

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pipCode language: YAML (yaml)

That single line — cache: pip — tells the action to cache pip’s download directory. The cache key is automatically derived from your pyproject.toml (or requirements.txt if present).

Run	Install time
First run (cold cache)	~15-30 seconds
Subsequent runs (warm cache)	~3-5 seconds

Note: The cache is scoped to the branch. A PR branch inherits the cache from its base branch (main), so the first run on a new PR is usually warm.

---

4. Add Job Dependencies

Use the needs keyword to create a dependency chain. Jobs listed in needs must pass before the current job starts.

  unit-tests:
    needs: lint
    runs-on: ubuntu-latestCode language: YAML (yaml)

  integration-tests:
    needs: lint
    runs-on: ubuntu-latestCode language: YAML (yaml)

Both unit-tests and integration-tests need lint. Since they do not need each other, they run in parallel after lint passes.

If lint fails, neither test job starts. This saves compute and gives faster feedback — you see “lint failed” in under 30 seconds instead of waiting for all jobs to finish.

---

5. Write the Optimized CI Workflow

Replace .github/workflows/ci.yml with the complete optimized version.

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies
        run: pip install -e ".[test]"

      - name: Lint
        run: ruff check .

      - name: Check formatting
        run: ruff format . --check

  unit-tests:
    needs: lint
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
      fail-fast: false

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          cache: pip

      - name: Install dependencies
        run: pip install -e ".[test]"

      - name: Run unit tests
        run: pytest -v -m "not integration" --junitxml=reports/junit.xml

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: unit-test-results-${{ matrix.python-version }}
          path: reports/junit.xml
          retention-days: 14

  integration-tests:
    needs: lint
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: helloci_user
          POSTGRES_PASSWORD: helloci_pass
          POSTGRES_DB: helloci_db
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U helloci_user -d helloci_db"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies
        run: pip install -e ".[test]"

      - name: Run integration tests
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: pytest -v -m integration --junitxml=reports/junit.xml

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: integration-test-results
          path: reports/junit.xml
          retention-days: 14Code language: YAML (yaml)

This is the final version of your CI workflow. It incorporates everything from Parts 4–8: matrix testing, service containers, secrets, artifacts, caching, and job dependencies.

---

6. Create the Nightly Workflow

Create .github/workflows/nightly.yml for scheduled heavy tests.

name: Nightly

on:
  schedule:
    - cron: "0 3 * * *"
  workflow_dispatch:

jobs:
  full-integration:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: helloci_user
          POSTGRES_PASSWORD: helloci_pass
          POSTGRES_DB: helloci_db
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U helloci_user -d helloci_db"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5

    steps:
      - name: Check out code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - name: Install dependencies
        run: pip install -e ".[test]"

      - name: Run all tests
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
        run: pytest -v --junitxml=reports/junit.xml

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: nightly-test-results
          path: reports/junit.xml
          retention-days: 7Code language: YAML (yaml)

---

7. Understand the Nightly Workflow

Schedule trigger:

on:
  schedule:
    - cron: "0 3 * * *"
  workflow_dispatch:Code language: YAML (yaml)

Field	Meaning
cron: "0 3 * * *"	Run at 03:00 UTC every day
workflow_dispatch	Allow manual triggers from the Actions tab

The cron syntax is minute hour day month weekday. Common patterns:

Cron	Schedule
0 3 * * *	Daily at 03:00 UTC
0 3 * * 1	Every Monday at 03:00 UTC
0 3 * * 1-5	Weekdays at 03:00 UTC
0 */6 * * *	Every 6 hours

The workflow_dispatch trigger adds a Run workflow button on the Actions tab. Use it to trigger the nightly workflow manually when debugging.

Note: GitHub does not guarantee exact cron timing. Scheduled workflows may be delayed by several minutes during periods of high load.

What the nightly runs:

The nightly job runs all tests — unit and integration — without the -m filter. This catches issues that might slip through when unit and integration tests are run separately on PRs.

The retention is set to 7 days because nightly reports accumulate fast. You usually only care about the most recent run.

---

8. The Complete Project Structure

Your final project structure.

~/projects/helloci/
├── .github/
│   └── workflows/
│       ├── ci.yml
│       ├── nightly.yml
│       └── release.yml
├── .gitignore
├── check.sh
├── pyproject.toml
├── src/
│   └── helloci/
│       ├── __init__.py
│       ├── cli.py
│       └── db.py
├── tests/
│   ├── __init__.py
│   ├── test_cli.py
│   ├── test_db.py
│   └── test_greet.py
└── reports/          (generated, gitignored)Code language: Shell Session (shell)

---

9. What Comes Next: Reusable Workflows

As your organization grows, you will have multiple repositories running similar CI pipelines. Instead of copying workflow YAML into every repo, you can create reusable workflows.

A reusable workflow lives in one repository and is called from others.

jobs:
  lint:
    uses: your-org/.github/.github/workflows/python-lint.yml@main
    with:
      python-version: "3.12"Code language: YAML (yaml)

The calling workflow passes inputs (with), and the reusable workflow runs the steps. This is the same concept as a function call — define once, use everywhere.

Reusable workflows are beyond the scope of this series, but they are the natural next step once you have a pipeline pattern you want to standardize.

Tip: Start with copy-paste across a few repositories. When you find yourself maintaining the same YAML in three or more repos, extract it into a reusable workflow.

---

10. The Full Pipeline at a Glance

Here is everything you built across all ten parts.

Part	What you added	CI/CD concept
1	Installable Python package	Build/install basics
2	pytest unit tests	First quality gate
3	ruff lint and format	Static analysis gate
4	GitHub Actions CI workflow	Automated checks on push/PR
5	Matrix testing (3.10, 3.11, 3.12)	Multi-version verification
6	Postgres integration tests	Service containers, real dependencies
7	Branch protection and secrets	Enforced merge gates, credential management
8	Artifact uploads	Evidence and reports from CI runs
9	Tag-triggered release to PyPI	Continuous Deployment
10	Caching, job deps, nightly runs	Pipeline optimization

---

Summary

You optimized the CI pipeline by splitting jobs, adding dependency caching, and creating a nightly scheduled workflow.

• Job dependencies (needs: lint) — fast feedback when lint fails, parallel test execution when it passes
• Pip caching (cache: pip) — cuts install time from 30 seconds to under 5
• Nightly schedule (cron: "0 3 * * *") — runs the full test suite daily without blocking PRs
• workflow_dispatch — manual trigger for debugging nightly runs
• Reusable workflows — the next step for scaling pipelines across repositories

This completes the series. You started with a single Python function and ended with a professional CI/CD pipeline: lint, unit tests, integration tests, branch protection, artifacts, automated releases, caching, and scheduled runs. Every concept builds on the previous one. Every part is something you will use in real projects.