| |

Jenkins CI/CD (7/11): Artifacts and Debugging Failed Builds

ByRex Bytes

Summary: You add artifact archiving and Docker log capture to your Jenkinsfile so that every build — pass or fail — produces downloadable evidence. You create a pytest plugin that generates a markdown integration report, wire archiveArtifacts into the pipeline’s post block, and practice a post-failure forensics workflow using real artifacts from a deliberately broken build.

Example Values Used in This Tutorial

KeyValue
Docker logs outputresults/docker-logs.txt
Integration reportresults/integration-report.md
Archive patternresults/**
Jenkins steparchiveArtifacts artifacts: 'results/**'
Previous partsParts 1-6 completed

0. Prerequisites

  • A working Jenkins controller at http://localhost:8080 (Part 1).
  • The helloci Python package with unit tests and integration tests committed to Git (Parts 2-6).
  • A Jenkinsfile with lint, unit test, and integration test stages using Docker Compose (Parts 3-6).
  • Docker and Docker Compose installed on the Jenkins agent.
  • Familiarity with the post { always { } } block from Part 4.

Note: Your pipeline from Part 6 should have a working integration test stage that spins up Postgres via Docker Compose, runs test_integration.py, and tears down the container. If that is not green, go back and fix it before continuing.

1. Why Evidence Capture Is a First-Class CI Feature

A passing build needs no explanation. A failing build needs a crime scene.

When a CI build fails, your first instinct is to reproduce the failure locally. Sometimes that works. Often it does not. The CI environment has different state — different container images, different network timing, different filesystem permissions. The failure that happened on Jenkins may not happen on your laptop.

This is why artifact archiving matters. Instead of trying to recreate the exact conditions, you capture them at the moment of failure:

  • Docker container logs show what the database or service actually did during the test run.
  • JUnit XML files tell you exactly which test failed and with what assertion error.
  • Integration reports summarize the run in human-readable form — what was tested, what passed, what failed, and how long each test took.

The principle is simple: if a build fails, the build itself should produce everything you need to diagnose the failure. You should never need SSH access to the Jenkins agent to figure out what went wrong.

2. Create the Integration Report Plugin

Pytest’s conftest.py hook system lets you generate custom reports without installing third-party plugins. You will create a conftest that writes a markdown summary to results/integration-report.md after the test session finishes.

Create the file tests/conftest.py (or add to it if it already exists):

import os
import time
from datetime import datetime, timezone


def pytest_configure(config):
    config._report_results = []
    config._report_start = time.time()


def pytest_runtest_logreport(report):
    if report.when == "call":
        report.config._report_results.append(
            {
                "name": report.nodeid,
                "outcome": report.outcome,
                "duration": round(report.duration, 3),
            }
        )


def pytest_sessionfinish(session, exitstatus):
    results = session.config._report_results
    elapsed = round(time.time() - session.config._report_start, 2)

    passed = sum(1 for r in results if r["outcome"] == "passed")
    failed = sum(1 for r in results if r["outcome"] == "failed")
    total = len(results)

    os.makedirs("results", exist_ok=True)
    timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC")

    with open("results/integration-report.md", "w") as f:
        f.write(f"# Integration Test Report\n\n")
        f.write(f"**Run at:** {timestamp}\n\n")
        f.write(f"**Total:** {total} | ")
        f.write(f"**Passed:** {passed} | ")
        f.write(f"**Failed:** {failed} | ")
        f.write(f"**Duration:** {elapsed}s\n\n")
        f.write(f"**Exit status:** {exitstatus}\n\n")
        f.write("| Test | Result | Duration |\n")
        f.write("|---|---|---|\n")
        for r in results:
            icon = "PASS" if r["outcome"] == "passed" else "FAIL"
            f.write(f"| {r['name']} | {icon} | {r['duration']}s |\n")Code language: Python (python)

This plugin hooks into three pytest events:

  • pytest_configure — initializes an empty results list and records the start time.
  • pytest_runtest_logreport — captures each test’s name, outcome, and duration after it runs.
  • pytest_sessionfinish — writes the markdown summary file when the session ends.

Run it locally to verify:

mkdir -p results
.venv/bin/pytest tests/test_integration.py -v
cat results/integration-report.mdCode language: Shell Session (shell)

You should see a markdown table listing each test with its pass/fail status and duration.

Tip: The conftest approach is deliberate. It requires zero additional dependencies and works in any environment where pytest runs. No pip install needed.

3. Capture Docker Logs on Failure

When an integration test fails against a Dockerized service, the test output tells you what your code saw. The container logs tell you what the service did. You need both.

Docker Compose makes this easy:

docker compose logs > results/docker-logs.txt 2>&1Code language: Shell Session (shell)

This command dumps the stdout and stderr of every service defined in your docker-compose.yml into a single file. The 2>&1 redirects stderr into the same file so you do not miss error output from Docker itself.

Run it locally after your integration tests to see what it captures:

docker compose up -d
.venv/bin/pytest tests/test_integration.py -v
docker compose logs > results/docker-logs.txt 2>&1
cat results/docker-logs.txtCode language: Shell Session (shell)

For a Postgres container, you will see startup messages, connection logs, and any SQL errors:

postgres-1  | PostgreSQL init process complete; ready for start up.
postgres-1  | LOG:  database system is ready to accept connections
postgres-1  | LOG:  statement: CREATE TABLE ci_check ...Code language: Shell Session (shell)

When a test fails because the database rejected a query or was not ready in time, these logs contain the answer.

docker compose down -vCode language: Shell Session (shell)

Warning: Always append || true to the docker compose logs command in your Jenkinsfile. If Docker Compose is not running (because the integration stage was skipped or never started), the command fails and you lose the entire post block. The || true ensures the pipeline continues even if there are no logs to capture.

4. Use archiveArtifacts in Jenkins

Jenkins has a built-in step called archiveArtifacts that saves files from the workspace to the build record. These files become downloadable from the build page — forever, or until the build is deleted.

The syntax is:

archiveArtifacts artifacts: 'results/**', allowEmptyArchive: trueCode language: Groovy (groovy)
  • artifacts: 'results/**' — an Ant-style glob that matches everything inside the results/ directory, including subdirectories.
  • allowEmptyArchive: true — prevents the step from failing if no files match the pattern. This is important because some stages might not produce artifacts (for example, if the build fails before reaching the integration test stage).

When combined with the junit step you added in Part 4, the post block now captures both structured test data (JUnit XML for the Jenkins dashboard) and raw diagnostic files (logs, reports) as downloadable artifacts.

StepPurpose
junit 'results/*.xml'Parse test results into Jenkins dashboard
archiveArtifacts artifacts: 'results/**'Save all files for download
docker compose logs > results/docker-logs.txtCapture service logs before teardown

The junit step reads the XML files and populates the Test Result page. The archiveArtifacts step saves everything — XML, logs, markdown reports — as downloadable files on the build page.

5. Update the Jenkinsfile

Open the Jenkinsfile in your repository root. Replace its contents with the following complete pipeline:

pipeline {
    agent any
    stages {
        stage('Setup Python') {
            steps {
                sh 'python3 -m venv .venv'
                sh '.venv/bin/pip install --upgrade pip'
            }
        }
        stage('Install Dependencies') {
            steps {
                sh '.venv/bin/pip install -e ".[test]"'
            }
        }
        stage('Lint') {
            steps {
                sh '.venv/bin/ruff check src/ tests/'
            }
        }
        stage('Unit Tests') {
            steps {
                sh 'mkdir -p results'
                sh '.venv/bin/pytest tests/test_greet.py --junitxml=results/junit.xml'
            }
        }
        stage('Integration Tests') {
            steps {
                sh 'docker compose up -d'
                sh 'docker compose exec -T postgres pg_isready -U testuser -d testdb --timeout=30'
                sh '.venv/bin/pytest tests/test_integration.py --junitxml=results/junit-integration.xml -v'
            }
        }
    }
    post {
        always {
            sh 'docker compose logs > results/docker-logs.txt 2>&1 || true'
            sh 'docker compose down -v || true'
            junit 'results/*.xml'
            archiveArtifacts artifacts: 'results/**', allowEmptyArchive: true
        }
    }
}Code language: Groovy (groovy)

Here is what changed compared to Part 6:

  • Docker log capture — docker compose logs > results/docker-logs.txt 2>&1 || true runs before teardown, saving container output to a file. The || true ensures the pipeline continues even if no containers are running.
  • Teardown moved after log capture — docker compose down -v now runs after the logs are saved, not before. Order matters: capture evidence first, clean up second.
  • JUnit glob expanded — junit 'results/*.xml' uses a wildcard to pick up both junit.xml (unit tests) and junit-integration.xml (integration tests).
  • archiveArtifacts added — archiveArtifacts artifacts: 'results/**', allowEmptyArchive: true saves everything in results/ as downloadable build artifacts.

The always keyword guarantees these steps run regardless of whether the build passed or failed. This is critical — failures are exactly when you need the artifacts most.

Note: The conftest.py you created in Section 2 automatically writes results/integration-report.md whenever pytest runs the integration tests. No additional Jenkinsfile changes are needed to capture it — the results/** glob picks it up.

6. Verify the conftest.py Is in Place

Before pushing, confirm your tests/conftest.py contains the report-generation hooks from Section 2. The file should be committed to your repository so it runs on the Jenkins agent.

git add tests/conftest.py
git statusCode language: Shell Session (shell)
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	new file:   tests/conftest.pyCode language: Shell Session (shell)

If you already have a tests/conftest.py with other fixtures, add the three hook functions (pytest_configure, pytest_runtest_logreport, pytest_sessionfinish) to the existing file rather than replacing it.

Commit both files:

git add Jenkinsfile tests/conftest.py
git commit -m "Add artifact archiving and integration report to pipeline"
git push origin mainCode language: Shell Session (shell)

7. Trigger a Build and Download Artifacts

Go to your pipeline job in Jenkins and click Build Now.

Once the build completes, click on the build number (e.g., #12) in the Build History. You will see two new elements on the build page:

  • Build Artifacts — a section listing every file matched by the results/** glob.
  • Test Result — the existing JUnit dashboard (from Part 4), now showing both unit and integration test results.

Click Build Artifacts. You should see:

results/
  docker-logs.txt
  integration-report.md
  junit-integration.xml
  junit.xmlCode language: Shell Session (shell)

Click any file to download it. Open integration-report.md to see the human-readable test summary. Open docker-logs.txt to see the Postgres container output.

These files are permanently attached to the build record. Six months from now, if someone asks “what happened in build #12?”, the answer is one click away.

Tip: You can also access artifacts via the Jenkins REST API. The URL pattern is http://localhost:8080/job/helloci/<BUILD_NUMBER>/artifact/results/. This is useful for scripts that pull artifacts from CI for further analysis.

8. Break a Test and Diagnose With Artifacts

The real value of artifact archiving shows up when something breaks. Intentionally introduce a failure in the integration test to see the full diagnostic workflow.

Open tests/test_integration.py and add a test that queries a table that does not exist:

def test_query_nonexistent_table(db_connection):
    cursor = db_connection.cursor()
    cursor.execute("SELECT * FROM this_table_does_not_exist")
    rows = cursor.fetchall()
    assert len(rows) > 0Code language: Python (python)

Commit and push:

git add tests/test_integration.py
git commit -m "Add intentionally broken integration test"
git push origin mainCode language: Shell Session (shell)

Trigger the build in Jenkins. It will fail at the Integration Tests stage.

Now click on the failed build number. Go to Build Artifacts and download these files:

results/integration-report.md — Open it. You will see a table where the broken test shows FAIL:

| Test | Result | Duration |
|---|---|---|
| tests/test_integration.py::test_db_connection | PASS | 0.012s |
| tests/test_integration.py::test_query_nonexistent_table | FAIL | 0.003s |Code language: Shell Session (shell)

This tells you immediately which test failed and how long it took.

results/junit-integration.xml — This is the machine-readable version. Jenkins already parsed it into the Test Result dashboard, but having the raw file means you can process it with other tools too.

results/docker-logs.txt — Open it and search for the error. You will find the Postgres error message:

postgres-1  | ERROR:  relation "this_table_does_not_exist" does not exist at character 15Code language: Shell Session (shell)

The container log confirms the root cause: the table does not exist. The test assumed a table that was never created. No SSH access to the Jenkins agent was needed. No “try to reproduce locally.” The build produced everything you needed.

9. Post-Failure Forensics Workflow

When a build fails, follow this triage order:

  1. Test Result page first — click the build, click Test Result. This tells you which test failed, the assertion error, and the stack trace. In 80% of cases, the answer is right here.
  2. Integration report second — download results/integration-report.md. This gives you a bird’s eye view: how many tests ran, how many passed, and how long the run took. If zero tests ran, the problem is in setup, not in tests.
  3. Docker logs third — download results/docker-logs.txt. If the test failure involves a database, message queue, or any Dockerized service, the container logs reveal what the service actually did. Look for ERROR, FATAL, or connection refused messages.
  4. Console output last — click Console Output only if the artifacts above do not explain the failure. Console output contains everything — pipeline setup, git checkout, pip install — so there is a lot of noise to wade through. Use it for infrastructure failures (Python not found, Docker not installed, network timeout).
SymptomLook here first
Assertion error in a testTest Result page
Test never ranIntegration report (zero tests)
Database errorDocker logs
Stage failed before testsConsole output
TimeoutConsole output + Docker logs

This hierarchy works because you built the evidence into the pipeline. Each artifact answers a different question, and together they cover the full diagnostic space.

10. Clean Up the Broken Test

Remove the intentionally broken test from tests/test_integration.py (delete the test_query_nonexistent_table function you added in Section 8).

Commit and push:

git add tests/test_integration.py
git commit -m "Remove intentionally broken integration test"
git push origin mainCode language: Shell Session (shell)

Trigger one more build to confirm everything is green. Check the Build Artifacts on the passing build — the artifacts are still captured on success, giving you a baseline to compare against when the next failure occurs.

Summary

You moved from “hope the console log has enough information” to “every build produces downloadable evidence” in one pipeline update. Here is what you accomplished:

  • Created a pytest conftest plugin that writes results/integration-report.md with a human-readable test summary after every run.
  • Added docker compose logs > results/docker-logs.txt 2>&1 || true to capture container output before teardown.
  • Added archiveArtifacts artifacts: 'results/**', allowEmptyArchive: true to save all build outputs as downloadable files on the Jenkins build page.
  • Updated the post { always { } } block to capture logs, tear down containers, parse JUnit XML, and archive artifacts — in that order.
  • Triggered a passing build and verified that artifacts appear on the build page.
  • Intentionally broke an integration test and diagnosed the failure using only downloaded artifacts — no SSH, no local reproduction.
  • Established a post-failure forensics workflow: Test Result, integration report, Docker logs, then console output.

The principle behind all of this: a CI build should be a self-contained forensic record. When it fails, the build itself tells you why. You should never need to log into the Jenkins agent or guess what happened.

Next up in Part 8: you add workspace cleanup, build timeouts, and retry logic so your pipeline handles the messy realities of long-running builds and stale workspaces.

Similar Posts

Leave a Reply