TopicTrick Logo
Topictrick
Home
Courses
YouTube
Blog

Learning Hubs

TOGAF & Enterprise ArchitectureMainframe: COBOL, CICS, IMS, DB2Claude API & AI Engineering
Utilities
Junior
Shop
Pricing
Loading...
DevOpsGitHub

Continuous Integration: The Bulletproof Pipeline

TT
TopicTrick Team
•
Continuous Integration: The Bulletproof Pipeline
← Back to GitHub Mastery

Continuous Integration: Building a Bulletproof Pipeline

Continuous Integration (CI) is the practice of automatically building, testing, and validating every code change before it can be merged. A bulletproof CI pipeline catches bugs before they reach production, enforces code quality standards, and gives developers confidence to ship frequently.

This guide builds a complete CI pipeline from scratch: linting and formatting checks, unit and integration tests, security scanning, build verification, artifact management, branch protection, and the discipline of the 10-minute rule. All examples use GitHub Actions.

What a Complete CI Pipeline Covers

text
Every pull request triggers:

┌─────────────┐   ┌─────────────┐   ┌─────────────┐   ┌─────────────┐
│    Lint +   │   │    Unit +   │   │  Security   │   │    Build    │
│   Format    │──▶│ Integration │──▶│    Scan     │──▶│   Verify   │
│             │   │    Tests    │   │             │   │             │
└─────────────┘   └─────────────┘   └─────────────┘   └─────────────┘
     ~1 min            ~3 min            ~2 min            ~2 min
                                                            â–¼
                                                    Merge allowed
StageWhat it checksBlocks merge?
LintCode style, unused imports, formattingYes
Type checkTypeScript compiler errorsYes
Unit testsIndividual function correctnessYes
Integration testsDatabase, API contract testsYes
Security scanDependency CVEs, secret leakageYes
BuildProduction bundle compilesYes
CoverageTest coverage thresholdConfigurable

The 10-Minute Rule

A CI pipeline that takes 30+ minutes gives developers no fast feedback loop. The rule: the entire CI suite must complete in under 10 minutes. Engineers waiting longer will start skipping CI or working around it.

text
Time budget:
  Lint + type check:     30 seconds
  Unit tests:           2–3 minutes
  Integration tests:    3–4 minutes
  Security scan:          1 minute
  Build:                  1 minute
  Total target:        < 10 minutes

Techniques to stay within budget:
  - Run lint/type-check/tests in parallel jobs
  - Cache node_modules, pip packages, Go modules
  - Skip unchanged packages in monorepos (Nx, Turborepo)
  - Run integration tests only on PR (not on every push)

Complete GitHub Actions CI Pipeline

yaml
# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

# Cancel in-progress runs for the same branch
concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: true

env:
  NODE_VERSION: '20'

jobs:
  # ── Job 1: Lint and type check ──────────────────────────────────────
  lint:
    name: Lint & Type Check
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: ESLint
        run: npm run lint

      - name: TypeScript type check
        run: npm run type-check

      - name: Prettier format check
        run: npm run format:check

  # ── Job 2: Unit tests ───────────────────────────────────────────────
  unit-tests:
    name: Unit Tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run unit tests with coverage
        run: npm run test:coverage

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          token: ${{ secrets.CODECOV_TOKEN }}
          files: ./coverage/lcov.info
          fail_ci_if_error: true

  # ── Job 3: Integration tests ────────────────────────────────────────
  integration-tests:
    name: Integration Tests
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_DB: testdb
          POSTGRES_USER: testuser
          POSTGRES_PASSWORD: testpass
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

      redis:
        image: redis:7-alpine
        options: >-
          --health-cmd "redis-cli ping"
          --health-interval 10s
          --health-retries 5
        ports:
          - 6379:6379

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run database migrations
        env:
          DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb
        run: npm run db:migrate

      - name: Run integration tests
        env:
          DATABASE_URL: postgresql://testuser:testpass@localhost:5432/testdb
          REDIS_URL: redis://localhost:6379
          JWT_SECRET: test-secret-for-ci
        run: npm run test:integration

  # ── Job 4: Security scan ────────────────────────────────────────────
  security:
    name: Security Scan
    runs-on: ubuntu-latest
    permissions:
      security-events: write
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Audit npm dependencies
        run: npm audit --audit-level=high

      - name: Run CodeQL analysis
        uses: github/codeql-action/init@v3
        with:
          languages: javascript, typescript

      - name: Perform CodeQL analysis
        uses: github/codeql-action/analyze@v3

  # ── Job 5: Build verification ───────────────────────────────────────
  build:
    name: Build
    runs-on: ubuntu-latest
    needs: [lint, unit-tests]    # Only build if lint and tests pass
    outputs:
      artifact-name: ${{ steps.artifact.outputs.name }}
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Build production bundle
        run: npm run build
        env:
          NODE_ENV: production

      - name: Set artifact name
        id: artifact
        run: echo "name=build-${{ github.sha }}" >> "$GITHUB_OUTPUT"

      - name: Upload build artifact
        uses: actions/upload-artifact@v4
        with:
          name: ${{ steps.artifact.outputs.name }}
          path: .next/
          retention-days: 7

Caching: The Biggest Speed-Up

Without caching, npm ci downloads and installs all dependencies on every run. With proper caching, it completes in seconds:

yaml
# Optimised dependency caching
- uses: actions/setup-node@v4
  with:
    node-version: '20'
    cache: 'npm'          # Built-in npm cache with actions/setup-node

# For more control, use actions/cache directly
- name: Cache node_modules
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-node-

# Python projects
- name: Cache pip
  uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}

# Go modules
- name: Cache Go modules
  uses: actions/cache@v4
  with:
    path: ~/go/pkg/mod
    key: ${{ runner.os }}-go-${{ hashFiles('go.sum') }}

The cache key includes a hash of the lock file. When package-lock.json changes, the cache is invalidated and dependencies are reinstalled fresh.

Linting and Formatting Setup

json
// package.json scripts
{
  "scripts": {
    "lint": "eslint src --ext .ts,.tsx --max-warnings 0",
    "lint:fix": "eslint src --ext .ts,.tsx --fix",
    "type-check": "tsc --noEmit",
    "format": "prettier --write .",
    "format:check": "prettier --check ."
  }
}
json
// .eslintrc.json
{
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/recommended",
    "plugin:@typescript-eslint/recommended-requiring-type-checking",
    "next/core-web-vitals"
  ],
  "rules": {
    "@typescript-eslint/no-unused-vars": "error",
    "@typescript-eslint/no-explicit-any": "error",
    "no-console": ["warn", { "allow": ["error", "warn"] }]
  }
}
json
// .prettierrc
{
  "semi": true,
  "singleQuote": true,
  "tabWidth": 2,
  "trailingComma": "es5",
  "printWidth": 100
}

The --max-warnings 0 flag fails the lint check if any ESLint warning exists — not just errors. This prevents warning accumulation over time.

Test Coverage Enforcement

typescript
// jest.config.ts
import type { Config } from 'jest';

const config: Config = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  collectCoverageFrom: [
    'src/**/*.{ts,tsx}',
    '!src/**/*.d.ts',
    '!src/**/*.stories.tsx',
    '!src/migrations/**',
  ],
  coverageThresholds: {
    global: {
      branches: 80,
      functions: 85,
      lines: 85,
      statements: 85,
    },
    // Stricter thresholds for critical modules
    './src/auth/**': {
      branches: 95,
      functions: 95,
      lines: 95,
    },
  },
  coverageReporters: ['lcov', 'text-summary'],
};

export default config;

If coverage drops below the threshold, npm run test:coverage exits with a non-zero code, failing the CI job.

Integration Tests with Real Services

GitHub Actions services spin up real Docker containers alongside your job:

yaml
services:
  postgres:
    image: postgres:16
    env:
      POSTGRES_DB: testdb
      POSTGRES_USER: testuser
      POSTGRES_PASSWORD: testpass
    options: >-
      --health-cmd pg_isready
      --health-interval 10s
      --health-timeout 5s
      --health-retries 5
    ports:
      - 5432:5432

The options: --health-cmd pg_isready makes the job wait until PostgreSQL is ready to accept connections before running your steps.

typescript
// src/__tests__/integration/users.test.ts
import { Pool } from 'pg';
import { createUser, getUserById } from '../../repositories/users';

describe('User repository integration', () => {
  let pool: Pool;

  beforeAll(async () => {
    pool = new Pool({ connectionString: process.env.DATABASE_URL });
    await pool.query('BEGIN');
  });

  afterAll(async () => {
    await pool.query('ROLLBACK');  // Clean up — leave database unchanged
    await pool.end();
  });

  it('creates and retrieves a user', async () => {
    const user = await createUser(pool, { email: 'test@example.com', name: 'Test User' });
    expect(user.id).toBeDefined();

    const found = await getUserById(pool, user.id);
    expect(found?.email).toBe('test@example.com');
  });
});

Wrapping each test suite in a transaction and rolling back ensures tests never leave persistent state in the test database.

Branch Protection: The Final Wall

Branch protection rules enforce that CI must pass before merging:

text
Settings → Branches → Branch protection rules → Add rule

Branch name pattern: main

☑ Require a pull request before merging
  ☑ Require approvals: 1
  ☑ Dismiss stale pull request approvals when new commits are pushed

☑ Require status checks to pass before merging
  ☑ Require branches to be up to date before merging
  Required status checks:
    ✓ Lint & Type Check
    ✓ Unit Tests
    ✓ Integration Tests
    ✓ Security Scan
    ✓ Build

☑ Require conversation resolution before merging
☑ Do not allow bypassing the above settings

With Do not allow bypassing enabled, even repository admins cannot push directly to main without passing CI. This is the most important setting — it prevents the "I'll just push directly in an emergency" pattern that erodes CI discipline.

Artifact Management: Build Once, Deploy Everywhere

yaml
# Build job uploads the artifact
- name: Upload build artifact
  uses: actions/upload-artifact@v4
  with:
    name: build-${{ github.sha }}
    path: |
      .next/
      public/
    retention-days: 30

# Deploy job downloads the same artifact (no rebuild)
deploy:
  needs: build
  runs-on: ubuntu-latest
  steps:
    - name: Download build artifact
      uses: actions/download-artifact@v4
      with:
        name: build-${{ needs.build.outputs.artifact-name }}
        path: .next/

    - name: Deploy to staging
      run: ./scripts/deploy.sh staging

Building once and deploying the same artifact to staging and production guarantees that what you tested is what you deployed. Rebuilding in the deploy job risks environmental differences changing the output.

Matrix Testing: Cross-Platform Verification

yaml
jobs:
  test:
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest]
        node: ['18', '20', '22']
        fail-fast: false    # Let all matrix jobs run; see all failures at once

    runs-on: ${{ matrix.os }}
    name: Test (Node ${{ matrix.node }} on ${{ matrix.os }})

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}
          cache: 'npm'
      - run: npm ci
      - run: npm test

For libraries published to npm, cross-platform testing catches Windows path separator bugs and Node.js version compatibility issues before users report them.

Detecting Secret Leakage

yaml
# Add to the security job
- name: Scan for leaked secrets
  uses: trufflesecurity/trufflehog@main
  with:
    path: ./
    base: ${{ github.event.repository.default_branch }}
    head: HEAD
    extra_args: --debug --only-verified

TruffleHog scans your commit diff for patterns matching API keys, tokens, and credentials. Running it on the diff (base to HEAD) rather than the full repo keeps it fast.

Additionally, add a .gitignore pre-commit hook:

bash
# .husky/pre-commit
#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

# Block commits that accidentally include .env files
if git diff --cached --name-only | grep -q "\.env$"; then
  echo "Error: .env file detected in staged changes. Remove it before committing."
  exit 1
fi

Monorepo CI: Only Test What Changed

For monorepos with multiple packages, running the full test suite on every commit wastes time:

yaml
# .github/workflows/ci.yml (monorepo with Nx)
jobs:
  affected:
    runs-on: ubuntu-latest
    outputs:
      apps: ${{ steps.affected.outputs.apps }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # Full history needed for Nx affected detection

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      - name: Get affected projects
        id: affected
        run: |
          AFFECTED=$(npx nx show projects --affected --base=origin/main --head=HEAD)
          echo "apps=$AFFECTED" >> "$GITHUB_OUTPUT"

  test:
    needs: affected
    if: needs.affected.outputs.apps != ''
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Test affected
        run: npx nx affected --target=test --base=origin/main --head=HEAD

Nx computes a dependency graph and only runs tests for packages whose code (or dependencies) changed.

Frequently Asked Questions

Q: Should CI run on every push or only on pull requests?

Run the full test suite on pull requests. For pushes to main (after merge), run the full suite plus additional steps like deployment. For pushes to feature branches, you can optionally run a lighter suite (lint + unit tests only) to give faster feedback during development — saving the full integration test suite for when a PR is opened. The important invariant: the full suite must pass before any commit lands on main.

Q: How do I handle flaky tests that randomly fail CI?

Flaky tests undermine trust in CI. Fix them rather than retrying. Use --retry sparingly and only as a temporary measure. Identify flakiness patterns: network-dependent tests without proper mocking, tests with shared mutable state, timing-sensitive tests using setTimeout. For integration tests, ensure each test cleans up after itself (wrap in transactions, seed fresh data per test). Track flakiness rates and treat any test with >1% flake rate as a bug.

Q: What is the difference between npm test and npm run test:coverage?

npm test runs tests and reports pass/fail. npm run test:coverage additionally instruments the code, collects which lines were executed, and generates a coverage report. Coverage collection adds ~20-30% to test runtime. Run npm run test:coverage in CI (to enforce thresholds and upload reports) and npm test locally for fast iteration. Some teams use --coverage --coverageThreshold only on the main branch CI and skip thresholds on feature branches.

Q: How do I prevent CI from becoming a bottleneck as the codebase grows?

Keep jobs under 3 minutes each by parallelising work. Use caching aggressively — cached npm ci takes 10 seconds vs 90 seconds uncached. For slow integration tests, consider splitting by domain (auth tests, payment tests, search tests) and running them in parallel matrix jobs. Profile which steps take the most time with the GitHub Actions timing view and optimise the top offenders first. The 10-minute total target should be a hard limit: if CI exceeds it, it is an engineering priority to bring it back.

Key Takeaway

A bulletproof CI pipeline catches bugs at the source: every pull request runs lint, type checks, unit tests, integration tests with real service dependencies, security scans, and build verification before merge is allowed. Cache dependencies to stay within the 10-minute target. Enforce branch protection so CI cannot be bypassed — even by admins. Upload build artifacts once and reuse them across deploy jobs to guarantee build-deploy consistency. Treat a failing CI run as a higher priority than new feature work — a broken pipeline stops the whole team.

Read next: GitHub Actions: Mastering YAML Syntax and Triggers →

Part of the GitHub Mastery Course — engineering the quality gate.