Chapter 05: Jenkins for MLOps🔗

"Jenkins is the automation engine that glues your ML pipeline together."

5.1 What is Jenkins?🔗

Jenkins is an open-source automation server written in Java. It enables developers and ML engineers to build, test, and deploy software and ML models automatically.

Key Features🔗

┌──────────────────────────────────────────────────────┐
│                 JENKINS FEATURES                     │
│                                                      │
│  🔌 1800+ Plugins    → integrates with anything      │
│  🌍 Distributed      → run jobs on multiple agents   │
│  📋 Pipeline-as-Code → Jenkinsfile in your repo      │
│  🔔 Notifications    → Slack, Email, PagerDuty       │
│  🔄 Triggers         → Git hooks, cron, webhooks     │
│  🖥️  Web UI          → visual pipeline monitoring    │
└──────────────────────────────────────────────────────┘

5.2 Jenkins Architecture🔗

┌──────────────────────────────────────────────────────────────────┐
│                    JENKINS ARCHITECTURE                          │
│                                                                  │
│   ┌─────────────────────┐                                        │
│   │   JENKINS MASTER    │ ← Web UI, Job Scheduling, Config       │
│   │   (Controller)      │                                        │
│   └──────────┬──────────┘                                        │
│              │                                                   │
│     ┌────────┼────────┐                                          │
│     ▼        ▼        ▼                                          │
│  ┌──────┐ ┌──────┐ ┌──────┐                                      │
│  │Agent │ │Agent │ │Agent │  ← Workers that run jobs             │
│  │  1   │ │  2   │ │  3   │                                      │
│  │Linux │ │Docker│ │Cloud │                                      │
│  └──────┘ └──────┘ └──────┘                                      │
└──────────────────────────────────────────────────────────────────┘

• Master/Controller: Manages the web UI, schedules builds, coordinates agents
• Agent: Machine that executes the actual pipeline steps

5.3 Jenkins Setup (Docker)🔗

Easiest way to run Jenkins locally:

# Pull and run Jenkins
docker run -d \
  --name jenkins \
  -p 8080:8080 \
  -p 50000:50000 \
  -v jenkins_home:/var/jenkins_home \
  jenkins/jenkins:lts

# Get initial admin password
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword

# Access Jenkins at: http://localhost:8080

5.4 Jenkinsfile — Pipeline as Code🔗

A Jenkinsfile defines your CI/CD pipeline in code, stored in your Git repo.

Two Syntaxes🔗

Declarative (recommended — simpler)    Scripted (flexible — Groovy)
─────────────────────────────────────  ─────────────────────────────
pipeline {                             node {
  agent any                              stage('Build') {
  stages {                                 sh 'make build'
    stage('Build') {                     }
      steps {                          }
        sh 'make build'
      }
    }
  }
}

5.5 Complete MLOps Jenkinsfile🔗

// Jenkinsfile — MLOps CI/CD Pipeline
pipeline {
    agent {
        docker {
            image 'python:3.10-slim'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
        }
    }

    environment {
        GCR_REGISTRY = 'gcr.io/my-gcp-project'
        MODEL_NAME   = 'ml-model'
        ACCURACY_THRESHOLD = '0.85'
    }

    stages {

        stage('🔧 Setup') {
            steps {
                sh 'pip install -r requirements.txt'
            }
        }

        stage('🧹 Lint & Format') {
            steps {
                sh 'flake8 src/ --max-line-length=100'
                sh 'black --check src/'
            }
        }

        stage('🧪 Unit Tests') {
            steps {
                sh 'pytest tests/unit/ -v --junitxml=reports/unit_tests.xml'
            }
            post {
                always {
                    junit 'reports/unit_tests.xml'
                }
            }
        }

        stage('📊 Data Validation') {
            steps {
                sh 'python src/validate_data.py'
            }
        }

        stage('🏋️ Train Model') {
            steps {
                sh 'python src/train.py --config config/train_config.yaml'
                archiveArtifacts artifacts: 'models/*.pkl', fingerprint: true
            }
        }

        stage('📈 Evaluate Model') {
            steps {
                sh 'python src/evaluate.py'
                sh '''
                    ACCURACY=$(python -c "import json; print(json.load(open('metrics/results.json'))['accuracy'])")
                    echo "Accuracy: $ACCURACY"
                    python -c "
import json, sys
m = json.load(open('metrics/results.json'))
print(f'Accuracy: {m[\"accuracy\"]}')
assert m['accuracy'] >= float('${ACCURACY_THRESHOLD}'), f'Below threshold!'
print('✅ Model quality check PASSED')
"
                '''
            }
        }

        stage('🐳 Build Docker Image') {
            steps {
                sh "docker build -t ${GCR_REGISTRY}/${MODEL_NAME}:${BUILD_NUMBER} ."
                sh "docker tag ${GCR_REGISTRY}/${MODEL_NAME}:${BUILD_NUMBER} ${GCR_REGISTRY}/${MODEL_NAME}:latest"
            }
        }

        stage('📤 Push to Registry') {
            steps {
                withCredentials([file(credentialsId: 'gcp-sa-key', variable: 'GCP_KEY')]) {
                    sh "gcloud auth activate-service-account --key-file=$GCP_KEY"
                    sh "gcloud auth configure-docker"
                    sh "docker push ${GCR_REGISTRY}/${MODEL_NAME}:${BUILD_NUMBER}"
                    sh "docker push ${GCR_REGISTRY}/${MODEL_NAME}:latest"
                }
            }
        }

        stage('🚀 Deploy to Staging') {
            steps {
                sh "kubectl set image deployment/ml-model ml-model=${GCR_REGISTRY}/${MODEL_NAME}:${BUILD_NUMBER} -n staging"
                sh "kubectl rollout status deployment/ml-model -n staging"
            }
        }

        stage('✅ Integration Tests') {
            steps {
                sh 'pytest tests/integration/ -v'
            }
        }

        stage('🌐 Deploy to Production') {
            when {
                branch 'main'
            }
            input {
                message "Deploy to production?"
                ok "Yes, deploy!"
            }
            steps {
                sh "kubectl set image deployment/ml-model ml-model=${GCR_REGISTRY}/${MODEL_NAME}:${BUILD_NUMBER} -n production"
                sh "kubectl rollout status deployment/ml-model -n production"
            }
        }
    }

    post {
        success {
            slackSend(color: 'good', message: "✅ ML Pipeline SUCCESS: Build #${BUILD_NUMBER}")
        }
        failure {
            slackSend(color: 'danger', message: "❌ ML Pipeline FAILED: Build #${BUILD_NUMBER}")
            emailext(
                subject: "FAILED: ML Pipeline Build ${BUILD_NUMBER}",
                body: "Check Jenkins: ${BUILD_URL}",
                to: "mlteam@company.com"
            )
        }
        always {
            cleanWs()
        }
    }
}

5.6 Jenkins Pipeline Visualization🔗

Stage View (Jenkins Blue Ocean):

 Setup │ Lint  │ Tests │ Data  │ Train │ Eval  │ Docker│ Push  │ Deploy│
       │       │       │ Valid │       │       │       │       │       │
  ✅  │  ✅  │  ✅  │  ✅  │  ✅  │  ✅  │  ✅  │  ✅  │  ✅  │
 2s   │  5s   │  30s  │  10s  │  5min │  15s  │  45s  │  20s  │  30s  │

5.7 Jenkins Triggers🔗

# Trigger types in Jenkinsfile:

# 1. Poll SCM — check GitHub every 5 minutes
triggers {
    pollSCM('H/5 * * * *')
}

# 2. Webhook — GitHub pushes trigger instantly (preferred)
# Configure in GitHub → Settings → Webhooks → Add webhook → http://jenkins:8080/github-webhook/

# 3. Cron — run nightly retraining at 2am
triggers {
    cron('0 2 * * *')
}

# 4. Upstream — trigger when another job finishes
triggers {
    upstream(upstreamProjects: 'data-pipeline-job', threshold: hudson.model.Result.SUCCESS)
}

5.8 Jenkins vs GitHub Actions🔗

Next Chapter → 06: Docker