Skip to content

GitOps Practices

Overview

This guide explores GitOps practices in the context of Java application development and deployment. GitOps is a modern approach to continuous delivery that uses Git as the single source of truth for declarative infrastructure and applications. We'll cover implementation strategies, tools, and best practices specifically for Java projects.

Prerequisites

Learning Objectives

  • Understand the core principles and benefits of GitOps
  • Implement GitOps workflows for Java applications
  • Configure GitOps tools like Flux and ArgoCD
  • Apply GitOps practices to Spring Boot and Java EE applications
  • Manage configuration and secrets using GitOps methodologies
  • Implement progressive delivery using GitOps
  • Troubleshoot common GitOps implementation issues

GitOps Fundamentals

What is GitOps?

GitOps is a set of practices that uses Git as the single source of truth for declarative infrastructure and applications. With GitOps:

  • Git repositories contain the desired state of your infrastructure
  • Automated processes ensure the actual state matches the desired state
  • All changes are made through Git operations (pull/merge requests)
  • Changes are automatically verified and deployed
┌───────────────────────────────────────────────────────────────────┐
│                      GitOps Workflow                               │
└───────────────────────────────────────────────────────────────────┘
     │                  │                    │                  │
     ▼                  ▼                    ▼                  ▼
┌──────────┐      ┌──────────┐       ┌───────────┐      ┌─────────────┐
│ Developer │      │    Git   │ ───▶ │  GitOps   │ ───▶ │ Kubernetes/ │
│           │      │ Repository│       │ Operator  │      │ Environment  │
│           │      │          │       │           │      │             │
└──────────┘      └──────────┘       └───────────┘      └─────────────┘
     │                                                         │
     └─────────────────────────────────────────────────────────┘
                          Observability

Core GitOps Principles

  1. Declarative Configuration: Infrastructure and application definitions are declarative, not imperative
  2. Version Controlled: All configuration is stored in Git, providing history, audit trails, and rollback capability
  3. Automated Deployment: Changes to the desired state are automatically applied to the system
  4. Continuous Reconciliation: System continuously ensures actual state matches desired state

Benefits for Java Applications

  • Consistency: Same deployment across all environments from development to production
  • Reliability: Reduced errors with automated deployments and drift detection
  • Auditability: Complete history of infrastructure and application changes
  • Developer-Centric: Developers use familiar Git workflows to drive deployments
  • Security: Improved security posture through PR reviews and access controls

GitOps Tools for Java Deployment

Flux CD

Flux is a tool that ensures your Kubernetes clusters match the desired state described in your Git repository.

Setting Up Flux for Java Applications

# Install Flux CLI
brew install fluxcd/tap/flux

# Check prerequisites
flux check --pre

# Bootstrap Flux on your cluster
flux bootstrap github \
  --owner=<your-github-username> \
  --repository=<repository-name> \
  --path=clusters/my-cluster \
  --personal

Configure Flux to monitor a Java application repository:

# java-app-source.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: java-application
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/example/java-app
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: java-application
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s"
  prune: true
  sourceRef:
    kind: GitRepository
    name: java-application
  validation: client
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: java-app
      namespace: default

Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.

Setting Up Argo CD for Java Applications

# Install Argo CD on your cluster
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Access Argo CD API Server
kubectl port-forward svc/argocd-server -n argocd 8080:443

Define a Java application in Argo CD:

# java-application.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: java-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/example/java-app.git
    targetRevision: HEAD
    path: k8s
  destination:
    server: https://kubernetes.default.svc
    namespace: java-apps
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

GitOps Repository Structure for Java Projects

A well-organized GitOps repository for Java applications might look like this:

├── apps/
│   └── java-app/
│       ├── base/
│       │   ├── deployment.yaml     # Core Kubernetes manifests for the Java app
│       │   ├── service.yaml
│       │   ├── configmap.yaml      # Application properties/configuration
│       │   └── kustomization.yaml
│       └── overlays/
│           ├── dev/
│           │   ├── kustomization.yaml
│           │   └── patch.yaml      # Dev-specific configuration
│           ├── staging/
│           │   ├── kustomization.yaml
│           │   └── patch.yaml      # Staging-specific configuration
│           └── production/
│               ├── kustomization.yaml
│               └── patch.yaml      # Production-specific configuration
├── infrastructure/
│   ├── monitoring/
│   │   ├── prometheus.yaml
│   │   └── grafana.yaml
│   └── ingress/
│       └── ingress.yaml
└── clusters/
    ├── dev-cluster/
    │   └── flux-system/            # Flux configuration for dev cluster
    └── prod-cluster/
        └── flux-system/            # Flux configuration for prod cluster

Implementing GitOps for Java Applications

Spring Boot Application GitOps

Example Kubernetes manifests for a Spring Boot application:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-app
  labels:
    app: spring-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spring-app
  template:
    metadata:
      labels:
        app: spring-app
    spec:
      containers:
      - name: spring-app
        image: example/spring-app:1.0.0  # Image tag will be updated by GitOps
        ports:
        - containerPort: 8080
        env:
        - name: SPRING_PROFILES_ACTIVE
          value: "prod"
        - name: JAVA_OPTS
          value: "-XX:+UseG1GC -Xmx512m -Xms256m"
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: 8080
          initialDelaySeconds: 20
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 15

Application Configuration Management

Managing Spring configuration properties through GitOps:

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: spring-app-config
data:
  application.properties: |
    server.port=8080
    spring.datasource.url=jdbc:postgresql://${DB_HOST}:5432/mydb
    spring.datasource.username=${DB_USER}
    spring.jpa.hibernate.ddl-auto=validate
    management.endpoints.web.exposure.include=health,info,metrics
    management.endpoint.health.probes.enabled=true
    management.health.livenessState.enabled=true
    management.health.readinessState.enabled=true

Secrets Management with Sealed Secrets

For sensitive configuration, use Sealed Secrets with GitOps:

# Install kubeseal
brew install kubeseal

# Create a sealed secret
kubectl create secret generic db-credentials \
  --from-literal=username=postgres \
  --from-literal=password=securepassword \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-credentials.yaml

# Reference secrets in deployment
spec:
  containers:
  - name: spring-app
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

Image Automation with Flux

Automatically update container images when new versions are available:

# image-automation.yaml
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageRepository
metadata:
  name: spring-app
  namespace: flux-system
spec:
  image: example/spring-app
  interval: 1m0s
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImagePolicy
metadata:
  name: spring-app
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: spring-app
  policy:
    semver:
      range: '>=1.0.0 <2.0.0'
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: spring-app
  namespace: flux-system
spec:
  interval: 1m0s
  sourceRef:
    kind: GitRepository
    name: java-application
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        email: fluxcdbot@users.noreply.github.com
        name: fluxcdbot
      messageTemplate: 'Update image to {{.NewTag}}'
  update:
    path: ./apps/java-app
    strategy: Setters

Progressive Delivery with GitOps

Canary Deployments Using Flagger

Implement canary deployments for Java applications with Flagger:

# canary.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: spring-app
  namespace: default
spec:
  provider: istio
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: spring-app
  progressDeadlineSeconds: 60
  service:
    port: 8080
    targetPort: 8080
    gateways:
    - public-gateway
    hosts:
    - app.example.com
  analysis:
    interval: 30s
    threshold: 10
    maxWeight: 50
    stepWeight: 5
    metrics:
    - name: request-success-rate
      thresholdRange:
        min: 99
      interval: 1m
    - name: request-duration
      thresholdRange:
        max: 500
      interval: 1m
    webhooks:
      - name: jmeter-load-test
        url: http://flagger-loadtester.test/
        timeout: 5s
        metadata:
          cmd: "jmeter -n -t /tests/spring-app-test.jmx -Jthreads=10 -Jduration=30"

A/B Testing with GitOps

Implementing A/B testing for Java applications:

# ab-test.yaml
apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: spring-app-ab-test
spec:
  # ... similar to canary setup
  analysis:
    # ... metrics like in canary
    match:
      - headers:
          x-user-type:
            regex: "^(premium|beta)$"
            value: "premium"
      - headers:
          cookie:
            regex: "^(.*?;)?(user=test)(;.*)?$"

Testing in GitOps Workflows

Automated Testing in GitOps Pipelines

Integrate testing as part of the GitOps workflow for Java applications:

# test-pipeline.yaml
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: java-app-test-pipeline
spec:
  params:
    - name: git-url
      type: string
    - name: git-revision
      type: string
  tasks:
    - name: git-clone
      taskRef:
        name: git-clone
      params:
        - name: url
          value: $(params.git-url)
        - name: revision
          value: $(params.git-revision)
    - name: run-tests
      runAfter:
        - git-clone
      taskRef:
        name: maven
      params:
        - name: GOALS
          value: ["test"]
      workspaces:
        - name: source
          workspace: shared-workspace
    - name: update-test-status
      runAfter:
        - run-tests
      taskRef:
        name: github-set-status
      params:
        - name: GITHUB_HOST_URL
          value: github.com
        - name: SHA
          value: $(params.git-revision)
        - name: STATE
          value: success
        - name: DESCRIPTION
          value: "Tests passed!"

Integration Tests After Deployment

Run integration tests after successful deployment with ArgoCD hooks:

apiVersion: batch/v1
kind: Job
metadata:
  name: spring-app-integration-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: integration-test
        image: example/spring-test-runner:latest
        command: ["java", "-jar", "/app/integration-tests.jar"]
        env:
        - name: APP_URL
          value: "http://spring-app:8080"
      restartPolicy: Never
  backoffLimit: 1

Monitoring and Observability in GitOps

Prometheus and Grafana Setup via GitOps

Deploy monitoring tools using GitOps:

# prometheus.yaml (simplified)
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: monitoring
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      app: spring-app
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
---
# service-monitor.yaml for Java app
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: spring-app-monitor
  namespace: monitoring
  labels:
    app: spring-app
spec:
  selector:
    matchLabels:
      app: spring-app
  endpoints:
  - port: http
    path: /actuator/prometheus
    interval: 15s

Implementing Dashboards as Code

Store Grafana dashboards in Git:

# grafana-dashboard.yaml
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  name: spring-app-dashboard
  namespace: monitoring
spec:
  json: >
    {
      "annotations": {...},
      "editable": true,
      "gnetId": null,
      "graphTooltip": 0,
      "id": null,
      "links": [],
      "panels": [
        {
          "aliasColors": {},
          "bars": false,
          "dashLength": 10,
          "dashes": false,
          "datasource": "Prometheus",
          "fill": 1,
          "fillGradient": 0,
          "gridPos": {
            "h": 8,
            "w": 12,
            "x": 0,
            "y": 0
          },
          "hiddenSeries": false,
          "id": 1,
          "legend": {...},
          "lines": true,
          "linewidth": 1,
          "nullPointMode": "null",
          "options": {
            "dataLinks": []
          },
          "percentage": false,
          "pointradius": 2,
          "points": false,
          "renderer": "flot",
          "seriesOverrides": [],
          "spaceLength": 10,
          "stack": false,
          "steppedLine": false,
          "targets": [
            {
              "expr": "jvm_memory_used_bytes{application=\"spring-app\", area=\"heap\"}",
              "legendFormat": "{{instance}} - {{id}}",
              "refId": "A"
            }
          ],
          "thresholds": [],
          "timeFrom": null,
          "timeRegions": [],
          "timeShift": null,
          "title": "JVM Memory Usage",
          "tooltip": {...},
          "type": "graph",
          "xaxis": {...},
          "yaxes": [...],
          "yaxis": {...}
        }
      ],
      "refresh": "10s",
      "schemaVersion": 22,
      "style": "dark",
      "tags": ["spring", "java"],
      "templating": {...},
      "time": {...},
      "timepicker": {...},
      "timezone": "",
      "title": "Spring Application Dashboard",
      "uid": "spring-app",
      "variables": {...},
      "version": 1
    }

GitOps Best Practices for Java Applications

Repository Structure Best Practices

  1. Separate Application and Infrastructure: Keep application code separate from deployment manifests
  2. Environment-Based Organization: Use directories or branches for different environments
  3. Kustomize or Helm: Use Kustomize or Helm for managing environment-specific configurations
  4. Clear Documentation: Document the repository structure and deployment process

Configuration Management

  1. Externalize Configurations: Keep application configuration external to the code
  2. Environment Variables: Use environment variables for deployment-specific values
  3. Secret Management: Implement secure secret management with tools like Sealed Secrets or Vault
  4. Config Validation: Validate configuration changes before applying them

CI/CD Integration

Integrate your existing Java CI pipeline with GitOps:

# .github/workflows/java-gitops.yml
name: Java GitOps CI

on:
  push:
    branches: [ main ]
    paths-ignore:
      - 'k8s/**'  # Avoid circular updates

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Set up JDK
      uses: actions/setup-java@v3
      with:
        java-version: '17'
        distribution: 'temurin'

    - name: Build with Maven
      run: mvn -B package --file pom.xml

    - name: Build and push Docker image
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        tags: example/spring-app:${{ github.sha }}

    - name: Update Kubernetes manifests
      run: |
        git config --global user.name 'CI Bot'
        git config --global user.email 'ci-bot@example.com'
        git clone https://github.com/example/gitops-repo.git
        cd gitops-repo
        # Update image tag in the appropriate kustomization.yaml
        sed -i "s|example/spring-app:.*|example/spring-app:${{ github.sha }}|" apps/java-app/base/kustomization.yaml
        git add .
        git commit -m "Update spring-app image to ${{ github.sha }}"
        git push

Drift Detection and Remediation

Configure your GitOps operator to detect and correct infrastructure drift:

# kustomization.yaml with drift detection
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: java-application
  namespace: flux-system
spec:
  interval: 10m
  path: "./k8s"
  prune: true  # Remove resources that are no longer in Git
  force: false # Don't force apply if there are conflicts
  sourceRef:
    kind: GitRepository
    name: java-application
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: spring-app
      namespace: default

Common Challenges and Solutions

Managing Database Migrations with GitOps

Use Flyway or Liquibase jobs in your GitOps workflow:

# database-migration-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: spring-app-db-migration
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - name: flyway
        image: flyway/flyway:8.5-alpine
        args:
          - migrate
        env:
          - name: FLYWAY_URL
            value: jdbc:postgresql://postgres:5432/mydb
          - name: FLYWAY_USER
            valueFrom:
              secretKeyRef:
                name: db-credentials
                key: username
          - name: FLYWAY_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-credentials
                key: password
          - name: FLYWAY_LOCATIONS
            value: classpath:db/migration
      restartPolicy: Never
  backoffLimit: 3

Handling Java Application Lifecycle Events

Use Kubernetes lifecycle hooks to handle application startup and shutdown:

spec:
  containers:
  - name: spring-app
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 10"]  # Allow time for graceful shutdown
    terminationGracePeriodSeconds: 60  # Give Spring Boot time to shut down gracefully

Multi-Cluster GitOps Approaches

Manage multiple clusters with GitOps:

# clusters/dev/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: "./apps/overlays/dev"
  prune: true
  sourceRef:
    kind: GitRepository
    name: gitops-repo

# clusters/prod/apps.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: "./apps/overlays/prod"
  prune: true
  sourceRef:
    kind: GitRepository
    name: gitops-repo

Conclusion

GitOps provides a powerful paradigm for deploying and managing Java applications in Kubernetes environments. By following GitOps practices, Java developers and operations teams can achieve more reliable, repeatable, and auditable deployments, with improved collaboration through Git-based workflows.

The declarative nature of GitOps works particularly well with containerized Java applications, providing a solid foundation for implementing continuous delivery while maintaining security and stability.

References