Md Sanwar Hossain

Software Engineer · Java · Spring Boot · Kubernetes · DevOps

Helm Chart Best Practices for Production Kubernetes: Subcharts, Testing & GitOps Rollout

Helm is the de-facto standard for packaging and deploying Kubernetes applications, but the gap between a working chart and a production-ready chart is enormous. This guide covers the patterns, tooling, and discipline that distinguish charts that survive contact with reality from those that cause outages.

Chart Structure and Naming Conventions

A production Helm chart is a deliberate directory structure, not just a collection of YAML files. Every template filename should reflect the Kubernetes resource kind it defines: deployment.yaml, service.yaml, hpa.yaml. Partials — named templates shared across files — live in _helpers.tpl. This naming convention is enforced by the Helm linter and makes templates scannable without opening each file.

mychart/
├── Chart.yaml
├── values.yaml
├── values.schema.json
├── charts/
└── templates/
    ├── _helpers.tpl
    ├── deployment.yaml
    ├── service.yaml
    ├── hpa.yaml
    ├── ingress.yaml
    ├── serviceaccount.yaml
    └── tests/
        └── test-connection.yaml

Every named template in _helpers.tpl should be prefixed with the chart name to avoid collisions when the chart is used as a subchart. The convention mychart.labels prevents subtle template-name collisions at the parent chart level.

{{- define "mychart.labels" -}}
helm.sh/chart: {{ include "mychart.chart" . }}
app.kubernetes.io/name: {{ include "mychart.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}

Values Schema Validation with JSON Schema

Adding a values.schema.json file to a Helm chart is one of the highest-leverage production improvements available. Without it, any typo in a values override silently deploys with incorrect configuration — a missing replica count becomes zero, a malformed resource limit creates an invalid pod spec only discovered at runtime. With JSON Schema, Helm validates values at install and upgrade time and produces a clear error before any Kubernetes API call is made.

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "required": ["image", "replicaCount"],
  "properties": {
    "replicaCount": {
      "type": "integer",
      "minimum": 1
    },
    "image": {
      "type": "object",
      "required": ["repository", "tag"],
      "properties": {
        "repository": { "type": "string" },
        "tag": { "type": "string" },
        "pullPolicy": {
          "type": "string",
          "enum": ["Always", "IfNotPresent", "Never"]
        }
      }
    }
  }
}

Schema validation catches the most common production incident caused by Helm: an operator intends to set image.tag=v1.2.3 but mistakenly sets image.Tag=v1.2.3. Without schema validation, Helm silently deploys the previous image tag. With schema validation, it fails immediately with a descriptive error.

Managing Subchart Dependencies

Subcharts allow composing charts from reusable components. A service that depends on PostgreSQL and Redis declares those as subchart dependencies in Chart.yaml. The condition field is essential for production use — it allows operators to disable a subchart when an external managed service is used instead, such as Amazon RDS or ElastiCache.

# Chart.yaml
apiVersion: v2
name: payment-service
version: 1.4.2
appVersion: "2.1.0"
dependencies:
  - name: postgresql
    version: "12.x.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: postgresql.enabled
  - name: redis
    version: "17.x.x"
    repository: "https://charts.bitnami.com/bitnami"
    condition: redis.enabled

Use global values for cross-chart configuration like image pull secrets, registry URLs, or environment labels that all subcharts must read. Values for subcharts are namespaced under the subchart name in values.yaml.

# values.yaml
global:
  imageRegistry: "registry.example.com"
  imagePullSecrets:
    - name: registry-credentials

postgresql:
  enabled: true
  auth:
    database: payments
    existingSecret: postgres-credentials

redis:
  enabled: true
  architecture: standalone

Helm Hooks for Lifecycle Management

Helm hooks execute Jobs or other resources at specific points in the release lifecycle. The most critical production hook is the database migration job: running migrations as a pre-upgrade hook ensures the schema is updated before the new application version starts, preventing the application from running against an incompatible schema.

# templates/migrations-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "mychart.fullname" . }}-migrations
  annotations:
    "helm.sh/hook": pre-upgrade,pre-install
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  ttlSecondsAfterFinished: 300
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrations
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          command: ["java", "-jar", "app.jar", "--spring.profiles.active=migrate-only"]
          env:
            - name: SPRING_DATASOURCE_URL
              valueFrom:
                secretKeyRef:
                  name: {{ .Values.database.secretName }}
                  key: url

The hook-delete-policy: before-hook-creation,hook-succeeded combination deletes the previous hook Job before creating a new one (preventing re-run conflicts) and cleans up successful Jobs so they do not accumulate in the cluster. Failed Jobs are retained for debugging.

Chart Testing with helm-unittest and ct

Two complementary tools cover the Helm chart testing surface: helm-unittest for unit-testing rendered templates without a cluster, and the Chart Testing tool (ct) for integration testing against a live cluster in CI.

helm-unittest renders templates with given values and asserts on the output. Write unit tests for every conditional in your templates and every values-driven behaviour:

# tests/deployment_test.yaml
suite: deployment tests
templates:
  - deployment.yaml
tests:
  - it: should set the correct replica count
    set:
      replicaCount: 3
    asserts:
      - equal:
          path: spec.replicas
          value: 3
  - it: should use custom image tag
    set:
      image.tag: "v2.5.1"
    asserts:
      - matchRegex:
          path: spec.template.spec.containers[0].image
          pattern: ":v2.5.1$"
  - it: should disable HPA when autoscaling is off
    set:
      autoscaling.enabled: false
    templates:
      - hpa.yaml
    asserts:
      - hasDocuments:
          count: 0

Chart Testing (ct) integrates with GitHub Actions to lint, install, and test changed charts on every pull request:

# .github/workflows/chart-test.yaml
name: Chart Testing
on: [pull_request]
jobs:
  chart-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: helm/chart-testing-action@v2.6.1
      - run: ct lint --config ct.yaml
      - uses: helm/kind-action@v1.9.0
      - run: ct install --config ct.yaml

Multi-Cluster GitOps Rollout with Argo CD

The Argo CD ApplicationSet controller enables multi-cluster Helm chart rollout from a single source of truth. A single ApplicationSet resource expands into per-cluster Application resources, each pointing at the same chart version with environment-specific values:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payment-service
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: staging
            url: https://kubernetes.staging.example.com
            valuesFile: values-staging.yaml
          - cluster: production-eu
            url: https://kubernetes.eu.example.com
            valuesFile: values-production-eu.yaml
  template:
    metadata:
      name: "payment-service-{{cluster}}"
    spec:
      source:
        repoURL: https://charts.example.com
        chart: payment-service
        targetRevision: "1.4.2"
        helm:
          valueFiles: ["{{valuesFile}}"]
      destination:
        server: "{{url}}"
        namespace: payments
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

With selfHeal: true, Argo CD continuously reconciles cluster state to Git state. Manual changes to the cluster are reverted within seconds, enforcing the Git repository as the single authoritative source of truth for every environment.

Security Hardening in Helm Charts

Production Helm charts must encode security best practices by default. Make the secure configuration the default and require explicit override to deviate from it. The most impactful defaults are non-root user, read-only root filesystem, and dropped Linux capabilities:

securityContext:
  runAsNonRoot: true
  runAsUser: {{ .Values.podSecurityContext.runAsUser | default 1000 }}
  fsGroup: {{ .Values.podSecurityContext.fsGroup | default 1000 }}
  seccompProfile:
    type: RuntimeDefault
containers:
  - name: {{ .Chart.Name }}
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: [ALL]

For secrets, use the existingSecret pattern: the chart accepts a Kubernetes Secret name and mounts it via secretKeyRef. The secret itself is managed outside Helm by External Secrets Operator or Vault Agent Injector. This prevents sensitive values from ever appearing in Helm release history stored as Kubernetes Secrets in the cluster.

"A Helm chart is an operational contract. The best production charts encode your organization's security posture, resource constraints, and lifecycle hooks so every deployment inherits them automatically — not because operators remember to add them."

Key Takeaways

• Add values.schema.json to validate inputs at install and upgrade time and prevent silent misconfiguration.
• Declare subchart dependencies with condition flags to support both bundled and external managed services.
• Use pre-upgrade hooks for database migrations with hook-delete-policy: before-hook-creation,hook-succeeded.
• Test every template conditional with helm-unittest before opening a pull request; use ct for cluster integration tests in CI.
• Use Argo CD ApplicationSets for multi-cluster rollout from a single Git source of truth with automated sync and self-healing.
• Default to secure: non-root user, read-only root filesystem, dropped Linux capabilities, and the existingSecret pattern for credentials.

Conclusion

The difference between a Helm chart that works in a demo and one that runs reliably in production comes down to four areas of discipline: validation (JSON schema, lint, unit tests), lifecycle management (hooks for migrations and cleanup), security defaults (non-root, secret references), and GitOps integration (ApplicationSets, automated sync). Investing in these patterns early pays compound returns as the fleet grows: every environment that consumes the chart inherits the same operational baseline, reducing both toil and outage risk.

Treat Helm charts as first-class deliverables reviewed with the same rigour as application code. A chart review that checks for schema validation, security context defaults, and hook policies prevents exactly the category of incidents that generic code review cannot catch: silent misconfigurations that surface only under production load.

Md Sanwar Hossain

Software Engineer · Java · Spring Boot · Microservices · Kubernetes · DevOps