Deploying Elasticsearch, Kibana, and Logstash on Kubernetes with ECK Operator

Elasticsearch is a powerful, distributed search and analytics engine, commonly used for log and event data. When running modern workloads, Kubernetes is often chosen for its orchestration, scalability, and resilience. Deploying Elasticsearch and the rest of the Elastic Stack on Kubernetes brings together robust search analytics with the operational flexibility of containers.

In this guide, I’ll walk you through deploying a complete Elastic Stack environment on Kubernetes, including:

• Installing the ECK operator
• Deploying an Elasticsearch cluster
• Setting up Kibana for visualisation
• Setting up Logstash for pipeline processing
• Exposing services via Ingress

What is the ECK Operator?

The Elastic Cloud on Kubernetes (ECK) Operator is a controller that makes it easy to deploy and manage Elastic Stack applications on Kubernetes. It supports lifecycle management, upgrades, security, persistent storage, and monitoring out-of-the-box. The ECK operator watches custom resources, provisions clusters, and automates many configuration steps for you.

Prerequisites

Requirement	Description
Kubernetes Cluster	A running Kubernetes cluster (v1.21+). You can set up one by following my previous post on Kubernetes or using my ansible playbook for Kubernetes.
kubectl	kubectl installed and configured to communicate with your cluster.
Helm	Helm v3.x installed (optional but highly recommended for simplified deployment).
StorageClass	A StorageClass configured for persistent volumes (PVs).
Ingress Controller	A LoadBalancer or Ingress Controller set up (e.g. NGINX, Traefik, or Istio) for external access. For bare metal clusters, see my guides on MetalLB and Traefik. for external access. For bare metal clusters, see my guides on MetalLB and Traefik.
Resources	At least 8GB of available memory across your cluster nodes to run the stack.
Concepts	Basic understanding of Kubernetes concepts (Pods, Services, Deployments).

Operator, Ingress, and Service Support in Helm Chart

• The Helm chart provided here supports ingress for Elasticsearch and Kibana.
• Logstash by default does not have an Ingress provided, but you can manually create an Ingress resource, a NodePort service, or a LoadBalancer to allow connections from outside your Kubernetes cluster (e.g. for syslog or Beats).

Step-by-Step Deployment Guide

Step 1: Create a Namespace

First, create a dedicated namespace for the ECK operator.

kubectl create namespace elastic-system

Step 2: Install the ECK Operator with Helm

Add the Elastic Helm repository

helm repo add elastic https://helm.elastic.co
helm repo update

Install the ECK operator

helm install elastic-operator elastic/eck-operator -n elastic-system

Verify the operator installation by checking that the ECK operator pod is running

kubectl -n elastic-system get pods

Step 3: Install Elastic Stack via Helm

For a more streamlined deployment process, use Helm to install the entire Elastic Stack in one go—this includes Elasticsearch, Kibana and Logstash. This approach simplifies installation and ensures integration between all components.

Create a values.yaml file

Create a file named elastic-stack-values.yaml with custom settings for your environment, for example:

eck-stack:
  eck-elasticsearch:
    enabled: true
    version: 9.1.2
    fullnameOverride: elasticsearch
    nodeSets:
      - name: default
        count: 1
        config:
          node.roles: ["master", "data", "ingest", "remote_cluster_client", "ml", "transform"]
        podTemplate:
          spec:
            containers:
              - name: elasticsearch
                resources:
                  requests:
                    memory: 4Gi
                    cpu: 1
                  limits:
                    memory: 8Gi
                    cpu: 4
            initContainers:
              - command:
                  - sh
                  - "-c"
                  - sysctl -w vm.max_map_count=262144
                name: sysctl
                securityContext:
                  privileged: true
                  runAsUser: 0
        volumeClaimTemplates:
          - metadata:
              name: elasticsearch-data
            spec:
              accessModes: [ "ReadWriteOnce" ]
              resources:
                requests:
                  storage: 100Gi
    monitoring:
      metrics:
        elasticsearchRefs:
          - name: elasticsearch
      logs:
        elasticsearchRefs:
          - name: elasticsearch
    ingress:
      enabled: true
      tls:
        enabled: true
        secretName: elasticsearch-tls
      className: nginx
      annotations:
        cert-manager.io/cluster-issuer: selfsigned-issuer
        nginx.ingress.kubernetes.io/rewrite-target: /
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      hosts:
        - host: elasticsearch.test.lab
          path: /

  eck-kibana:
    enabled: true
    version: 9.1.2
    fullnameOverride: kibana
    elasticsearchRef:
      name: elasticsearch
    count: 1
    podTemplate:
      spec:
        containers:
          - name: kibana
            resources:
              requests:
                memory: 1Gi
                cpu: 1
              limits:
                memory: 2Gi
                cpu: 2
    monitoring:
      metrics:
        elasticsearchRefs:
          - name: elasticsearch
      logs:
        elasticsearchRefs:
          - name: elasticsearch
    ingress:
      enabled: true
      tls:
        enabled: true
        secretName: kibana-tls
      className: nginx
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
        nginx.ingress.kubernetes.io/ssl-passthrough: "false"
        nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        cert-manager.io/cluster-issuer: selfsigned-issuer
      hosts:
        - host: kibana.test.lab
          path: /
    config:
      server.publicBaseUrl: "https://kibana.test.lab"
      xpack.fleet.outputs:
        - id: es-internal
          name: internal
          type: elasticsearch
          hosts: ["https://elasticsearch-es-http.elastic-stack.svc:9200"]
          is_default: true
          is_default_monitoring: true
          is_internal: true
          config:
            ssl.verification_mode: none
        - id: es-external
          name: external
          type: elasticsearch
          hosts: ["https://elasticsearch.test.lab:443"]
          config:
            ssl.verification_mode: none
      xpack.fleet.fleetServerHosts:
        - id: eck-fleet-server
          name: Internel Fleet Server
          host_urls: ["https://fleet-server-agent-http.elastic-stack.svc:8220"]
          is_internal: true
          is_default: true
        - id: eck-fleet-server-ingress
          name: External Fleet Server
          host_urls: ["https://fleet-server.test.lab:443"]
      xpack.fleet.packages:
        - name: system
          version: latest
        - name: elastic_agent
          version: latest
        - name: fleet_server
          version: latest
      xpack.fleet.agentPolicies:
        - name: Fleet Server on ECK policy
          id: eck-fleet-server
          namespace: default
          is_managed: true
          monitoring_enabled:
            - logs
            - metrics
          package_policies:
            - name: fleet_server-1
              id: fleet_server-1
              package:
                name: fleet_server
        - name: Elastic Agent on ECK policy
          id: eck-agent
          namespace: default
          is_managed: true
          monitoring_enabled:
            - logs
            - metrics
          unenroll_timeout: 900
          package_policies:
            - package:
                name: system
              name: system-1
               
  eck-logstash:
    enabled: true
    version: 9.1.2
    fullnameOverride: logstash
    count: 1
    monitoring:
      metrics:
        elasticsearchRefs:
          - name: elasticsearch
      logs:
        elasticsearchRefs:
          - name: elasticsearch
    elasticsearchRefs:
      - clusterName: eck
        name: elasticsearch
    podTemplate:
      spec:
        containers:
          - name: logstash
            resources:
              requests:
                memory: 2Gi
                cpu: 0.5
              limits:
                memory: 2Gi
                cpu: 1
            env:
              - name: ECK_ES_USER
                value: "elastic"
              - name: ECK_ES_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: elasticsearch-es-elastic-user
                    key: elastic
    pipelines:
      - pipeline.id: syslog
        config.string: |
          input {
            udp {
              port => 1514
              host => "0.0.0.0"
              type => "syslog"
            }
          }
          filter {}
          output {
            elasticsearch {
              hosts => [ "${ECK_ES_HOSTS}" ]
              user => "${ECK_ES_USER}"
              password => "${ECK_ES_PASSWORD}"
              index => "syslogs"
              ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"
            }
            stdout { codec => rubydebug }
          }

      - pipeline.id: snmp
        config.string: |-
          input {
            udp {
              port => 162
              host => "0.0.0.0"
              type => "snmptrap"
            }
          }
          filter {}
          output {
            elasticsearch {
              hosts => [ "${ECK_ES_HOSTS}" ]
              user => "${ECK_ES_USER}"
              password => "${ECK_ES_PASSWORD}"
              index => "snmp_traps"
              ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"
            }
            stdout { codec => rubydebug }
          }
      
    services:
      - name: logstash-udp
        service:
          spec:
            type: LoadBalancer
            selector:
              common.k8s.elastic.co/type: logstash
            ports:
              - name: syslog
                protocol: UDP
                port: 1514
                targetPort: 1514
              - name: snmp
                protocol: UDP
                port: 162
                targetPort: 162

  eck-apm-server:
    enabled: false
  eck-fleet-server:
    enabled: false
epr:
  enabled: false

Next steps

Continue by customising the values file for storage, security, and resource settings. Run helm install with your values file to deploy the stack.

helm install elastic-stack -f elastic-stack-values.yaml ./path-to-your-helm-chart -n elastic-system

Key Warning on Ingress and External Access (Important)

When using the Elastic Stack Helm chart, there is a fundamental difference in how components are exposed:

The official Helm chart logic only provides baked-in Kubernetes Ingress resources for Elasticsearch and Kibana (since they use standard HTTP/HTTPS).

For Logstash (and Fleet Server, if not using Ingress): You cannot use a standard HTTP Ingress resource because these services typically rely on raw TCP/UDP for data ingestion (Logstash) or specialised WebSocket connections (Fleet). To get data from outside the cluster, you must choose an alternative manual method:

1. NodePort Service: Expose the service on a high port on every cluster node.
2. LoadBalancer Service: If your cluster supports it (e.g., MetalLB), expose the service via a layer-4 LoadBalancer.
3. Manual Ingress Controller Configuration: If using NGINX/Traefik, you must manually configure a TCP/UDP Passthrough rule in the Ingress Controller’s ConfigMap, completely separate from the standard Kubernetes Ingress resource.

Step 4: Verification and Access

Verify Pods

Check that your cluster components have successfully been provisioned by the ECK operator.

Conclusion

With the ECK Operator and Helm, you can quickly bootstrap the Elastic Stack on Kubernetes, manage upgrades, and ensure best practices for security and performance. If you need more advanced setups—such as cross-cluster search, custom node sizing, or log forwarding—Elastic provides documentation and examples for further tuning.

Happy deploying! For troubleshooting tips or production checklists, check out my other guides.