This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Configuration

Configuration parameters/options for CRD and Helm Charts

1 - Elasticsearch Config

Elasticsearch configuration paramaters with respect to logging operator

Elasticsearch configuration is easily customizable using helm as well kubectl. Since all the configurations are in the form YAML file, it can be easily changed and customized.

The values.yaml file for Elasticsearch setup can be found here. But if the setup is not done using Helm, in that case Kubernetes manifests needs to be customized.

Helm Chart Parameters

Name Value Description
clusterName elastic-prod Name of the elasticsearch cluster
esVersion 7.17.0 Major and minor version of elaticsearch
esPlugins [] Plugins list to install inside elasticsearch
esKeystoreSecret - Keystore secret to include in elasticsearch cluster
customConfiguration {} Additional configuration parameters for elasticsearch
esSecurity.enabled true To enabled the xpack security of elasticsearch
esMaster.replicas 3 Number of replicas for elasticsearch master node
esMaster.storage.storageSize 20Gi Size of the elasticsearch persistent volume for master
esMaster.storage.accessModes [ReadWriteOnce] Access modes of the elasticsearch persistent volume for master
esMaster.storage.storageClass default Storage class of the elasticsearch persistent volume for master
esMaster.jvmMaxMemory 1Gi Java max memory for elasticsearch master node
esMaster.jvmMinMemory 1Gi Java min memory for elasticsearch master node
esMaster.resources {} Resources for elasticsearch master pods
esMaster.nodeSelectors {} Nodeselectors map key-values for elasticsearch master pods
esMaster.affinity {} Affinity and anit-affinity for elasticsearch master pods
esMaster.tolerations {} Tolerations and taints for elasticsearch master pods
esData.replicas 3 Number of replicas for elasticsearch data node
esData.storage.storageSize 50Gi Size of the elasticsearch persistent volume for data
esData.storage.accessModes [ReadWriteOnce] Access modes of the elasticsearch persistent volume for data
esData.storage.storageClass default Storage class of the elasticsearch persistent volume for data
esData.jvmMaxMemory 1Gi Java max memory for elasticsearch data node
esData.jvmMinMemory 1Gi Java min memory for elasticsearch data node
esData.resources {} Resources for elasticsearch data pods
esData.nodeSelectors {} Nodeselectors map key-values for elasticsearch data pods
esData.affinity {} Affinity and anit-affinity for elasticsearch data pods
esData.tolerations {} Tolerations and taints for elasticsearch data pods
esIngestion.replicas - Number of replicas for elasticsearch ingestion node
esIngestion.storage.storageSize - Size of the elasticsearch persistent volume for ingestion
esIngestion.storage.accessModes - Access modes of the elasticsearch persistent volume for ingestion
esIngestion.storage.storageClass - Storage class of the elasticsearch persistent volume for ingestion
esIngestion.jvmMaxMemory - Java max memory for elasticsearch ingestion node
esIngestion.jvmMinMemory - Java min memory for elasticsearch ingestion node
esIngestion.resources - Resources for elasticsearch ingestion pods
esIngestion.nodeSelectors - Nodeselectors map key-values for elasticsearch ingestion pods
esIngestion.affinity - Affinity and anit-affinity for elasticsearch ingestion pods
esIngestion.tolerations - Tolerations and taints for elasticsearch ingestion pods
esClient.replicas - Number of replicas for elasticsearch ingestion node
esClient.storage.storageSize - Size of the elasticsearch persistent volume for client
esClient.storage.accessModes - Access modes of the elasticsearch persistent volume for client
esClient.storage.storageClass - Storage class of the elasticsearch persistent volume for client
esClient.jvmMaxMemory - Java max memory for elasticsearch client node
esClient.jvmMinMemory - Java min memory for elasticsearch client node
esClient.resources - Resources for elasticsearch client pods
esClient.nodeSelectors - Nodeselectors map key-values for elasticsearch client pods
esClient.affinity - Affinity and anit-affinity for elasticsearch client pods
esClient.tolerations - Tolerations and taints for elasticsearch client pods

CRD Object Definition Parameters

These are the parameters that are currently supported by the Logging Operator for the Elastisearch setup:-

  • esClusterName
  • esVersion
  • esMaster
  • esData
  • esIngestion
  • esClient
  • esSecurity
  • customConfig

esClusterName

esClusterName is a parameter to define the name of elasticsearch cluster.

esClusterName: "prod"

esVersion

esVersion is a CRD option through which we can define the version of elasticsearch.

esVersion: "7.16.0"

esPlugins

esPlugins is a CRD parameter through which we can define the list of plugins that needs to install inside elasticsearch cluster.

esPlugins: ["respository-s3", "repository-gcs"]

esKeystoreSecret

esKeystoreSecret is a CRD parameter through which we can define the keystore related secret to include in elasticsearch cluster.

esKeystoreSecret: keystore-secret

esMaster

esMaster is a general configuration parameter for Elasticsearch CRD for defining the configuration of Elasticsearch Master node. This includes Kubernetes related configurations and Elasticsearch properties related configurations.

  esMaster:
    replicas: 2
    storage:
      storageSize: 2Gi
      accessModes: [ReadWriteOnce]
      storageClass: do-block-storage
    jvmMaxMemory: "512m"
    jvmMinMemory: "512m"
    kubernetesConfig:
      elasticAffinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: beta.kubernetes.io/os
                    operator: In
                    values:
                      - linux
      nodeSelectors:
        kubernetes.io/os: linux
      priorityClassName: system-node-critical
      resources:
        requests:
          cpu: 101m
          memory: 512Mi
        limits:
          cpu: 2000m
          memory: 2Gi
      tolerations:
        - key: "example-key"
          operator: "Exists"
          effect: "NoSchedule"

Note:- All properties defined under kubernetesConfig can be used for other elasticsearch node types as well.

esData

esData is a general configuration parameter for Elasticsearch CRD for defining the configuration of Elasticsearch Data node. This includes Kubernetes related configurations and Elasticsearch properties related configurations.

  esData:
    replicas: 2
    storage:
      storageSize: 2Gi
      accessModes: [ReadWriteOnce]
      storageClass: do-block-storage
    jvmMaxMemory: "512m"
    jvmMinMemory: "512m"

esIngestion

esIngestion is a general configuration parameter for Elasticsearch CRD for defining the configuration of Elasticsearch Ingestion node. This includes Kubernetes related configurations and Elasticsearch properties related configurations.

  esIngestion:
    replicas: 2
    storage:
      storageSize: 2Gi
      accessModes: [ReadWriteOnce]
      storageClass: do-block-storage
    jvmMaxMemory: "512m"
    jvmMinMemory: "512m"

esClient

esClient is a general configuration parameter for Elasticsearch CRD for defining the configuration of Elasticsearch Client node. This includes Kubernetes related configurations and Elasticsearch properties related configurations.

  esClient:
    replicas: 2
    storage:
      storageSize: 2Gi
      accessModes: [ReadWriteOnce]
      storageClass: do-block-storage
    jvmMaxMemory: "512m"
    jvmMinMemory: "512m"

esSecurity

esSecurity s the security specification for Elasticsearch CRD. If we want to enable authentication and TLS, in that case, we can enable this configuration. To enable the authentication we need to provide secret reference in Kubernetes.

  esSecurity:
    autoGeneratePassword: true
    tlsEnabled: true  
#   existingSecret: elastic-custom-password

customConfig

customConfig is a Elasticsearch config file parameter through which we can provide custom configuration to elasticsearch nodes. This property is applicable for all types of nodes in elasticsearch.

  esMaster:
    replicas: 3
    storage:
      storageSize: 2Gi
      accessModes: [ReadWriteOnce]
      storageClass: do-block-storage
    customConfig: elastic-additional-config

2 - Fluentd Config

Elasticsearch configuration paramaters with respect to logging operator

Fluentd configuration is easily customizable using helm as well kubectl. Since all the configurations are in the form YAML file, it can be easily changed and customized.

The values.yaml file for Fluentd setup can be found here. But if the setup is not done using Helm, in that case Kubernetes manifests needs to be customized.

Helm chart parameters

Name Values Description
elasticSearchHost elasticsearch-master Hostname or URL of the elasticsearch server
indexNameStrategy namespace_name Strategy for creating indexes like:- namespace_name or pod_name
resources {} Resources for fluentd daemonset pods
nodeSelectors {} Nodeselectors map key-values for fluentd daemonset pods
affinity {} Affinity and anit-affinity for fluentd daemonset pods
tolerations {} Tolerations and taints for fluentd daemonset pods
customConfiguration {} Custom configuration parameters for fluentd
additionalConfiguration {} Additional configuration parameters for fluentd
esSecurity.enabled true To enabled the xpack security of fluentd
esSecurity.elasticSearchPassword elasticsearch-password Credentials for elasticsearch authentication

CRD object definition parameters

These are the parameters that are currently supported by the Logging Operator for the Fluentd setup:-

  • esCluster
  • indexNameStrategy
  • esSecurity
  • customConfig
  • additionalConfig
  • kubernetesConfig

esCluster

esCluster is a general parameter of Fluentd CRD for providing the information about Elasticsearch nodes.

  esCluster:
    host: elasticsearch-master

indexNameStrategy

indexNameStrategy naming standard for the indexes created inside the Elasticsearch cluster, It could be based on namespace like kubernetes-marketing-2022-07-04 or based on application/pod name kubernetes-gateway-application-2022-07-04.

  indexNameStrategy: namespace_name

esSecurity

esSecurity s the security specification for Fluentd CRD. If we want to enable authentication and TLS, in that case, we can enable this configuration. To enable the authentication we need to provide secret reference in Kubernetes.

  esSecurity:
    tlsEnabled: true
    existingSecret: elasticsearch-password

customConfig

customConfig is a field of Fluentd definition through which existing configuration of Fluentd can be overwritten, but be cautious while making this change because it can break the Fluentd.

  customConfig: fluentd-custom-config

additionalConfig

additionalConfig is a field of Fluentd definition through which additional configuration can be mounted inside the Fluentd log-shipper. Additional configmap will be part of fluentd configuration.

  additionalConfig: fluentd-additional-config

kubernetesConfig

kubernetesConfig is the general configuration paramater for Fluentd CRD in which we are defining the Kubernetes related configuration details like- image, tag, imagePullPolicy, and resources.

  kubernetesConfig:
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 2000m
        memory: 2Gi

3 - Kibana Config

Kibana configuration paramaters with respect to logging operator

Kibana configuration is easily customizable using helm as well kubectl. Since all the configurations are in the form YAML file, it can be easily changed and customized.

The values.yaml file for Kibana setup can be found here. But if the setup is not done using Helm, in that case Kubernetes manifests needs to be customized.

Helm chart parameters

Name Value Description
replicas 1 Number of deployment replicas for kibana
esCluster.esURL https://elasticsearch-master:9200 Hostname or URL of the elasticsearch server
esCluster.esVersion 7.17.0 Version of the kibana in pair with elasticsearch
esCluster.clusterName elasticsearch Name of the elasticsearch created by elasticsearch crd
resources {} Resources for kibana visualization pods
nodeSelectors {} Nodeselectors map key-values for kibana visualization pods
affinity {} Affinity and anit-affinity for kibana visualization pods
tolerations {} Tolerations and taints for kibana visualization pods
esSecurity.enabled true To enabled the xpack security of kibana
esSecurity.elasticSearchPassword elasticsearch-password Credentials for elasticsearch authentication
externalService.enabled false To create a LoadBalancer service of kibana
ingress.enabled false To enable the ingress resource for kibana
ingress.host kibana.opstree.com Hostname or URL on which kibana will be exposed
ingress.tls.enabled false To enable SSL on kibana ingress resource
ingress.tls.secret tls-secret SSL certificate for kibana ingress resource

CRD object definition parameters

These are the parameters that are currently supported by the Logging Operator for the Kibana setup:-

  • replicas
  • esCluster
  • esSecurity
  • kubernetesConfig

replicas

replicas is field definition of Kibana CRD in which we can define how many replicas/instances of Kibana we would like to run. Similar field like replicas in deployment and replicasets.

  replicas: 1

esCluster

esCluster is a general parameter of Fluentd CRD for providing the information about Elasticsearch nodes.

  esCluster:
    host: https://elasticsearch-master:9200
    esVersion: 7.16.0
    clusterName: elasticsearch

esSecurity

esSecurity s the security specification for Fluentd CRD. If we want to enable authentication and TLS, in that case, we can enable this configuration. To enable the authentication we need to provide secret reference in Kubernetes.

  esSecurity:
    tlsEnabled: true
    existingSecret: elasticsearch-password

kubernetesConfig

kubernetesConfig is the general configuration paramater for Fluentd CRD in which we are defining the Kubernetes related configuration details like- image, tag, imagePullPolicy, and resources.

  kubernetesConfig:
    resources:
      requests:
        cpu: 100m
        memory: 100Mi
      limits:
        cpu: 2000m
        memory: 2Gi