Advanced: Configuration Overrides

To ensure all products in the Kubedoop run quickly and securely, all products provide corresponding default configurations. However, sometimes you may need to override these default configurations. This document will introduce how to override configurations in the Kubedoop.

All products support override mechanisms to provide a simple and flexible way to change program configurations, environment variables, and operator configurations.

Configuration Overrides

In roles and role groups, you can use the configOverrides field to override default configurations.

apiVersion: hdfs.kubedoop.dev/v1alpha1
kind: HdfsCluster
metadata:
  labels:
    app.kubernetes.io/name: hdfscluster
    app.kubernetes.io/instance: hdfscluster-sample
    app.kubernetes.io/part-of: hdfs-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: hdfs-operator
  name: hdfscluster-sample
spec:
  image:
    repository: qury.io/zncdata/hadoop
    tag: 3.3.4
  clusterConfig:
    zookeeperDiscoveryZNode: sample-hdfs
  nameNode:
    configOverrides:
      hdfs-site.xml:
        fs.trash.interval: "5"
    roleGroups:
      default:
        replicas: 2
        configOverrides:
          hdfs-site.xml:
            dfs.namenode.num.checkpoints.retained: "3"

In the above example, we defined:

• The hdfs-site.xml configuration of the namenode overrides fs.trash.interval to 5.
• The hdfs-site.xml configuration of the namenode role group default overrides dfs.namenode.num.checkpoints.retained to 3.

Environment Variable Overrides

In roles and role groups, you can use the envOverrides field to override default environment variables.

apiVersion: hdfs.kubedoop.dev/v1alpha1
kind: HdfsCluster
metadata:
  labels:
    app.kubernetes.io/name: hdfscluster
    app.kubernetes.io/instance: hdfscluster-sample
    app.kubernetes.io/part-of: hdfs-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: hdfs-operator
  name: hdfscluster-sample
spec:
  image:
    repository: qury.io/zncdatadev/hadoop
    tag: 3.3.4
  clusterConfig:
    zookeeperDiscoveryZNode: sample-hdfs
  nameNode:
    configOverrides:
      hdfs-site.xml:
        fs.trash.interval: "5"
    roleGroups:
      default:
        replicas: 2
        envOverrides:
          FOO: "bar"

In the above example, by adding the envOverrides field in the namenode role group default, we override the value of the environment variable FOO to bar.

Command Line Argument Overrides

In roles and role groups, you can use the commandOverrides field to override default command line arguments.

apiVersion: hdfs.kubedoop.dev/v1alpha1
kind: HdfsCluster
metadata:
  labels:
    app.kubernetes.io/name: hdfscluster
    app.kubernetes.io/instance: hdfscluster-sample
    app.kubernetes.io/part-of: hdfs-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: hdfs-operator
  name: hdfscluster-sample
spec:
  image:
    repository: qury.io/zncdatadev/hadoop
    tag: 3.3.4
  clusterConfig:
    zookeeperDiscoveryZNode: sample-hdfs
  nameNode:
    roleGroups:
      default:
        replicas: 2
        commandOverrides:
          - "-Dfoo=bar"

In the above example, by adding the commandOverrides field in the namenode role group default, we add the command line argument -Dfoo=bar to the namenode's startup command.

Pod Template Overrides

In roles and role groups, you can use the podTemplateOverrides field to override default pod templates.

apiVersion: hdfs.kubedoop.dev/v1alpha1
kind: HdfsCluster
metadata:
  labels:
    app.kubernetes.io/name: hdfscluster
    app.kubernetes.io/instance: hdfscluster-sample
    app.kubernetes.io/part-of: hdfs-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: hdfs-operator
  name: hdfscluster-sample
spec:
  image:
    repository: qury.io/zncdatadev/hadoop
    tag: 3.3.4
  clusterConfig:
    zookeeperDiscoveryZNode: sample-hdfs
  nameNode:
    podOverrides:
      spec:
        tolerations:
          - key: "key"
            operator: "Equal"
            value: "value"
            effect: "NoSchedule"
    roleGroups:
      default:
        replicas: 2
        podOverrides:
          metadata:
            labels:
              foo: bar

podOverrides is a field that overrides the default pod template through PodTemplateSpec. Any valid PodTemplateSpec configuration can be used in podOverrides.

In the above example, we added tolerations to the namenode role, and all namenode role groups will inherit this tolerations. At the same time, we added the metadata.labels field to the namenode role group default, and all namenode role groups will inherit this metadata.labels.

The levels of pod configuration overrides are as follows (from low to high):

• PodTemplateSpec generated by the operator
• Role-level PodTemplateSpec
• Role group-level PodTemplateSpec