--- /dev/null
+---
+layout: global
+title: HDFS charts
+---
+
+# HDFS charts
+
+Helm charts for launching HDFS daemons in a K8s cluster. The main entry-point
+chart is `hdfs-k8s`, which is a uber-chart that specifies other charts as
+dependency subcharts. This means you can launch all HDFS components using
+`hdfs-k8s`.
+
+Note that the HDFS charts are currently in pre-alpha quality. They are also
+being heavily revised and are subject to change.
+
+HDFS on K8s supports the following features:
+ - namenode high availability (HA): HDFS namenode daemons are in charge of
+ maintaining file system metadata concerning which directories have which
+ files and where are the file data. Namenode crash will cause service outage.
+ HDFS can run two namenodes in active/standby setup. HDFS on K8s supports HA.
+ - K8s persistent volumes (PV) for metadata: Namenode crash will cause service
+ outage. Losing namenode metadata can lead to loss of file system. HDFS on
+ K8s can store the metadata in remote K8s persistent volumes so that metdata
+ can remain intact even if both namenode daemons are lost or restarted.
+ - K8s HostPath volumes for file data: HDFS datanodes daemons store actual
+ file data. File data should also survive datanode crash or restart. HDFS on
+ K8s stores the file data on the local disks of the K8s cluster nodes using
+ K8s HostPath volumes. (We plan to switch to a better mechanism, K8s
+ persistent local volumes)
+ - Kerberos: Vanilla HDFS is not secure. Intruders can easily write custom
+ client code, put a fake user name in requests and steal data. Production
+ HDFS often secure itself using Kerberos. HDFS on K8s supports Kerberos.
+
+Here is the list of all charts.
+
+ - hdfs-k8s: main uber-chart. Launches other charts.
+ - hdfs-namenode-k8s: a statefulset and other K8s components for launching HDFS
+ namenode daemons, which maintains file system metadata. The chart supports
+ namenode high availability (HA).
+ - hdfs-datanode-k8s: a daemonset and other K8s components for launching HDFS
+ datanode daemons, which are responsible for storing file data.
+ - hdfs-config-k8s: a configmap containing Hadoop config files for HDFS.
+ - zookeeper: This chart is NOT in this repo. But hdfs-k8s pulls the zookeeper
+ chart in the incubator remote repo
+ (https://kubernetes-charts-incubator.storage.googleapis.com/)
+ as a dependency and launhces zookeeper daemons. Zookeeper makes sure
+ only one namenode is active in the HA setup, while the other namenode
+ becomes standby. By default, we will launch three zookeeper servers.
+ - hdfs-journalnode-k8s: a statefulset and other K8s components for launching
+ HDFS journalnode quorums, which ensures the file system metadata are
+ properly shared among the two namenode daemons in the HA setup.
+ By default, we will launch three journalnode servers.
+ - hdfs-client-k8s: a pod that is configured to run Hadoop client commands
+ for accessing HDFS.
+ - hdfs-krb5-k8s: a size-1 statefulset and other K8s components for launching
+ a Kerberos server, which can be used to secure HDFS. Disabled by default.
+ - hdfs-simple-namenode-k8s: Disabled by default. A simpler setup of the
+ namenode that launches only one namenode. i.e. This does not support HA. It
+ does not support Kerberos nor persistent volumes either. As it does not
+ support HA, we also don't need zookeeper nor journal nodes. You may prefer
+ this if you want the simplest possible setup.
+
+# Prerequisite
+
+Requires Kubernetes 1.6+ as the `namenode` and `datanodes` are using
+`ClusterFirstWithHostNet`, which was introduced in Kubernetes 1.6
+
+# Usage
+
+## Basic
+
+The HDFS daemons can be launched using the main `hdfs-k8s` chart. First, build
+the main chart using:
+
+```
+ $ helm repo add incubator \
+ https://kubernetes-charts-incubator.storage.googleapis.com/
+ $ helm dependency build charts/hdfs-k8s
+```
+
+Zookeeper, journalnodes and namenodes need persistent volumes for storing
+metadata. By default, the helm charts do not set the storage class name for
+dynamically provisioned volumes, nor does it use persistent volume selectors for
+static persistent volumes.
+
+This means it will rely on a provisioner for default storage volume class for
+dynamic volumes. Or if your cluster has statically provisioned volumes, the
+chart will match existing volumes entirely based on the size requirements. To
+override this default behavior, you can specify storage volume classes for
+dynamic volumes, or volume selectors for static volumes. See below for how to
+set these options.
+
+ - namenodes: Each of the two namenodes needs at least a 100 GB volume. i.e.
+ Yon need two 100 GB volumes. This can be overridden by the
+ `hdfs-namenode-k8s.persistence.size` option.
+ You can also override the storage class or the selector using
+ `hdfs-namenode-k8s.persistence.storageClass`, or
+ `hdfs-namenode-k8s.persistence.selector` respectively. For details, see the
+ values.yaml file inside `hdfs-namenode-k8s` chart dir.
+ - zookeeper: You need three > 5 GB volumes. i.e. Each of the two zookeeper
+ servers will need at least 5 GB in the volume. Can be overridden by
+ the `zookeeper.persistence.size` option. You can also override
+ the storage class using `zookeeper.persistence.storageClass`.
+ - journalnodes: Each of the three journalnodes will need at least 20 GB in
+ the volume. The size can be overridden by the
+ `hdfs-journalnode-k8s.persistence.size` option.
+ You can also override the storage class or the selector using
+ `hdfs-journalnode-k8s.persistence.storageClass`, or
+ `hdfs-journalnode-k8s.persistence.selector` respectively. For details, see the
+ values.yaml file inside `hdfs-journalnode-k8s` chart dir.
+ - kerberos: The single Kerberos server will need at least 20 GB in the volume.
+ The size can be overridden by the `hdfs-krb5-k8s.persistence.size` option.
+ You can also override the storage class or the selector using
+ `hdfs-krb5-k8s.persistence.storageClass`, or
+ `hdfs-krb5-k8s.persistence.selector` respectively. For details, see the
+ values.yaml file inside `hdfs-krb5-k8s` chart dir.
+
+Then launch the main chart. Specify the chart release name say "my-hdfs",
+which will be the prefix of the K8s resource names for the HDFS components.
+
+```
+ $ helm install -n my-hdfs charts/hdfs-k8s
+```
+
+Wait for all daemons to be ready. Note some daemons may restart themselves
+a few times before they become ready.
+
+```
+ $ kubectl get pod -l release=my-hdfs
+
+ NAME READY STATUS RESTARTS AGE
+ my-hdfs-client-c749d9f8f-d5pvk 1/1 Running 0 2m
+ my-hdfs-datanode-o7jia 1/1 Running 3 2m
+ my-hdfs-datanode-p5kch 1/1 Running 3 2m
+ my-hdfs-datanode-r3kjo 1/1 Running 3 2m
+ my-hdfs-journalnode-0 1/1 Running 0 2m
+ my-hdfs-journalnode-1 1/1 Running 0 2m
+ my-hdfs-journalnode-2 1/1 Running 0 1m
+ my-hdfs-namenode-0 1/1 Running 3 2m
+ my-hdfs-namenode-1 1/1 Running 3 2m
+ my-hdfs-zookeeper-0 1/1 Running 0 2m
+ my-hdfs-zookeeper-1 1/1 Running 0 2m
+ my-hdfs-zookeeper-2 1/1 Running 0 2m
+```
+
+Namenodes and datanodes are currently using the K8s `hostNetwork` so they can
+see physical IPs of each other. If they are not using `hostNetowrk`,
+overlay K8s network providers such as weave-net may mask the physical IPs,
+which will confuse the data locality later inside namenodes.
+
+Finally, test with the client pod:
+
+```
+ $ _CLIENT=$(kubectl get pods -l app=hdfs-client,release=my-hdfs -o name | \
+ cut -d/ -f 2)
+ $ kubectl exec $_CLIENT -- hdfs dfsadmin -report
+ $ kubectl exec $_CLIENT -- hdfs haadmin -getServiceState nn0
+ $ kubectl exec $_CLIENT -- hdfs haadmin -getServiceState nn1
+
+ $ kubectl exec $_CLIENT -- hadoop fs -rm -r -f /tmp
+ $ kubectl exec $_CLIENT -- hadoop fs -mkdir /tmp
+ $ kubectl exec $_CLIENT -- sh -c \
+ "(head -c 100M < /dev/urandom > /tmp/random-100M)"
+ $ kubectl exec $_CLIENT -- hadoop fs -copyFromLocal /tmp/random-100M /tmp
+```
+
+## Kerberos
+
+Kerberos can be enabled by setting a few related options:
+
+```
+ $ helm install -n my-hdfs charts/hdfs-k8s \
+ --set global.kerberosEnabled=true \
+ --set global.kerberosRealm=MYCOMPANY.COM \
+ --set tags.kerberos=true
+```
+
+This will launch all charts including the Kerberos server, which will become
+ready pretty soon. However, HDFS daemon charts will be blocked as the deamons
+require Kerberos service principals to be available. So we need to unblock
+them by creating those principals.
+
+First, create a configmap containing the common Kerberos config file:
+
+```
+ _MY_DIR=~/krb5
+ mkdir -p $_MY_DIR
+ _KDC=$(kubectl get pod -l app=hdfs-krb5,release=my-hdfs --no-headers \
+ -o name | cut -d/ -f2)
+ _run kubectl cp $_KDC:/etc/krb5.conf $_MY_DIR/tmp/krb5.conf
+ _run kubectl create configmap my-hdfs-krb5-config \
+ --from-file=$_MY_DIR/tmp/krb5.conf
+```
+
+Second, create the service principals and passwords. Kerberos requires service
+principals to be host specific. Some HDFS daemons are associated with your K8s
+cluster nodes' physical host names say kube-n1.mycompany.com, while others are
+associated with Kubernetes virtual service names, for instance
+my-hdfs-namenode-0.my-hdfs-namenode.default.svc.cluster.local. You can get
+the list of these host names like:
+
+```
+ $ _HOSTS=$(kubectl get nodes \
+ -o=jsonpath='{.items[*].status.addresses[?(@.type == "Hostname")].address}')
+
+ $ _HOSTS+=$(kubectl describe configmap my-hdfs-config | \
+ grep -A 1 -e dfs.namenode.rpc-address.hdfs-k8s \
+ -e dfs.namenode.shared.edits.dir |
+ grep "<value>" |
+ sed -e "s/<value>//" \
+ -e "s/<\/value>//" \
+ -e "s/:8020//" \
+ -e "s/qjournal:\/\///" \
+ -e "s/:8485;/ /g" \
+ -e "s/:8485\/hdfs-k8s//")
+```
+
+Then generate per-host principal accounts and password keytab files.
+
+```
+ $ _SECRET_CMD="kubectl create secret generic my-hdfs-krb5-keytabs"
+ $ for _HOST in $_HOSTS; do
+ kubectl exec $_KDC -- kadmin.local -q \
+ "addprinc -randkey hdfs/$_HOST@MYCOMPANY.COM"
+ kubectl exec $_KDC -- kadmin.local -q \
+ "addprinc -randkey HTTP/$_HOST@MYCOMPANY.COM"
+ kubectl exec $_KDC -- kadmin.local -q \
+ "ktadd -norandkey -k /tmp/$_HOST.keytab hdfs/$_HOST@MYCOMPANY.COM HTTP/$_HOST@MYCOMPANY.COM"
+ kubectl cp $_KDC:/tmp/$_HOST.keytab $_MY_DIR/tmp/$_HOST.keytab
+ _SECRET_CMD+=" --from-file=$_MY_DIR/tmp/$_HOST.keytab"
+ done
+```
+
+The above was building a command using a shell variable `SECRET_CMD` for
+creating a K8s secret that contains all keytab files. Run the command to create
+the secret.
+
+```
+ $ $_SECRET_CMD
+```
+
+This will unblock all HDFS daemon pods. Wait until they become ready.
+
+Finally, test the setup using the following commands:
+
+```
+ $ _NN0=$(kubectl get pods -l app=hdfs-namenode,release=my-hdfs -o name | \
+ head -1 | \
+ cut -d/ -f2)
+ $ kubectl exec $_NN0 -- sh -c "(apt install -y krb5-user > /dev/null)" \
+ || true
+ $ kubectl exec $_NN0 -- \
+ kinit -kt /etc/security/hdfs.keytab \
+ hdfs/my-hdfs-namenode-0.my-hdfs-namenode.default.svc.cluster.local@MYCOMPANY.COM
+ $ kubectl exec $_NN0 -- hdfs dfsadmin -report
+ $ kubectl exec $_NN0 -- hdfs haadmin -getServiceState nn0
+ $ kubectl exec $_NN0 -- hdfs haadmin -getServiceState nn1
+ $ kubectl exec $_NN0 -- hadoop fs -rm -r -f /tmp
+ $ kubectl exec $_NN0 -- hadoop fs -mkdir /tmp
+ $ kubectl exec $_NN0 -- hadoop fs -chmod 0777 /tmp
+ $ kubectl exec $_KDC -- kadmin.local -q \
+ "addprinc -randkey user1@MYCOMPANY.COM"
+ $ kubectl exec $_KDC -- kadmin.local -q \
+ "ktadd -norandkey -k /tmp/user1.keytab user1@MYCOMPANY.COM"
+ $ kubectl cp $_KDC:/tmp/user1.keytab $_MY_DIR/tmp/user1.keytab
+ $ kubectl cp $_MY_DIR/tmp/user1.keytab $_CLIENT:/tmp/user1.keytab
+
+ $ kubectl exec $_CLIENT -- sh -c "(apt install -y krb5-user > /dev/null)" \
+ || true
+
+ $ kubectl exec $_CLIENT -- kinit -kt /tmp/user1.keytab user1@MYCOMPANY.COM
+ $ kubectl exec $_CLIENT -- sh -c \
+ "(head -c 100M < /dev/urandom > /tmp/random-100M)"
+ $ kubectl exec $_CLIENT -- hadoop fs -ls /
+ $ kubectl exec $_CLIENT -- hadoop fs -copyFromLocal /tmp/random-100M /tmp
+```
+
+## Advanced options
+
+### Setting HostPath volume locations for datanodes
+
+HDFS on K8s stores the file data on the local disks of the K8s cluster nodes
+using K8s HostPath volumes. You may want to change the default locations. Set
+global.dataNodeHostPath to override the default value. Note the option
+takes a list in case you want to use multiple disks.
+
+```
+ $ helm install -n my-hdfs charts/hdfs-k8s \
+ --set "global.dataNodeHostPath={/mnt/sda1/hdfs-data0,/mnt/sda1/hdfs-data1}"
+```
+
+### Using an existing zookeeper quorum
+
+By default, HDFS on K8s pulls in the zookeeper chart in the incubator remote
+repo (https://kubernetes-charts-incubator.storage.googleapis.com/) as a
+dependency and launhces zookeeper daemons. But your K8s cluster may already
+have a zookeeper quorum.
+
+It is possible to use the existing zookeeper. We just need set a few options
+in the helm install command line. It should be something like:
+
+```
+ $helm install -n my-hdfs charts/hdfs-k8s \
+ --set condition.subchart.zookeeper=false \
+ --set global.zookeeperQuorumOverride=zk-0.zk-svc.default.svc.cluster.local:2181,zk-1.zk-svc.default.svc.cluster.local:2181,zk-2.zk-svc.default.svc.cluster.local:2181
+```
+
+Setting `condition.subchart.zookeeper` to false prevents the uber-chart from
+bringing in zookeeper as sub-chart. And the `global.zookeeperQuorumOverride`
+option specifies the custom address for a zookeeper quorum. Use your
+zookeeper address here.
+
+### Pinning namenodes to specific K8s cluster nodes
+
+Optionally, you can attach labels to some of your k8s cluster nodes so that
+namenodes will always run on those cluster nodes. This can allow your HDFS
+client outside the Kubernetes cluster to expect stable IP addresses. When used
+by those outside clients, Kerberos expects the namenode addresses to be stable.
+
+```
+ $ kubectl label nodes YOUR-HOST-1 hdfs-namenode-selector=hdfs-namenode
+ $ kubectl label nodes YOUR-HOST-2 hdfs-namenode-selector=hdfs-namenode
+```
+
+You should add the nodeSelector option to the helm chart command:
+
+```
+ $ helm install -n my-hdfs charts/hdfs-k8s \
+ --set hdfs-namenode-k8s.nodeSelector.hdfs-namenode-selector=hdfs-namenode \
+ ...
+```
+
+### Excluding datanodes from some K8s cluster nodes
+
+You may want to exclude some K8s cluster nodes from datanodes launch target.
+For instance, some K8s clusters may let the K8s cluster master node launch
+a datanode. To prevent this, label the cluster nodes with
+`hdfs-datanode-exclude`.
+
+```
+ $ kubectl label node YOUR-CLUSTER-NODE hdfs-datanode-exclude=yes
+```
+
+### Launching with a non-HA namenode
+
+You may want non-HA namenode since it is the simplest possible setup.
+Note this won't launch zookeepers nor journalnodes.
+
+The single namenode is supposed to be pinned to a cluster host using a node
+label. Attach a label to one of your K8s cluster node.
+
+```
+ $ kubectl label nodes YOUR-CLUSTER-NODE hdfs-namenode-selector=hdfs-namenode-0
+```
+
+The non-HA setup does not even use persistent vlumes. So you don't even
+need to prepare persistent volumes. Instead, it is using hostPath volume
+of the pinned cluster node. So, just launch the chart while
+setting options to turn off HA. You should add the nodeSelector option
+so that the single namenode would find the hostPath volume of the same cluster
+node when the pod restarts.
+
+```
+ $ helm install -n my-hdfs charts/hdfs-k8s \
+ --set tags.ha=false \
+ --set tags.simple=true \
+ --set global.namenodeHAEnabled=false \
+ --set hdfs-simple-namenode-k8s.nodeSelector.hdfs-namenode-selector=hdfs-namenode-0
+```
+
+# Security
+
+## K8s secret containing Kerberos keytab files
+
+The Kerberos setup creates a K8s secret containing all the keytab files of HDFS
+daemon service princialps. This will be mounted onto HDFS daemon pods. You may
+want to restrict access to this secret using k8s
+[RBAC](https://kubernetes.io/docs/admin/authorization/rbac/), to minimize
+exposure of the keytab files.
+
+## HostPath volumes
+`Datanode` daemons run on every cluster node. They also mount k8s `hostPath`
+local disk volumes. You may want to restrict access of `hostPath`
+using `pod security policy`.
+See [reference](https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac/README.md))
+
+## Credits
+
+Many charts are using public Hadoop docker images hosted by
+[uhopper](https://hub.docker.com/u/uhopper/).
Code Review / demo.git / blobdiff