8 Helm charts for launching HDFS daemons in a K8s cluster. The main entry-point
9 chart is `hdfs-k8s`, which is a uber-chart that specifies other charts as
10 dependency subcharts. This means you can launch all HDFS components using
13 Note that the HDFS charts are currently in pre-alpha quality. They are also
14 being heavily revised and are subject to change.
16 HDFS on K8s supports the following features:
17 - namenode high availability (HA): HDFS namenode daemons are in charge of
18 maintaining file system metadata concerning which directories have which
19 files and where are the file data. Namenode crash will cause service outage.
20 HDFS can run two namenodes in active/standby setup. HDFS on K8s supports HA.
21 - K8s persistent volumes (PV) for metadata: Namenode crash will cause service
22 outage. Losing namenode metadata can lead to loss of file system. HDFS on
23 K8s can store the metadata in remote K8s persistent volumes so that metdata
24 can remain intact even if both namenode daemons are lost or restarted.
25 - K8s HostPath volumes for file data: HDFS datanodes daemons store actual
26 file data. File data should also survive datanode crash or restart. HDFS on
27 K8s stores the file data on the local disks of the K8s cluster nodes using
28 K8s HostPath volumes. (We plan to switch to a better mechanism, K8s
29 persistent local volumes)
30 - Kerberos: Vanilla HDFS is not secure. Intruders can easily write custom
31 client code, put a fake user name in requests and steal data. Production
32 HDFS often secure itself using Kerberos. HDFS on K8s supports Kerberos.
34 Here is the list of all charts.
36 - hdfs-k8s: main uber-chart. Launches other charts.
37 - hdfs-namenode-k8s: a statefulset and other K8s components for launching HDFS
38 namenode daemons, which maintains file system metadata. The chart supports
39 namenode high availability (HA).
40 - hdfs-datanode-k8s: a daemonset and other K8s components for launching HDFS
41 datanode daemons, which are responsible for storing file data.
42 - hdfs-config-k8s: a configmap containing Hadoop config files for HDFS.
43 - zookeeper: This chart is NOT in this repo. But hdfs-k8s pulls the zookeeper
44 chart in the incubator remote repo
45 (https://kubernetes-charts-incubator.storage.googleapis.com/)
46 as a dependency and launhces zookeeper daemons. Zookeeper makes sure
47 only one namenode is active in the HA setup, while the other namenode
48 becomes standby. By default, we will launch three zookeeper servers.
49 - hdfs-journalnode-k8s: a statefulset and other K8s components for launching
50 HDFS journalnode quorums, which ensures the file system metadata are
51 properly shared among the two namenode daemons in the HA setup.
52 By default, we will launch three journalnode servers.
53 - hdfs-client-k8s: a pod that is configured to run Hadoop client commands
55 - hdfs-krb5-k8s: a size-1 statefulset and other K8s components for launching
56 a Kerberos server, which can be used to secure HDFS. Disabled by default.
57 - hdfs-simple-namenode-k8s: Disabled by default. A simpler setup of the
58 namenode that launches only one namenode. i.e. This does not support HA. It
59 does not support Kerberos nor persistent volumes either. As it does not
60 support HA, we also don't need zookeeper nor journal nodes. You may prefer
61 this if you want the simplest possible setup.
65 Requires Kubernetes 1.6+ as the `namenode` and `datanodes` are using
66 `ClusterFirstWithHostNet`, which was introduced in Kubernetes 1.6
72 The HDFS daemons can be launched using the main `hdfs-k8s` chart. First, build
76 $ helm repo add incubator \
77 https://kubernetes-charts-incubator.storage.googleapis.com/
78 $ helm dependency build charts/hdfs-k8s
81 Zookeeper, journalnodes and namenodes need persistent volumes for storing
82 metadata. By default, the helm charts do not set the storage class name for
83 dynamically provisioned volumes, nor does it use persistent volume selectors for
84 static persistent volumes.
86 This means it will rely on a provisioner for default storage volume class for
87 dynamic volumes. Or if your cluster has statically provisioned volumes, the
88 chart will match existing volumes entirely based on the size requirements. To
89 override this default behavior, you can specify storage volume classes for
90 dynamic volumes, or volume selectors for static volumes. See below for how to
93 - namenodes: Each of the two namenodes needs at least a 100 GB volume. i.e.
94 Yon need two 100 GB volumes. This can be overridden by the
95 `hdfs-namenode-k8s.persistence.size` option.
96 You can also override the storage class or the selector using
97 `hdfs-namenode-k8s.persistence.storageClass`, or
98 `hdfs-namenode-k8s.persistence.selector` respectively. For details, see the
99 values.yaml file inside `hdfs-namenode-k8s` chart dir.
100 - zookeeper: You need three > 5 GB volumes. i.e. Each of the two zookeeper
101 servers will need at least 5 GB in the volume. Can be overridden by
102 the `zookeeper.persistence.size` option. You can also override
103 the storage class using `zookeeper.persistence.storageClass`.
104 - journalnodes: Each of the three journalnodes will need at least 20 GB in
105 the volume. The size can be overridden by the
106 `hdfs-journalnode-k8s.persistence.size` option.
107 You can also override the storage class or the selector using
108 `hdfs-journalnode-k8s.persistence.storageClass`, or
109 `hdfs-journalnode-k8s.persistence.selector` respectively. For details, see the
110 values.yaml file inside `hdfs-journalnode-k8s` chart dir.
111 - kerberos: The single Kerberos server will need at least 20 GB in the volume.
112 The size can be overridden by the `hdfs-krb5-k8s.persistence.size` option.
113 You can also override the storage class or the selector using
114 `hdfs-krb5-k8s.persistence.storageClass`, or
115 `hdfs-krb5-k8s.persistence.selector` respectively. For details, see the
116 values.yaml file inside `hdfs-krb5-k8s` chart dir.
118 Then launch the main chart. Specify the chart release name say "my-hdfs",
119 which will be the prefix of the K8s resource names for the HDFS components.
122 $ helm install -n my-hdfs charts/hdfs-k8s
125 Wait for all daemons to be ready. Note some daemons may restart themselves
126 a few times before they become ready.
129 $ kubectl get pod -l release=my-hdfs
131 NAME READY STATUS RESTARTS AGE
132 my-hdfs-client-c749d9f8f-d5pvk 1/1 Running 0 2m
133 my-hdfs-datanode-o7jia 1/1 Running 3 2m
134 my-hdfs-datanode-p5kch 1/1 Running 3 2m
135 my-hdfs-datanode-r3kjo 1/1 Running 3 2m
136 my-hdfs-journalnode-0 1/1 Running 0 2m
137 my-hdfs-journalnode-1 1/1 Running 0 2m
138 my-hdfs-journalnode-2 1/1 Running 0 1m
139 my-hdfs-namenode-0 1/1 Running 3 2m
140 my-hdfs-namenode-1 1/1 Running 3 2m
141 my-hdfs-zookeeper-0 1/1 Running 0 2m
142 my-hdfs-zookeeper-1 1/1 Running 0 2m
143 my-hdfs-zookeeper-2 1/1 Running 0 2m
146 Namenodes and datanodes are currently using the K8s `hostNetwork` so they can
147 see physical IPs of each other. If they are not using `hostNetowrk`,
148 overlay K8s network providers such as weave-net may mask the physical IPs,
149 which will confuse the data locality later inside namenodes.
151 Finally, test with the client pod:
154 $ _CLIENT=$(kubectl get pods -l app=hdfs-client,release=my-hdfs -o name | \
156 $ kubectl exec $_CLIENT -- hdfs dfsadmin -report
157 $ kubectl exec $_CLIENT -- hdfs haadmin -getServiceState nn0
158 $ kubectl exec $_CLIENT -- hdfs haadmin -getServiceState nn1
160 $ kubectl exec $_CLIENT -- hadoop fs -rm -r -f /tmp
161 $ kubectl exec $_CLIENT -- hadoop fs -mkdir /tmp
162 $ kubectl exec $_CLIENT -- sh -c \
163 "(head -c 100M < /dev/urandom > /tmp/random-100M)"
164 $ kubectl exec $_CLIENT -- hadoop fs -copyFromLocal /tmp/random-100M /tmp
169 Kerberos can be enabled by setting a few related options:
172 $ helm install -n my-hdfs charts/hdfs-k8s \
173 --set global.kerberosEnabled=true \
174 --set global.kerberosRealm=MYCOMPANY.COM \
175 --set tags.kerberos=true
178 This will launch all charts including the Kerberos server, which will become
179 ready pretty soon. However, HDFS daemon charts will be blocked as the deamons
180 require Kerberos service principals to be available. So we need to unblock
181 them by creating those principals.
183 First, create a configmap containing the common Kerberos config file:
188 _KDC=$(kubectl get pod -l app=hdfs-krb5,release=my-hdfs --no-headers \
189 -o name | cut -d/ -f2)
190 _run kubectl cp $_KDC:/etc/krb5.conf $_MY_DIR/tmp/krb5.conf
191 _run kubectl create configmap my-hdfs-krb5-config \
192 --from-file=$_MY_DIR/tmp/krb5.conf
195 Second, create the service principals and passwords. Kerberos requires service
196 principals to be host specific. Some HDFS daemons are associated with your K8s
197 cluster nodes' physical host names say kube-n1.mycompany.com, while others are
198 associated with Kubernetes virtual service names, for instance
199 my-hdfs-namenode-0.my-hdfs-namenode.default.svc.cluster.local. You can get
200 the list of these host names like:
203 $ _HOSTS=$(kubectl get nodes \
204 -o=jsonpath='{.items[*].status.addresses[?(@.type == "Hostname")].address}')
206 $ _HOSTS+=$(kubectl describe configmap my-hdfs-config | \
207 grep -A 1 -e dfs.namenode.rpc-address.hdfs-k8s \
208 -e dfs.namenode.shared.edits.dir |
210 sed -e "s/<value>//" \
213 -e "s/qjournal:\/\///" \
215 -e "s/:8485\/hdfs-k8s//")
218 Then generate per-host principal accounts and password keytab files.
221 $ _SECRET_CMD="kubectl create secret generic my-hdfs-krb5-keytabs"
222 $ for _HOST in $_HOSTS; do
223 kubectl exec $_KDC -- kadmin.local -q \
224 "addprinc -randkey hdfs/$_HOST@MYCOMPANY.COM"
225 kubectl exec $_KDC -- kadmin.local -q \
226 "addprinc -randkey HTTP/$_HOST@MYCOMPANY.COM"
227 kubectl exec $_KDC -- kadmin.local -q \
228 "ktadd -norandkey -k /tmp/$_HOST.keytab hdfs/$_HOST@MYCOMPANY.COM HTTP/$_HOST@MYCOMPANY.COM"
229 kubectl cp $_KDC:/tmp/$_HOST.keytab $_MY_DIR/tmp/$_HOST.keytab
230 _SECRET_CMD+=" --from-file=$_MY_DIR/tmp/$_HOST.keytab"
234 The above was building a command using a shell variable `SECRET_CMD` for
235 creating a K8s secret that contains all keytab files. Run the command to create
242 This will unblock all HDFS daemon pods. Wait until they become ready.
244 Finally, test the setup using the following commands:
247 $ _NN0=$(kubectl get pods -l app=hdfs-namenode,release=my-hdfs -o name | \
250 $ kubectl exec $_NN0 -- sh -c "(apt install -y krb5-user > /dev/null)" \
252 $ kubectl exec $_NN0 -- \
253 kinit -kt /etc/security/hdfs.keytab \
254 hdfs/my-hdfs-namenode-0.my-hdfs-namenode.default.svc.cluster.local@MYCOMPANY.COM
255 $ kubectl exec $_NN0 -- hdfs dfsadmin -report
256 $ kubectl exec $_NN0 -- hdfs haadmin -getServiceState nn0
257 $ kubectl exec $_NN0 -- hdfs haadmin -getServiceState nn1
258 $ kubectl exec $_NN0 -- hadoop fs -rm -r -f /tmp
259 $ kubectl exec $_NN0 -- hadoop fs -mkdir /tmp
260 $ kubectl exec $_NN0 -- hadoop fs -chmod 0777 /tmp
261 $ kubectl exec $_KDC -- kadmin.local -q \
262 "addprinc -randkey user1@MYCOMPANY.COM"
263 $ kubectl exec $_KDC -- kadmin.local -q \
264 "ktadd -norandkey -k /tmp/user1.keytab user1@MYCOMPANY.COM"
265 $ kubectl cp $_KDC:/tmp/user1.keytab $_MY_DIR/tmp/user1.keytab
266 $ kubectl cp $_MY_DIR/tmp/user1.keytab $_CLIENT:/tmp/user1.keytab
268 $ kubectl exec $_CLIENT -- sh -c "(apt install -y krb5-user > /dev/null)" \
271 $ kubectl exec $_CLIENT -- kinit -kt /tmp/user1.keytab user1@MYCOMPANY.COM
272 $ kubectl exec $_CLIENT -- sh -c \
273 "(head -c 100M < /dev/urandom > /tmp/random-100M)"
274 $ kubectl exec $_CLIENT -- hadoop fs -ls /
275 $ kubectl exec $_CLIENT -- hadoop fs -copyFromLocal /tmp/random-100M /tmp
280 ### Setting HostPath volume locations for datanodes
282 HDFS on K8s stores the file data on the local disks of the K8s cluster nodes
283 using K8s HostPath volumes. You may want to change the default locations. Set
284 global.dataNodeHostPath to override the default value. Note the option
285 takes a list in case you want to use multiple disks.
288 $ helm install -n my-hdfs charts/hdfs-k8s \
289 --set "global.dataNodeHostPath={/mnt/sda1/hdfs-data0,/mnt/sda1/hdfs-data1}"
292 ### Using an existing zookeeper quorum
294 By default, HDFS on K8s pulls in the zookeeper chart in the incubator remote
295 repo (https://kubernetes-charts-incubator.storage.googleapis.com/) as a
296 dependency and launhces zookeeper daemons. But your K8s cluster may already
297 have a zookeeper quorum.
299 It is possible to use the existing zookeeper. We just need set a few options
300 in the helm install command line. It should be something like:
303 $helm install -n my-hdfs charts/hdfs-k8s \
304 --set condition.subchart.zookeeper=false \
305 --set global.zookeeperQuorumOverride=zk-0.zk-svc.default.svc.cluster.local:2181,zk-1.zk-svc.default.svc.cluster.local:2181,zk-2.zk-svc.default.svc.cluster.local:2181
308 Setting `condition.subchart.zookeeper` to false prevents the uber-chart from
309 bringing in zookeeper as sub-chart. And the `global.zookeeperQuorumOverride`
310 option specifies the custom address for a zookeeper quorum. Use your
311 zookeeper address here.
313 ### Pinning namenodes to specific K8s cluster nodes
315 Optionally, you can attach labels to some of your k8s cluster nodes so that
316 namenodes will always run on those cluster nodes. This can allow your HDFS
317 client outside the Kubernetes cluster to expect stable IP addresses. When used
318 by those outside clients, Kerberos expects the namenode addresses to be stable.
321 $ kubectl label nodes YOUR-HOST-1 hdfs-namenode-selector=hdfs-namenode
322 $ kubectl label nodes YOUR-HOST-2 hdfs-namenode-selector=hdfs-namenode
325 You should add the nodeSelector option to the helm chart command:
328 $ helm install -n my-hdfs charts/hdfs-k8s \
329 --set hdfs-namenode-k8s.nodeSelector.hdfs-namenode-selector=hdfs-namenode \
333 ### Excluding datanodes from some K8s cluster nodes
335 You may want to exclude some K8s cluster nodes from datanodes launch target.
336 For instance, some K8s clusters may let the K8s cluster master node launch
337 a datanode. To prevent this, label the cluster nodes with
338 `hdfs-datanode-exclude`.
341 $ kubectl label node YOUR-CLUSTER-NODE hdfs-datanode-exclude=yes
344 ### Launching with a non-HA namenode
346 You may want non-HA namenode since it is the simplest possible setup.
347 Note this won't launch zookeepers nor journalnodes.
349 The single namenode is supposed to be pinned to a cluster host using a node
350 label. Attach a label to one of your K8s cluster node.
353 $ kubectl label nodes YOUR-CLUSTER-NODE hdfs-namenode-selector=hdfs-namenode-0
356 The non-HA setup does not even use persistent vlumes. So you don't even
357 need to prepare persistent volumes. Instead, it is using hostPath volume
358 of the pinned cluster node. So, just launch the chart while
359 setting options to turn off HA. You should add the nodeSelector option
360 so that the single namenode would find the hostPath volume of the same cluster
361 node when the pod restarts.
364 $ helm install -n my-hdfs charts/hdfs-k8s \
365 --set tags.ha=false \
366 --set tags.simple=true \
367 --set global.namenodeHAEnabled=false \
368 --set hdfs-simple-namenode-k8s.nodeSelector.hdfs-namenode-selector=hdfs-namenode-0
373 ## K8s secret containing Kerberos keytab files
375 The Kerberos setup creates a K8s secret containing all the keytab files of HDFS
376 daemon service princialps. This will be mounted onto HDFS daemon pods. You may
377 want to restrict access to this secret using k8s
378 [RBAC](https://kubernetes.io/docs/admin/authorization/rbac/), to minimize
379 exposure of the keytab files.
382 `Datanode` daemons run on every cluster node. They also mount k8s `hostPath`
383 local disk volumes. You may want to restrict access of `hostPath`
384 using `pod security policy`.
385 See [reference](https://github.com/kubernetes/examples/blob/master/staging/podsecuritypolicy/rbac/README.md))
389 Many charts are using public Hadoop docker images hosted by
390 [uhopper](https://hub.docker.com/u/uhopper/).