3 [Horovod](https://eng.uber.com/horovod/) is a distributed training framework for TensorFlow, and it's provided by UBER. The goal of Horovod is to make distributed Deep Learning fast and easy to use. And it provides [Horovod in Docker](https://github.com/uber/horovod/blob/master/docs/docker.md) to streamline the installation process.
7 This chart bootstraps Horovod which is a Distributed TensorFlow Framework on a Kubernetes cluster using the Helm Package Manager. It deploys Horovod workers as statefulsets, and the Horovod master as a job, then discover the host list automatically.
11 - Kubernetes cluster v1.8+
15 You can use the dockerfile image provided along with this package. The benefit of this dockerfile is it contains many additional packages that the data science engineers usually require like spark, tensorflow, pytorch, matplotlib, nltk,
18 Before building the docker image, first build and make a Spark distribution following the instructions in http://spark.apache.org/docs/latest/building-spark.html
19 If this docker file is being used in the context of building your images from a Spark distribution, the docker build command should be invoked from the top level directory of the Spark distribution. E.g.:
22 docker build -t spark:latest -f kubernetes/dockerfiles/spark/ubuntu18.04/Dockerfile .
25 Once you build the spark image, go inside the spark package and place the file "keras_mnist_advanced_modified.py" in the dirctory: examples/src/main/python/tensorflow/. Create the 'tensorflow' directory if it doesnt exists.
26 We do this because we the file keras_mnist_advanced_modified.py is optimized for CPU running and we want this file to be automatically present in the final docker image that we build.
29 docker build -t spark-tf-keras-horovod-pytorch:latest -f kubernetes/dockerfiles/spark/ubuntu18.04/Dockerfile .
36 export SSH_KEY_DIR=`mktemp -d`
38 yes | ssh-keygen -N "" -f id_rsa
41 ## Create the values.yaml
43 To run Horovod with GPU, you can create `values.yaml` like below
46 # cat << EOF > ~/values.yaml
51 $(cat $SSH_KEY_DIR/id_rsa | sed 's/^/ /g')
54 $(cat $SSH_KEY_DIR/id_rsa.pub | sed 's/^/ /g')
59 repository: uber/horovod
60 tag: 0.12.1-tf1.8.0-py3.5
63 repository: uber/horovod
64 tag: 0.12.1-tf1.8.0-py3.5
66 - "mpirun -np 3 --hostfile /horovod/generated/hostfile --mca orte_keep_fqdn_hostnames t --allow-run-as-root --display-map --tag-output --timestamp-output sh -c '/opt/conda/envs/tf_env/bin/python /opt/spark/examples/src/main/python/tensorflow/keras_mnist_advanced_modified.py'"
70 For most cases, the overlay network impacts the Horovod performance greatly, so we should apply `Host Network` solution. To run Horovod with Host Network and GPU, you can create `values.yaml` like below
74 # cat << EOF > ~/values.yaml
82 $(cat $SSH_KEY_DIR/id_rsa | sed 's/^/ /g')
85 $(cat $SSH_KEY_DIR/id_rsa.pub | sed 's/^/ /g')
91 repository: uber/horovod
92 tag: 0.12.1-tf1.8.0-py3.5
95 repository: uber/horovod
96 tag: 0.12.1-tf1.8.0-py3.5
98 - "mpirun -np 3 --hostfile /horovod/generated/hostfile --mca orte_keep_fqdn_hostnames t --allow-run-as-root --display-map --tag-output --timestamp-output sh -c '/opt/conda/envs/tf_env/bin/python /opt/spark/examples/src/main/python/tensorflow/keras_mnist_advanced_modified.py'"
103 NOTE: A sample values.yaml is provided for reference. After adding the above changes, we should have a values.yml similar to that.
106 > notice: the difference is that you should set `useHostNetwork` as true, then set another ssh port rather than `22`
108 ## Installing the Chart
110 To install the chart with the release name `mnist`:
113 $ helm install --values ~/values.yaml --name mnist stable/horovod
116 ## Uninstalling the Chart
118 To uninstall/delete the `mnist` deployment:
124 The command removes all the Kubernetes components associated with the chart and
127 ## Upgrading an existing Release to a new major version
128 A major chart version change (like v1.2.3 -> v2.0.0) indicates that there is an
129 incompatible breaking change needing manual actions.
132 This version removes the `chart` label from the `spec.selector.matchLabels`
133 which is immutable since `StatefulSet apps/v1beta2`. It has been inadvertently
134 added, causing any subsequent upgrade to fail. See https://github.com/helm/charts/issues/7726.
136 In order to upgrade, delete the Horovod StatefulSet before upgrading, supposing your Release is named `my-release`:
139 $ kubectl delete statefulsets.apps --cascade=false my-release
144 The following table lists the configurable parameters of the Horovod
145 chart and their default values.
147 | Parameter | Description | Default |
148 |-----------|-------------|---------|
149 | `useHostNetwork` | Host network | `false` |
150 | `ssh.port` | The ssh port | `22` |
151 | `ssh.useSecrets` | Determine if using the secrets for ssh | `false` |
152 | `worker.number`| The worker's number | `5` |
153 | `worker.image.repository` | horovod worker image | `uber/horovod` |
154 | `worker.image.pullPolicy` | `pullPolicy` for the worker | `IfNotPresent` |
155 | `worker.image.tag` | `tag` for the worker | `0.12.1-tf1.8.0-py3.5` |
156 | `resources`| pod resource requests & limits| `{}`|
157 | `worker.env` | worker's environment variables | `{}` |
158 | `master.image.repository` | horovod master image | `uber/horovod` |
159 | `master.image.tag` | `tag` for the master | `0.12.1-tf1.8.0-py3.5` |
160 | `master.image.pullPolicy` | image pullPolicy for the master image| `IfNotPresent` |
161 | `master.args` | master's args | `{}` |
162 | `master.env` | master's environment variables | `{}` |