src/main/resources/docker-compose/README.md

   1 [![Docker Pulls](https://img.shields.io/docker/pulls/wurstmeister/kafka.svg)](https://hub.docker.com/r/wurstmeister/kafka/)
   2 [![Docker Stars](https://img.shields.io/docker/stars/wurstmeister/kafka.svg)](https://hub.docker.com/r/wurstmeister/kafka/)
   3 [![](https://badge.imagelayers.io/wurstmeister/kafka:latest.svg)](https://imagelayers.io/?images=wurstmeister/kafka:latest)
   4
   5 kafka-docker
   6 ============
   7
   8 Dockerfile for [Apache Kafka](http://kafka.apache.org/)
   9
  10 The image is available directly from https://registry.hub.docker.com/
  11
  12 ##Pre-Requisites
  13
  14 - install docker-compose [https://docs.docker.com/compose/install/](https://docs.docker.com/compose/install/)
  15 - modify the ```KAFKA_ADVERTISED_HOST_NAME``` in ```docker-compose.yml``` to match your docker host IP (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers.)
  16 - if you want to customise any Kafka parameters, simply add them as environment variables in ```docker-compose.yml```, e.g. in order to increase the ```message.max.bytes``` parameter set the environment to ```KAFKA_MESSAGE_MAX_BYTES: 2000000```. To turn off automatic topic creation set ```KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'```
  17
  18 ##Usage
  19
  20 Start a cluster:
  21
  22 - ```docker-compose up -d ```
  23
  24 Add more brokers:
  25
  26 - ```docker-compose scale kafka=3```
  27
  28 Destroy a cluster:
  29
  30 - ```docker-compose stop```
  31
  32 ##Note
  33
  34 The default ```docker-compose.yml``` should be seen as a starting point. By default each broker will get a new port number and broker id on restart. Depending on your use case this might not be desirable. If you need to use specific ports and broker ids, modify the docker-compose configuration accordingly, e.g. [docker-compose-single-broker.yml](https://github.com/wurstmeister/kafka-docker/blob/master/docker-compose-single-broker.yml):
  35
  36 - ```docker-compose -f docker-compose-single-broker.yml up```
  37
  38 ##Broker IDs
  39
  40 If you don't specify a broker id in your docker-compose file, it will automatically be generated (see [https://issues.apache.org/jira/browse/KAFKA-1070](https://issues.apache.org/jira/browse/KAFKA-1070). This allows scaling up and down. In this case it is recommended to use the ```--no-recreate``` option of docker-compose to ensure that containers are not re-created and thus keep their names and ids.
  41
  42
  43 ##Automatically create topics
  44
  45 If you want to have kafka-docker automatically create topics in Kafka during
  46 creation, a ```KAFKA_CREATE_TOPICS``` environment variable can be
  47 added in ```docker-compose.yml```.
  48
  49 Here is an example snippet from ```docker-compose.yml```:
  50
  51         environment:
  52           KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1"
  53
  54 ```Topic 1``` will have 1 partition and 3 replicas, ```Topic 2``` will have 1 partition and 1 replica.
  55
  56 ##Advertised hostname
  57
  58 You can configure the advertised hostname in different ways
  59
  60 1. explicitly, using ```KAFKA_ADVERTISED_HOST_NAME```
  61 2. via a command, using ```HOSTNAME_COMMAND```, e.g. ```HOSTNAME_COMMAND: "route -n | awk '/UG[ \t]/{print $$2}'"```
  62
  63 When using commands, make sure you review the "Variable Substitution" section in [https://docs.docker.com/compose/compose-file/](https://docs.docker.com/compose/compose-file/)
  64
  65 If ```KAFKA_ADVERTISED_HOST_NAME``` is specified, it takes presendence over ```HOSTNAME_COMMAND```
  66
  67 For AWS deployment, you can use the Metadata service to get the container host's IP:
  68 ```
  69 HOSTNAME_COMMAND=wget -t3 -T2 -qO-  http://169.254.169.254/latest/meta-data/local-ipv4
  70 ```
  71 Reference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html
  72
  73 ##Tutorial
  74
  75 [http://wurstmeister.github.io/kafka-docker/](http://wurstmeister.github.io/kafka-docker/)
  76
  77
  78