2 ======================================
4 This is the repository for VES Collector for Open DCAE.
6 Virtual Event Streaming (VES) Collector is RESTful collector for processing JSON messages into DCAE. The collector verifies the source (when authentication is enabled) and validates the events against VES schema before distributing to DMAAP MR topics for downstream system to subscribe. The VESCollector also provides configurable event transformation function and event distribution to DMAAP MR topics.
8 The collector supports individual events or eventbatch posted to collector end-point(s) and post them to interface/bus for other application to subscribe.
11 ### Build Instructions
13 This project is organized as a mvn project and has "org.onap.dcaegen2" as parent project. The build generate a jar and package into docker container.
16 git clone ssh://vv770d@gerrit.onap.org:29418/dcaegen2/collectors/ves
21 VES Collector is a Spring Boot application
27 Build the image (it will go into your local docker repository)
32 Run the image using docker
34 docker run -p 8080:8080 -p 8443:8443 <container_id>
37 Run the image using docker-compose.yml
41 ### Developer mode - run application from IDE
43 To connect with "real" Consul instance you need to activate developer mode during starting VES application from IDE.
44 Attention: Without developer mode (default mode) Ves started from IDE will not use Consul.
46 1. Configure host mapping
48 For Linux: In host file add mapping for config-binding-service
51 SET_HERE_IP config-binding-service
53 2. At lab open port for config-binding-service
55 - Get basic information about config-binding-service
58 ubuntu@onap-7607-rke-node:~$ kubectl -n onap get services | grep config-binding-service
59 config-binding-service ClusterIP 10.43.227.68 <none> 10000/TCP,10443/TCP 6d2h
61 - Edit config-binding-service to change ClusterIP to NodePort to expose port
64 kubectl -n onap edit service config-binding-service
67 - Get information about opened port for config-binding-service
70 ubuntu@onap-7607-rke-node:~$ kubectl -n onap get services | grep config-binding-service
71 config-binding-service NodePort 10.43.227.68 <none> 10000:31029/TCP,10443:32719/TCP 6d2h
74 3. Run application with properties
76 -DdevMode=true -DcbsPort=31029
78 ### Generate auth credential
80 Library to generate new cryptographic password is stored in dcaegen2/sdk -"security/crypt-password"
82 or download artifact from:
84 https://nexus.onap.org/#nexus-search;quick~crypt-password
88 java -jar crypt-password-<version>.jar password_to_crypt
91 ### Mechanism to validate Certificate Subject DN value
92 When application is in certOnly or certBasicAuth mode then certificates are also validated by regexp in certSubjectMatcher.properties, only SubjectDn field in certificate description are checked.
93 Default regexp value is .* means that we approve all SubjectDN values.
95 ### Environment variables in Docker Container
96 Most of the configuration of how VESCollector should be started and managed is done through environment variables.
97 Some of them are set during the image build process and some of them are defined manually or by
98 a particular deployment system.
100 Variables set manually / coming from deployment system:
102 - DMAAPHOST - should contain an address to DMaaP, so that event publishing can work
103 - CONFIG_BINDING_SERVICE - should be a name of CBS
104 - CONFIG_BINDING_SERVICE_SERVICE_PORT - should be an http port of CBS
105 - HOSTNAME - should be a name of VESCollector application as it is registered in CBS catalog
107 ### Docker file system layout
108 The main directory where all code resides in docker container
109 looks like this and is located in /opt/app/VESCollector
111 <host>:/opt/app/VESCollector# ls
112 Dockerfile bin etc lib logs specs
114 - bin contains sh scripts (path here is denoted by env var $SCRIPTS_PATH)
115 - etc contains various application configuration, most notably it reflects 'etc' directory from repository
116 - lib contains all libraries that are pulled into the app during maven build
117 - logs contains all application logs, especially collector.log file which is a main log file denoted by $MAIN_LOG_FILE variable
118 - specs contains json schemas specs for ves-collector
120 ## Managing application in Docker container
121 Scripts directory contain .sh scripts that are used to start & stop & configure the VESCollector application
122 inside the docker image.
123 These scripts are packaged inside the docker image by a mvn assembly & docker plugins.
125 ## How the application starts inside container
126 General flow goes like this
127 - Docker image is build, and it points docker-entry.sh as the entrypoint.
128 - Docker-entry point, depending on the deployment type,
129 configures a bunch of things and starts the application in a separate process
130 and loops indefinitely to hold the docker container process.
133 For R1 - image/version pushed to nexus3
135 nexus3.onap.org:10003/snapshots/onap/org.onap.dcaegen2.collectors.ves.vescollector:<version>
140 VESCollector in DCAE will be deployed as mS via DCAEGEN2 controller. A blueprint will be generated (CLAMP/SDC) which will fetch the docker image and install on the dockerhost identified. VESCollector on startup will query the configbindingService for updated configuration and starts the service. When configuration change is detected by DCAEGEN2 controller (via policy flow) - then contoller will notify Collector to fetch new configuration again.
142 For testing purpose, the docker image includes preset configuration which can be ran without DCAEGEN2 platform.
146 ### Consul - Dynamic configuration
149 Application properties like /etc/collector.properties and Dmaap configuration /etc/DmaapConfig.json are updated frequently by configuration stored in Consul(CBS)
150 http://<kubernetes_host_ip>:30270/ui/#/dc1/kv/<vescollector_SCN>
151 Configuration stored in Consul have bigger priority and always will override local configuration so all configuration modification should be done using Consul update on corresponding kv store.
152 Frequently how often configuration will be fetch from Consul server is manageable in /etc/collector.properties property "collector.dynamic.config.update.frequency={time in minutes}".
153 To fetch configuration from Consul, VES collector uses CBS client from DCAE SDK.
155 Sample configuration of VESCollector K-V store can be found under /dpo/data-formats/ConsulConfig.json
157 ### How to send event locally
159 1. In /etc/hosts add: 127.0.0.1 onap-dmaap
160 2. Go into: ./src/test/resources/dmaap-msg-router
161 3. Run: docker-compose -f message-router-compose.yml up -d
162 4. Run ves application
163 5. Now you can send events to ves
164 6. Check topics on message-router: curl http://127.0.0.1:3904/topics
168 For R1 as only measurement and faults are expected in ONAP, configuration are preset currently sto support these two topics only.
172 1) Get the VESCollector image from Nexus
173 docker pull nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:latest
174 2) Start the container (change the DMAAPHOST environment value to running DMAAP instance host)
175 docker run -d -p 8080:8080/tcp -p 8443:8443/tcp -P -e DMAAPHOST='10.0.0.174' nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1
176 3) Login into container and tail /opt/app/VESCollector/logs/collector.log
177 4) Simulate event into VEScollector (can be done from different vm or same)
178 curl -i -X POST -d @measurement.txt --header "Content-Type: application/json" https://localhost:8443/eventListener/v5 -k
179 or curl -i -X POST -d @measurement.txt --header "Content-Type: application/json" http://localhost:8080/eventListener/v5 -k
180 Note: If DMAAPHOST provided is invalid, you will see exception around publish on the collector.logs (collector queues and attempts to resend the event hence exceptions reported will be periodic). If you don’t want to see the error, publish to dmaap can be disabled by changing either “collector.dmaap.streamid” on etc/collector.properties OR by modifying the “name” defined on etc/DmaapConfig.json.
182 Any changes to property within container requires collector restart
183 cd /opt/app/VESCollector/
184 ./bin/appController.sh stop
185 ./bin/appController.sh start
187 5) If DMAAP instance (and DMAAPHOST passed during VESCollector startup) and VES input is valid, then events will be pushed to below topics depending on the domain
188 Fault :http://<dmaaphost>:3904/events/unauthenticated.SEC_FAULT_OUTPUT
189 Measurement : http://<dmaaphost>:3904/events/unauthenticated.SEC_MEASUREMENT_OUTPUT
190 6) When test is done – do ensure to remove the container (docker rm -f <containerid>) to avoid port conflict
193 Authentication is set by default to "noauth" (via auth.method property) on the container; below are the steps for enabling HTTPS/authentication for VESCollector.
195 1) Login to the container
196 2) Open /opt/app/VESCollector/etc/collector.properties and edit below properties
197 a) Comment below property (with authentication enabled, standard http should be disabled)
198 collector.service.port=8080
199 b) Enable basic-authentication
200 auth.method=basicAuth
201 Note: The actual credentials is stored part of header.authlist parameter. This is list of userid,password values. Default configuration has below set
202 sample1,$2a$10$pgjaxDzSuc6XVFEeqvxQ5u90DKJnM/u7TJTcinAlFJVaavXMWf/Zi|vdnsagg,$2a$10$C45JhiRSY.qXTBfzWST3Q.AmwKlPRMc67c33O0U9hOH8KSGaweN4m
203 where password maps to same value as username.
204 Password is generated by crypt-password tool (https://nexus.onap.org/#nexus-search;quick~crypt-password)
205 3) Restart the collector
206 cd /opt/app/VESCollector
207 ./bin/appController.sh stop
208 ./bin/appController.sh start
209 4) Exit from container and ensure tcp port on VM is not hanging on finwait – you can execute “netstat -an | grep 8443” . If under FIN_WAIT2, wait for server to release.
210 5) Simulate via curl (Note - username/pwd will be required)
211 Example of successfull POST:
212 vv770d@osdcae-dev-16:~$ curl -i -u 'sample1:sample1' -X POST -d @volte.txt --header "Content-Type: application/json" https://localhost:8443/eventListener/v5 -k
214 Server: Apache-Coyote/1.1
215 X-Rathravane: ~ software is craft ~
216 Content-Type: application/json;charset=ISO-8859-1
218 Date: Thu, 21 Sep 2017 22:23:49 GMT
221 Example of authentication failure:
222 vv770d@osdcae-dev-16:~$ curl -i -X POST -d @volte.txt --header "Content-Type: application/json" https://localhost:8443/eventListener/v5 -k
223 HTTP/1.1 401 Unauthorized
224 Server: Apache-Coyote/1.1
225 X-Rathravane: ~ software is craft ~
226 Content-Type: application/json;charset=ISO-8859-1
228 Date: Thu, 21 Sep 2017 22:20:43 GMT
230 {"requestError":{"GeneralException":{"MessagID":"\"POL2000\"","text":"\"Unauthorized user\""}}}
232 Note: In general support for HTTPS also require certificate/keystore be installed on target VM with FS mapped into the container for VESCollector to load. For demo and testing purpose - a self signed certificate is included within docker build. When deployed via DCAEGEN2 platform - these configuration will be overridden dynamically to map to required path/certificate name. This will be exercised post R1 though.
235 A client's certificate verification is disabled on the container by default; below are the steps for enabling mutual TLS authentication for VESCollector.
237 1) Login to the container
238 2) Open /opt/app/VESCollector/etc/collector.properties and edit below properties
239 a) Comment below property (with authentication enabled, standard http should be disabled)
240 collector.service.port=8080
241 b) Enable a client's certificate verification
242 auth.method=certOnly (only certificate verification)
244 auth.method=certBasicAuth ( certificate verification with basic auth verification )
245 3) Restart the collector
246 cd /opt/app/VESCollector
247 ./bin/appController.sh stop
248 ./bin/appController.sh start
249 4) Exit from container and ensure tcp port on VM is not hanging on finwait – you can execute “netstat -an | grep 8443” . If under FIN_WAIT2, wait for server to release.
250 5) In order for VESCollector to accept a connection from a client, the client has to use TLS certificate signed by CA that is present in VESCollector truststore. If a default VESCollector truststore is used then a client's certificate may be generated using following steps:
251 a) Generate a client's private key
252 openssl genrsa -out client.key 2048
253 b) Create the signing
254 openssl req -new -key client.key -out client.csr
255 c) Create the client's certificate (CA key password should be obtained from [VESCollectorRepository]/certs/password)
256 openssl x509 -req -in client.csr -CA [VESCollectorRepository]/certs/rootCA.crt -CAkey [VESCollectorRepository]/certs/rootCA.key -CAcreateserial -out client.crt -days 500 -sha256
257 6) Simulate via curl (assuming that the certificate was created via step 5)
258 Example of successfull POST:
259 curl -i -X POST -d @event.json --header "Content-Type: application/json" https://localhost:8443/eventListener/v7 -k --cert client.crt --key client.key
263 Content-Type: application/json
265 Date: Wed, 21 Nov 2018 11:37:58 GMT
267 Example of authentication failure (without a client's certificate):
268 curl -i -X POST -d @event.json --header "Content-Type: application/json" https://localhost:8443/eventListener/v7 -k
269 curl: (35) error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate