ab90cebef3f0ee266e28f834506f910bccf37c91
[dcaegen2.git] / docs / sections / installation.rst
1 DCAE mS Installation
2 ====================
3
4 The below steps covers manual setup of DCAE VM’s and DCAE service
5 components.
6
7 VESCollector
8 ------------
9
10  
11
12 DCAE VES Collector can be configured on VM with ubuntu-16.04 image
13 (m1.small should suffice if this is only service) and 20Gb cinder
14 storage
15
16 1. Install docker
17
18   sudo apt-get update
19
20   sudo apt install `docker.io <http://docker.io/>`__
21
22 1. Pull the latest container from onap nexus
23
24     sudo docker login -u docker -p docker
25     `nexus.onap.org <http://nexus.onap.org/>`__:10001
26
27     sudo docker pull
28     `nexus.onap.org <http://nexus.onap.org/>`__:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1
29
30 1. Start the VESCollector with below command
31
32     sudo docker run -d --name vescollector -p 8080:8080/tcp -p
33     8443:8443/tcp -P -e DMAAPHOST='<dmaap IP>'
34     `nexus.onap.org <http://nexus.onap.org/>`__:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1
35
36      Note:  Change the dmaaphost to required DMAAP ip. To change the
37     dmaap information for a running container,  stop the active
38     container and rerun above command changing the dmaap IP.
39
40 1. Verification
41
42 i.  Check logs under container /opt/app/VESCollector/logs/collector.log
43         for errors
44
45 ii. If no active feed, you can simulate an event into collector via curl
46
47     curl -i  -X POST -d @<sampleves> --header "Content-Type:
48     application/json" http://localhost:8080/eventListener/v5 -k
49
50     Note: If DMAAPHOST provided is invalid, you will see exception
51     around publish on the collector.logs (collector queues and attempts
52     to resend the event hence exceptions reported will be periodic). 
53
54 i. Below two topic configuration are pre-set into this container.  When
55        valid DMAAP instance ip was provided and VES events are received,
56        the collector will post to below topics.
57
58     Fault -
59      http://<dmaaphost>:3904/events/unauthenticated.SEC\_FAULT\_OUTPUT
60
61     Measurement
62     -http://<dmaaphost>:3904/events/unauthenticated.SEC\_MEASUREMENT\_OUTPUT
63
64 VM Init
65 ~~~~~~
66
67 To address windriver server in-stability, the below **init.sh** script
68 was used to start the container on VM restart.  
69
70 +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
71 | #!/bin/sh                                                                                                                                                                                    |
72 |                                                                                                                                                                                              |
73 | sudo docker ps \| grep "vescollector"                                                                                                                                                        |
74 |                                                                                                                                                                                              |
75 | if [ $? -ne 0 ]; then                                                                                                                                                                        |
76 |                                                                                                                                                                                              |
77 |         sudo docker login -u docker -p docker nexus.onap.org:10001                                                                                                                           |
78 |                                                                                                                                                                                              |
79 |         sudo docker pull nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1                                                                                         |
80 |                                                                                                                                                                                              |
81 |         sudo docker rm -f vescollector                                                                                                                                                       |
82 |                                                                                                                                                                                              |
83 |         echo "Collector process not running - $(date)" >> /home/ubuntu/startuplog                                                                                                            |
84 |                                                                                                                                                                                              |
85 |         sudo docker run -d --name vescollector -p 8080:8080/tcp -p 8443:8443/tcp -P -e DMAAPHOST='10.12.25.96' nexus.onap.org:10001/onap/org.onap.dcaegen2.collectors.ves.vescollector:1.1   |
86 |                                                                                                                                                                                              |
87 | else                                                                                                                                                                                         |
88 |                                                                                                                                                                                              |
89 |         echo "Collector process running - $(date)" >> /home/ubuntu/startuplog                                                                                                                |
90 |                                                                                                                                                                                              |
91 | fi                                                                                                                                                                                           |
92 +==============================================================================================================================================================================================+
93 +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
94
95 This script was invoked via VM init script (rc.d).
96
97 ln -s /home/ubuntu/init.sh /etc/init.d/init.sh
98
99 sudo  update-rc.d init.sh start 2
100
101  
102
103 ThresholdCrossingAnalysis (TCA/CDAP)
104 ------------------------------------
105
106 The platform deploys CDAP as cluster and instantiates TCA. For the
107 manual setup, we will leverage the CDAP SDK docker container to deploy
108 TCA instances.  To setup TCA, choose VM with ubuntu-16.04 image,
109 m1.medium size and 50gb cinder volumes.
110
111 1. Install docker
112
113      sudo apt-get update
114
115      sudo apt install `docker.io <http://docker.io/>`__
116
117 1. Pull CDAP SDK container
118
119 sudo docker pull caskdata/cdap-standalone:4.1.2
120
121 1. Deploy and run the CDAP container
122
123     sudo docker run -d --name cdap-sdk-2 -p 11011:11011 -p 11015:11015
124     caskdata/cdap-standalone:4.1.2
125
126 1. Create Namespace on CDAP application
127
128 curl -X PUT http://localhost:11015/v3/namespaces/cdap_tca_hi_lo
129
130 1. Create TCA app preference file under ~ubuntu as below
131
132 +--------------------------------------------------------------------------------------------------------------------------------------------+
133 | {                                                                                                                                          |
134 |                                                                                                                                            |
135 |   "publisherContentType" : "application/json",                                                                                             |
136 |                                                                                                                                            |
137 |   "publisherHostName" : "10.12.25.96",                                                                                                     |
138 |                                                                                                                                            |
139 |   "publisherHostPort" : "3904",                                                                                                            |
140 |                                                                                                                                            |
141 |   "publisherMaxBatchSize" : "1",                                                                                                           |
142 |                                                                                                                                            |
143 |   "publisherMaxRecoveryQueueSize" : "100000",                                                                                              |
144 |                                                                                                                                            |
145 |   "publisherPollingInterval" : "20000",                                                                                                    |
146 |                                                                                                                                            |
147 |   "publisherProtocol" : "http",                                                                                                            |
148 |                                                                                                                                            |
149 |   "publisherTopicName" : "unauthenticated.DCAE\_CL\_OUTPUT",                                                                               |
150 |                                                                                                                                            |
151 |   "subscriberConsumerGroup" : "OpenDCAE-c1",                                                                                               |
152 |                                                                                                                                            |
153 |   "subscriberConsumerId" : "c1",                                                                                                           |
154 |                                                                                                                                            |
155 |   "subscriberContentType" : "application/json",                                                                                            |
156 |                                                                                                                                            |
157 |   "subscriberHostName" : "10.12.25.96",                                                                                                    |
158 |                                                                                                                                            |
159 |   "subscriberHostPort" : "3904",                                                                                                           |
160 |                                                                                                                                            |
161 |   "subscriberMessageLimit" : "-1",                                                                                                         |
162 |                                                                                                                                            |
163 |   "subscriberPollingInterval" : "20000",                                                                                                   |
164 |                                                                                                                                            |
165 |   "subscriberProtocol" : "http",                                                                                                           |
166 |                                                                                                                                            |
167 |   "subscriberTimeoutMS" : "-1",                                                                                                            |
168 |                                                                                                                                            |
169 |   "subscriberTopicName" : "unauthenticated.SEC\_MEASUREMENT\_OUTPUT",                                                                      |
170 |                                                                                                                                            |
171 |   "enableAAIEnrichment" : false,                                                                                                           |
172 |                                                                                                                                            |
173 |   "aaiEnrichmentHost" : "10.12.25.72",                                                                                                     |
174 |                                                                                                                                            |
175 |   "aaiEnrichmentPortNumber" : 8443,                                                                                                        |
176 |                                                                                                                                            |
177 |   "aaiEnrichmentProtocol" : "https",                                                                                                       |
178 |                                                                                                                                            |
179 |   "aaiEnrichmentUserName" : "DCAE",                                                                                                        |
180 |                                                                                                                                            |
181 |   "aaiEnrichmentUserPassword" : "DCAE",                                                                                                    |
182 |                                                                                                                                            |
183 |   "aaiEnrichmentIgnoreSSLCertificateErrors" : false,                                                                                       |
184 |                                                                                                                                            |
185 |   "aaiVNFEnrichmentAPIPath" : "/aai/v11/network/generic-vnfs/generic-vnf",                                                                 |
186 |                                                                                                                                            |
187 |   "aaiVMEnrichmentAPIPath" :  "/aai/v11/search/nodes-query",                                                                               |
188 |                                                                                                                                            |
189 |   "tca\_policy" : "{                                                                                                                       |
190 |                                                                                                                                            |
191 |         \\"domain\\": \\"measurementsForVfScaling\\",                                                                                      |
192 |                                                                                                                                            |
193 |         \\"metricsPerEventName\\": [{                                                                                                      |
194 |                                                                                                                                            |
195 |                 \\"eventName\\": \\"vFirewallBroadcastPackets\\",                                                                          |
196 |                                                                                                                                            |
197 |                 \\"controlLoopSchemaType\\": \\"VNF\\",                                                                                    |
198 |                                                                                                                                            |
199 |                 \\"policyScope\\": \\"DCAE\\",                                                                                             |
200 |                                                                                                                                            |
201 |                 \\"policyName\\": \\"DCAE.Config\_tca-hi-lo\\",                                                                            |
202 |                                                                                                                                            |
203 |                 \\"policyVersion\\": \\"v0.0.1\\",                                                                                         |
204 |                                                                                                                                            |
205 |                 \\"thresholds\\": [{                                                                                                       |
206 |                                                                                                                                            |
207 |                         \\"closedLoopControlName\\": \\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\\",                     |
208 |                                                                                                                                            |
209 |                         \\"version\\": \\"1.0.2\\",                                                                                        |
210 |                                                                                                                                            |
211 |                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.vNicUsageArray[\*].receivedTotalPacketsDelta\\",        |
212 |                                                                                                                                            |
213 |                         \\"thresholdValue\\": 300,                                                                                         |
214 |                                                                                                                                            |
215 |                         \\"direction\\": \\"LESS\_OR\_EQUAL\\",                                                                            |
216 |                                                                                                                                            |
217 |                         \\"severity\\": \\"MAJOR\\",                                                                                       |
218 |                                                                                                                                            |
219 |                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
220 |                                                                                                                                            |
221 |                 }, {                                                                                                                       |
222 |                                                                                                                                            |
223 |                         \\"closedLoopControlName\\": \\"ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a\\",                     |
224 |                                                                                                                                            |
225 |                         \\"version\\": \\"1.0.2\\",                                                                                        |
226 |                                                                                                                                            |
227 |                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.vNicUsageArray[\*].receivedTotalPacketsDelta\\",        |
228 |                                                                                                                                            |
229 |                         \\"thresholdValue\\": 700,                                                                                         |
230 |                                                                                                                                            |
231 |                         \\"direction\\": \\"GREATER\_OR\_EQUAL\\",                                                                         |
232 |                                                                                                                                            |
233 |                         \\"severity\\": \\"CRITICAL\\",                                                                                    |
234 |                                                                                                                                            |
235 |                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
236 |                                                                                                                                            |
237 |                 }]                                                                                                                         |
238 |                                                                                                                                            |
239 |         }, {                                                                                                                               |
240 |                                                                                                                                            |
241 |                 \\"eventName\\": \\"vLoadBalancer\\",                                                                                      |
242 |                                                                                                                                            |
243 |                 \\"controlLoopSchemaType\\": \\"VM\\",                                                                                     |
244 |                                                                                                                                            |
245 |                 \\"policyScope\\": \\"DCAE\\",                                                                                             |
246 |                                                                                                                                            |
247 |                 \\"policyName\\": \\"DCAE.Config\_tca-hi-lo\\",                                                                            |
248 |                                                                                                                                            |
249 |                 \\"policyVersion\\": \\"v0.0.1\\",                                                                                         |
250 |                                                                                                                                            |
251 |                 \\"thresholds\\": [{                                                                                                       |
252 |                                                                                                                                            |
253 |                         \\"closedLoopControlName\\": \\"ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3\\",                          |
254 |                                                                                                                                            |
255 |                         \\"version\\": \\"1.0.2\\",                                                                                        |
256 |                                                                                                                                            |
257 |                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.vNicUsageArray[\*].receivedTotalPacketsDelta\\",        |
258 |                                                                                                                                            |
259 |                         \\"thresholdValue\\": 300,                                                                                         |
260 |                                                                                                                                            |
261 |                         \\"direction\\": \\"GREATER\_OR\_EQUAL\\",                                                                         |
262 |                                                                                                                                            |
263 |                         \\"severity\\": \\"CRITICAL\\",                                                                                    |
264 |                                                                                                                                            |
265 |                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
266 |                                                                                                                                            |
267 |                 }]                                                                                                                         |
268 |                                                                                                                                            |
269 |         }, {                                                                                                                               |
270 |                                                                                                                                            |
271 |                 \\"eventName\\": \\"Measurement\_vGMUX\\",                                                                                 |
272 |                                                                                                                                            |
273 |                 \\"controlLoopSchemaType\\": \\"VNF\\",                                                                                    |
274 |                                                                                                                                            |
275 |                 \\"policyScope\\": \\"DCAE\\",                                                                                             |
276 |                                                                                                                                            |
277 |                 \\"policyName\\": \\"DCAE.Config\_tca-hi-lo\\",                                                                            |
278 |                                                                                                                                            |
279 |                 \\"policyVersion\\": \\"v0.0.1\\",                                                                                         |
280 |                                                                                                                                            |
281 |                 \\"thresholds\\": [{                                                                                                       |
282 |                                                                                                                                            |
283 |                         \\"closedLoopControlName\\": \\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\\",                          |
284 |                                                                                                                                            |
285 |                         \\"version\\": \\"1.0.2\\",                                                                                        |
286 |                                                                                                                                            |
287 |                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.additionalMeasurements[\*].arrayOfFields[0].value\\",   |
288 |                                                                                                                                            |
289 |                         \\"thresholdValue\\": 0,                                                                                           |
290 |                                                                                                                                            |
291 |                         \\"direction\\": \\"EQUAL\\",                                                                                      |
292 |                                                                                                                                            |
293 |                         \\"severity\\": \\"MAJOR\\",                                                                                       |
294 |                                                                                                                                            |
295 |                         \\"closedLoopEventStatus\\": \\"ABATED\\"                                                                          |
296 |                                                                                                                                            |
297 |                 }, {                                                                                                                       |
298 |                                                                                                                                            |
299 |                         \\"closedLoopControlName\\": \\"ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e\\",                          |
300 |                                                                                                                                            |
301 |                         \\"version\\": \\"1.0.2\\",                                                                                        |
302 |                                                                                                                                            |
303 |                         \\"fieldPath\\": \\"$.event.measurementsForVfScalingFields.additionalMeasurements[\*].arrayOfFields[0].value\\",   |
304 |                                                                                                                                            |
305 |                         \\"thresholdValue\\": 0,                                                                                           |
306 |                                                                                                                                            |
307 |                         \\"direction\\": \\"GREATER\\",                                                                                    |
308 |                                                                                                                                            |
309 |                         \\"severity\\": \\"CRITICAL\\",                                                                                    |
310 |                                                                                                                                            |
311 |                         \\"closedLoopEventStatus\\": \\"ONSET\\"                                                                           |
312 |                                                                                                                                            |
313 |                 }]                                                                                                                         |
314 |                                                                                                                                            |
315 |         }]                                                                                                                                 |
316 |                                                                                                                                            |
317 | }"                                                                                                                                         |
318 |                                                                                                                                            |
319 | }                                                                                                                                          |
320 +============================================================================================================================================+
321 +--------------------------------------------------------------------------------------------------------------------------------------------+
322
323      Note: Dmaap configuration are specified on this file on
324     publisherHostName and subscriberHostName. To be changed as
325     required\*\*
326
327 1. Copy below script to CDAP server (this gets latest image from nexus
328    and deploys TCA application) and execute it
329
330 +---------------------------------------------------------------------------------------------------------------------------------------------------------------+
331 | #!/bin/sh                                                                                                                                                     |
332 |                                                                                                                                                               |
333 | TCA\_JAR=dcae-analytics-cdap-tca-2.0.0.jar                                                                                                                    |
334 |                                                                                                                                                               |
335 | rm -f /home/ubuntu/$TCA\_JAR                                                                                                                                  |
336 |                                                                                                                                                               |
337 | cd /home/ubuntu/                                                                                                                                              |
338 |                                                                                                                                                               |
339 | wget https://nexus.onap.org/service/local/repositories/staging/content/org/onap/dcaegen2/analytics/tca/dcae-analytics-cdap-tca/2.0.0/$TCA\_JAR                |
340 |                                                                                                                                                               |
341 | if [ $? -eq 0 ]; then                                                                                                                                         |
342 |                                                                                                                                                               |
343 |         if [ -f /home/ubuntu/$TCA\_JAR ]; then                                                                                                                |
344 |                                                                                                                                                               |
345 |                 echo "Restarting TCA CDAP application using $TCA\_JAR artifact"                                                                               |
346 |                                                                                                                                                               |
347 |         else                                                                                                                                                  |
348 |                                                                                                                                                               |
349 |                 echo "ERROR: $TCA\_JAR missing"                                                                                                               |
350 |                                                                                                                                                               |
351 |                 exit 1                                                                                                                                        |
352 |                                                                                                                                                               |
353 |         fi                                                                                                                                                    |
354 |                                                                                                                                                               |
355 | else                                                                                                                                                          |
356 |                                                                                                                                                               |
357 |         echo "ERROR: $TCA\_JAR not found in nexus"                                                                                                            |
358 |                                                                                                                                                               |
359 |         exit 1                                                                                                                                                |
360 |                                                                                                                                                               |
361 | fi                                                                                                                                                            |
362 |                                                                                                                                                               |
363 | # stop programs                                                                                                                                               |
364 |                                                                                                                                                               |
365 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/stop                                      |
366 |                                                                                                                                                               |
367 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/stop                                     |
368 |                                                                                                                                                               |
369 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/stop                                              |
370 |                                                                                                                                                               |
371 | # delete application                                                                                                                                          |
372 |                                                                                                                                                               |
373 | curl -X DELETE http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca# delete artifact                                                          |
374 |                                                                                                                                                               |
375 | curl -X DELETE http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/artifacts/dcae-analytics-cdap-tca/versions/2.0.0# load artifact                         |
376 |                                                                                                                                                               |
377 | curl -X POST --data-binary @/home/ubuntu/$TCA\_JAR http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/artifacts/dcae-analytics-cdap-tcaecho# create app   |
378 |                                                                                                                                                               |
379 | curl -X PUT -d @/home/ubuntu/tca\_app\_config.json http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca# load preferences                     |
380 |                                                                                                                                                               |
381 | curl -X PUT -d @/home/ubuntu/tca\_app\_preferences.json http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/preferences# start programs      |
382 |                                                                                                                                                               |
383 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start                                     |
384 |                                                                                                                                                               |
385 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start                                    |
386 |                                                                                                                                                               |
387 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/startecho# get status of programs                 |
388 |                                                                                                                                                               |
389 | curl http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/status                                            |
390 |                                                                                                                                                               |
391 | curl http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/status                                           |
392 |                                                                                                                                                               |
393 | curl http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/statusecho                                                |
394 +===============================================================================================================================================================+
395 +---------------------------------------------------------------------------------------------------------------------------------------------------------------+
396
397 1. Verify TCA application and logs via CDAP GUI processes
398
399     The overall flow can be checked here
400
401 TCA Configuration Change
402 ~~~~~~~~~~~~~~~~~~~~~~~
403
404 Typical configuration changes include changing DMAAP host and/or Policy
405 configuration. If necessary, modify the file on step #5 and run the
406 script noted as step #6 to redeploy TCA with updated configuration.
407
408 VM Init
409 ~~~~~~
410
411 To address windriver server in-stability, the below **init.sh** script
412 was used to restart the container on VM restart.  This script was
413 invoked via VM init script (rc.d).
414
415 +------------------------------------------------------------------------------------------------------------------------------+
416 | #!/bin/sh                                                                                                                    |
417 |                                                                                                                              |
418 | #docker run -d --name cdap-sdk -p 11011:11011 -p 11015:11015 caskdata/cdap-standalone:4.1.2                                  |
419 |                                                                                                                              |
420 | sudo docker restart cdap-sdk-2                                                                                               |
421 |                                                                                                                              |
422 | sleep 30                                                                                                                     |
423 |                                                                                                                              |
424 | # start program                                                                                                              |
425 |                                                                                                                              |
426 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRPublisherWorker/start    |
427 |                                                                                                                              |
428 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/workers/TCADMaaPMRSubscriberWorker/start   |
429 |                                                                                                                              |
430 | curl -X POST http://localhost:11015/v3/namespaces/cdap\_tca\_hi\_lo/apps/dcae-tca/flows/TCAVESCollectorFlow/start            |
431 +==============================================================================================================================+
432 +------------------------------------------------------------------------------------------------------------------------------+
433
434  
435
436 This script was invoked via VM init script (rc.d).
437
438 ln -s /home/ubuntu/init.sh /etc/init.d/init.sh
439
440 sudo  update-rc.d init.sh start 2
441