1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
5 .. _docs_vfw_edgex_multicloud_k8s:
9 vFW/Edgex with Multicloud Kubernetes Plugin: Setting Up and Configuration
10 -------------------------------------------------------------------------
15 This use case covers the deployment of vFW and Edgex HELM Charts in a Kubernetes
16 based cloud region via the multicloud-k8s plugin.
17 The multicloud-k8s plugin provides APIs to upload self-contained HELM Charts
18 that can be customized via the profile API and later installed in a particular
21 When the installation is complete (all the pods are either in running or
27 https://github.com/onap/multicloud-k8s/tree/master/kud/demo/firewall
29 EdgeXFoundry Helm Chart link:
30 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
32 https://github.com/onap/multicloud-k8s/tree/master/kud/tests/vnfs/edgex/helm/edgex
35 Create CSAR with Helm chart as an artifact
36 ------------------------------------------
38 The CSAR is a heat template package with Helm chart in it. The basic package
41 * an **environment file**
42 * a **base_dummy.yaml file** (example)
44 * a **tar.gz** file (of Helm chart)
46 These files must be zipped before onboarding.
47 One thing to pay much attention to is the naming convention which MUST
48 be followed while making the tgz.
50 **NOTE**: The Naming convention is for the helm chart tgz file.
52 **Naming convention follows the format:**
54 <free format string>\_\ ***cloudtech***\ \_<technology>\_<subtype>.extension
56 1. *Cloudtech:* is a fixed pattern and should not be changed if not
58 2. *Technology:* k8s, azure, aws
59 3. *Subtype*: charts, day0, config template
60 4. *Extension*: zip, tgz, csar
62 **NOTE**: The .tgz file must be a tgz created from the top level helm chart
63 folder. I.e. a folder that contains a Chart.yaml file in it.
64 For vFW use case the content of tgz file must look as follows:
68 $ helm package firewall
70 $ tar -tf firewall-0.1.0.tgz
74 firewall/templates/onap-private-net.yaml
75 firewall/templates/_helpers.tpl
76 firewall/templates/protected-private-net.yaml
77 firewall/templates/deployment.yaml
78 firewall/templates/unprotected-private-net.yaml
80 firewall/charts/sink/.helmignore
81 firewall/charts/sink/Chart.yaml
82 firewall/charts/sink/templates/configmap.yaml
83 firewall/charts/sink/templates/_helpers.tpl
84 firewall/charts/sink/templates/service.yaml
85 firewall/charts/sink/templates/deployment.yaml
86 firewall/charts/sink/values.yaml
87 firewall/charts/packetgen/.helmignore
88 firewall/charts/packetgen/Chart.yaml
89 firewall/charts/packetgen/templates/_helpers.tpl
90 firewall/charts/packetgen/templates/deployment.yaml
91 firewall/charts/packetgen/values.yaml
94 An example of the contents inside a heat template package can be found hereafter.
99 MANIFEST.json base_dummy.env base_dummy.yaml
100 vfw_cloudtech_k8s_charts.tgz vfw_k8s_demo.zip
106 Key thing is note the addition of cloud artifact
110 type: "CLOUD_TECHNOLOGY_SPECIFIC_ARTIFACTS"
117 "file": "base_dummy.yaml",
122 "file": "base_dummy.env",
128 "file": "vfw_cloudtech_k8s_charts.tgz",
129 "type": "CLOUD_TECHNOLOGY_SPECIFIC_ARTIFACTS"
137 It is an example of the minimal HEAT template.
141 ##==================LICENSE_START========================================
143 ## Copyright (C) 2019 Intel Corporation
144 ## SPDX-License-Identifier: Apache-2.0
146 ##==================LICENSE_END===========================================
148 heat_template_version: 2016-10-14
149 description: Heat template to deploy dummy VNF
155 description: Dummy name
159 label: id of vnommand to read (GET) Definition
160 description: Provided by ONAP
165 description: Provided by ONAP
170 description: Provided by ONAP
174 label: Image name or ID
175 description: Dummy image name
180 description: Dummy flavor
184 type: OS::Nova::Server
186 name: { get_param: dummy_name_0 }
187 image: { get_param: dummy_image_name }
188 flavor: { get_param: dummy_flavor_name } metadata: { vnf_name: { get_param: vnf_name }, vnf_id: { get_param: vnf_id }, vf_module_id: { get_param: vf_module_id }}
196 vnf_id: PROVIDED_BY_ONAP
197 vnf_name: PROVIDED_BY_ONAP
198 vf_module_id: PROVIDED_BY_ONAP
199 dummy_name_0: dummy_1_0
200 dummy_image_name: dummy
201 dummy_flavor_name: dummy.default
206 For onboarding instructions please refer to steps 4-9 from
207 `vFWCL instantiation, testing and debuging wiki page <https://wiki.onap.org/display/DW/vFWCL+instantiation%2C+testing%2C+and+debuging>`_.
209 Steps for installing KUD Cloud
210 ------------------------------
212 Follow the link to install KUD Kubernetes Deployment. KUD contains all the
213 packages required for running vfw use case.
215 Kubernetes Baremetal deployment instructions can be found in
216 `Kubernetes Baremetal deployment setup instructions wiki page <https://wiki.onap.org/display/DW/Kubernetes+Baremetal+deployment+setup+instructions>`_
218 REGISTER KUD CLOUD REGION with K8s-Plugin
219 -----------------------------------------
221 You must declare the KUD as a new cloud region in ONAP thanks to ONAP multicloud
224 POST connectivity info
225 ~~~~~~~~~~~~~~~~~~~~~~
227 Create a the post.json file as follows:
232 "cloud-region" : "<name>", // Must be unique
233 "cloud-owner" : "<owner>",
234 "other-connectivity-list" : {
237 Then send the POST HTTP request as described below:
241 curl -i -F "metadata=<post.json;type=application/json" -F file=@
242 /home/ad_kkkamine/.kube/config -X POST http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info
244 GET Connectivity Info
245 ~~~~~~~~~~~~~~~~~~~~~~
249 curl -i -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info/{name}
252 DELETE Connectivity Info
253 ~~~~~~~~~~~~~~~~~~~~~~~~
257 curl -i -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info/{name}
260 UPDATE/PUT Connectivity Info
261 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
265 curl -i -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info/{name}
267 Register KUD Cloud region with AAI
268 ----------------------------------
270 A tenant must be added to the k8s cloud region.
271 The 'easy' way is to have the ESR information (in step 1 of cloud
272 registration) pointing to a real OpenStack tenant (e.g. the OOF tenant in
273 the lab where we tested).
275 This will cause multicloud to add the tenant to the k8s cloud region and
276 then, similarly to #10 in the
277 `vFW HPA casablanca official documentation <https://onap.readthedocs.io/en/casablanca/submodules/integration.git/docs/docs_vfwHPA.html#docs-vfw-hpa>`_,
278 the service-subscription can be added to that object.
280 **NOTE:** use same name cloud-region and cloud-owner name
282 An example is shown below for K8s cloud but following the steps 1,2,3
284 `Multicloud Windriver Plugin documentation <https://docs.onap.org/projects/onap-multicloud-framework/en/latest/multicloud-plugin-windriver/UserGuide-MultiCloud-WindRiver-TitaniumCloud.html?highlight=multicloud>`_.
285 The sample input below is for k8s cloud type.
287 **Step 1**: Cloud Registration/ Create a cloud region to represent the instance
289 Note: the highlighted part of the body refers to an existing OpenStack
290 tenant (OOF in this case). It is provided as an illustration and must be adapted
291 according to your configuration.
295 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v13/cloud-infrastructure/cloud-regions/cloud-region/k8scloudowner4/k8sregionfour
297 "cloud-owner": "k8scloudowner4",
298 "cloud-region-id": "k8sregionfour",
300 "owner-defined-type": "t1",
301 "cloud-region-version": "1.0",
302 "complex-name": "clli1",
303 "cloud-zone": "CloudZone",
304 "sriov-automation": false,
305 "cloud-extra-info":"{\"openstack-region-id\":\"k8sregionthree\"}",
306 "esr-system-info-list": {
309 "esr-system-info-id": "55f97d59-6cc3-49df-8e69-926565f00066",
310 "service-url": "http://10.12.25.2:5000/v3",
312 "password": "onapdemo",
313 "system-type": "VIM",
314 "ssl-insecure": true,
315 "cloud-domain": "Default",
316 "default-tenant": "OOF",
317 "tenant-id": "6bbd2981b210461dbc8fe846df1a7808",
318 "system-status": "active"
324 **Step 2:** Add a complex to the cloud
326 Adding an already existing complex is enough. You do not need to create new ones.
330 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v13/cloud-infrastructure/cloud-regions/cloud-region/k8scloudowner4/k8sregionfour/relationship-list/relationship
332 "related-to": "complex",
333 "related-link": "/aai/v13/cloud-infrastructure/complexes/complex/clli1",
334 "relationship-data": [
336 "relationship-key": "complex.physical-location-id",
337 "relationship-value": "clli1"
342 **Step 3:** Trigger the Multicloud plugin registration process
346 POST http://{{MSB_IP}}:{{MSB_PORT}}/api/multicloud-titaniumcloud/v1/k8scloudowner4/k8sregionfour/registry
349 This step allws the registration of the K8S cloud with Multicloud. It also
350 reaches out and adds tenant information to the cloud (see example below, you
351 will see all kinds of flavor, image information that are associated with the OOF
354 If you did not follow the procedure described above then you will have to connect
355 on AAI point and manually add a tenant to the cloud region.
357 Once the tenant declared, you can add the service-subscription to it:
359 **Step 4:** Create a Service Type
363 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v13/service-design-and-creation/services/service/vfw-k8s
365 "service-description": "vfw-k8s",
366 "service-id": "vfw-k8s"
369 Add subscription service info to the service type of the customer.
373 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v16/business/customers/customer/Demonstration/service-subscriptions/service-subscription/vfw-k8s
375 "service-type": "vfw-k8s"
378 Add Service-Subscription to the tenant. The parameter resource-version is a
383 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v16/cloud-infrastructure/cloud-regions/cloud-region/k8scloudowner4/k8sregionfour/tenants/tenant/6bbd2981b210461dbc8fe846df1a7808?resource-version=1559345527327
385 "tenant-id": "6bbd2981b210461dbc8fe846df1a7808",
386 "tenant-name": "OOF",
387 "resource-version": "1559345527327",
388 "relationship-list": {
391 "related-to": "service-subscription",
392 "relationship-label": "org.onap.relationships.inventory.Uses",
393 "related-link": "/aai/v13/business/customers/customer/Demonstration/service-subscriptions/service-subscription/vfw-k8s",
394 "relationship-data": [
396 "relationship-key": "customer.global-customer-id",
397 "relationship-value": "Demonstration"
400 "relationship-key": "service-subscription.service-type",
401 "relationship-value": "vfw-k8s"
412 Once the cloud region is properly declared, it is possible to onboard the service
413 in the SDC and triggers a distribution from the SDC to the main ONAP components.
414 SO and other services are notified. A sdc listener is also getting the
415 distribution information in the multicloud sidecar.
416 When distribution happens it takes tar.gz file and uploads it to k8s plugin.
418 Create Profile Manually
419 -----------------------
421 K8s-plugin artifacts start in the form of Definitions. These are nothing
422 but Helm Charts wrapped with some metadata about the chart itself. Once
423 the Definitions are created, some profiles can be created. Finally it is
424 possible to customize the definition and instantiate it in the targeted
427 **NOTE:** Refer this link_ for complete API lists and documentation:
429 .. _link : https://wiki.onap.org/display/DW/MultiCloud+K8s-Plugin-service+API
431 The profile consists in:
433 * the **manifest.yaml**. It contains the details for the profile
434 * a **HELM** values override yaml file: It can have any name as long as it matches
435 the corresponding entry in the **manifest.yaml**
436 * Additional files organized in a folder structure: all these files should have
437 a corresponding entry in **manifest.yaml** file
439 Create a Profile Artifact
440 ~~~~~~~~~~~~~~~~~~~~~~~~~
444 > cd multicloud-k8s/kud/tests/vnfs/testrb/helm/profile
450 testfol/subdir/deployment.yaml
452 # Create profile tar.gz
454 > tar -cf profile.tar *
456 > mv profile.tar.gz ../
458 The manifest file contains the following parameters:
465 values: "values_override.yaml"
467 - filepath: testfol/subdir/deployment.yaml
468 chartpath: vault-consul-dev/templates/deployment.yaml
470 **Note:** values: "values\_override.yaml" can **be** empty **file** **if**
471 you are creating **a** dummy **profile**
473 **Note:** A dummy profile does not need any customization. The following is
474 optional in the manifest file.
479 - filepath: testfol/subdir/deployment.yaml
480 chartpath: vault-consul-dev/templates/deployment.yaml
482 The name of the Definition is retrived from SDC service distribution stage.
484 Retrieve the definition name
485 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
487 Execute the following command on the ONAP K8s Rancher host to read the definition
488 name and its version:
492 kubectl logs -n onap `kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep multicloud-k8s | head -1` -c multicloud-k8s
494 From the output read the name of the definition which is "rb-name" and
495 "rb-version" respectively
499 127.0.0.1 - - [15/Jul/2019:07:56:21 +0000] "POST /v1/rb/definition/test-rbdef/1/content HTTP/1.1"
504 With this information, it is possible to upload the profile with the following JSON data
509 "rb-name": "test-rbdef",
511 "profile-name": "p1",
512 "release-name": "r1", //If release-name is not provided, profile-name will be used
513 "namespace": "testnamespace1",
514 "kubernetes-version": "1.13.5"
522 curl -i -d @create_rbprofile.json -X POST http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile
526 UPLOAD artifact for Profile
527 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
531 curl -i --data-binary @profile.tar.gz -X POST http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile/p1/content
539 curl -i http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile
541 curl -i http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile/p1
549 curl -i -X DELETE http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile/p1
555 Instantiation is done by SO. SO then talks to Multi Cloud-broker via MSB
556 and that in turn looks up the cloud region in AAI to find the endpoint.
557 If k8sregion one is properly registered in AAI (SO check),
558 then the broker will know that it needs to talk to k8s-plugin based on
559 the type of the registration.
561 Instantiate the created Profile via the following REST API
562 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
566 Using the following JSON:
568 "cloud-region": "kud",
569 "profile-name": "p1",
570 "rb-name":"test-rbdef",
576 **NOTE**: Make sure that the namespace is already created before instantiation.
578 Instantiate the profile with the ID provided above
580 Instantiate a Profile
581 ~~~~~~~~~~~~~~~~~~~~~
585 curl -d @create_rbinstance.json http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/instance
588 The command shall return the following JSON
596 "profile-name": "profile1",
597 "cloud-region": "kud",
598 "namespace": "testns",
619 "Version": "v1beta1",
620 "Kind": "StatefulSet"
622 "Name": "profile1-mongo"
627 Delete Instantiated Kubernetes resources
628 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
630 The **id** field from the returned JSON can be used to **DELETE** the
631 resources created in the previous step. This executes a Delete operation
632 using the Kubernetes API.
636 curl -X DELETE http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/instance/ZKMTSaxv
639 GET Instantiated Kubernetes resources
640 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
642 The **id** field from the returned JSON can be used to **GET** the
643 resources created in the previous step. This executes a get operation
644 using the Kubernetes API.
648 curl -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/instance/ZKMTSaxv
651 `*\ https://github.com/onap/oom/blob/master/kubernetes/multicloud/resources/config/provider-plugin.json <https://github.com/onap/oom/blob/master/kubernetes/multicloud/resources/config/provider-plugin.json>`_
653 Create User parameters
654 ~~~~~~~~~~~~~~~~~~~~~~
656 We need to create parameters that ultimately get translated as:
663 "attribute_name": "definition-name",
664 "attribute_value": "edgex"
667 "attribute_name": "definition-version",
668 "attribute_value": "v1"
671 "attribute_name": "profile-name",
672 "attribute_value": "profile1"