1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
5 .. _docs_vfw_edgex_multicloud_k8s:
7 vFW/Edgex with Multicloud Kubernetes Plugin: Setting Up and Configuration
8 -------------------------------------------------------------------------
13 This use case covers the deployment of vFW and Edgex HELM Charts in a Kubernetes
14 based cloud region via the multicloud-k8s plugin.
15 The multicloud-k8s plugin provides APIs to upload self-contained HELM Charts
16 that can be customized via the profile API and later installed in a particular
19 When the installation is complete (all the pods are either in running or
25 https://github.com/onap/multicloud-k8s/tree/master/kud/demo/firewall
27 EdgeXFoundry Helm Chart link:
28 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
30 https://github.com/onap/multicloud-k8s/tree/master/kud/tests/vnfs/edgex/helm/edgex
33 Create CSAR with Helm chart as an artifact
34 ------------------------------------------
36 The CSAR is a heat template package with Helm chart in it. The basic package
39 * an **environment file**
40 * a **base_dummy.yaml file** (example)
42 * a **tar.gz** file (of Helm chart)
44 These files must be zipped before onboarding.
45 One thing to pay much attention to is the naming convention which MUST
46 be followed while making the tgz.
48 **NOTE**: The Naming convention is for the helm chart tgz file.
50 **Naming convention follows the format:**
52 <free format string>\_\ ***cloudtech***\ \_<technology>\_<subtype>.extension
54 1. *Cloudtech:* is a fixed pattern and should not be changed if not
56 2. *Technology:* k8s, azure, aws
57 3. *Subtype*: charts, day0, config template
58 4. *Extension*: zip, tgz, csar
60 **NOTE**: The .tgz file must be a tgz created from the top level helm chart
61 folder. I.e. a folder that contains a Chart.yaml file in it.
62 For vFW use case the content of tgz file must look as follows:
66 $ helm package firewall
68 $ tar -tf firewall-0.1.0.tgz
72 firewall/templates/onap-private-net.yaml
73 firewall/templates/_helpers.tpl
74 firewall/templates/protected-private-net.yaml
75 firewall/templates/deployment.yaml
76 firewall/templates/unprotected-private-net.yaml
78 firewall/charts/sink/.helmignore
79 firewall/charts/sink/Chart.yaml
80 firewall/charts/sink/templates/configmap.yaml
81 firewall/charts/sink/templates/_helpers.tpl
82 firewall/charts/sink/templates/service.yaml
83 firewall/charts/sink/templates/deployment.yaml
84 firewall/charts/sink/values.yaml
85 firewall/charts/packetgen/.helmignore
86 firewall/charts/packetgen/Chart.yaml
87 firewall/charts/packetgen/templates/_helpers.tpl
88 firewall/charts/packetgen/templates/deployment.yaml
89 firewall/charts/packetgen/values.yaml
92 An example of the contents inside a heat template package can be found hereafter.
97 MANIFEST.json base_dummy.env base_dummy.yaml
98 vfw_cloudtech_k8s_charts.tgz vfw_k8s_demo.zip
104 Key thing is note the addition of cloud artifact
108 type: "CLOUD_TECHNOLOGY_SPECIFIC_ARTIFACTS"
115 "file": "base_dummy.yaml",
120 "file": "base_dummy.env",
126 "file": "vfw_cloudtech_k8s_charts.tgz",
127 "type": "CLOUD_TECHNOLOGY_SPECIFIC_ARTIFACTS"
135 It is an example of the minimal HEAT template.
139 ##==================LICENSE_START========================================
141 ## Copyright (C) 2019 Intel Corporation
142 ## SPDX-License-Identifier: Apache-2.0
144 ##==================LICENSE_END===========================================
146 heat_template_version: 2016-10-14
147 description: Heat template to deploy dummy VNF
153 description: Dummy name
157 label: id of vnommand to read (GET) Definition
158 description: Provided by ONAP
163 description: Provided by ONAP
168 description: Provided by ONAP
172 label: Image name or ID
173 description: Dummy image name
178 description: Dummy flavor
182 type: OS::Nova::Server
184 name: { get_param: dummy_name_0 }
185 image: { get_param: dummy_image_name }
186 flavor: { get_param: dummy_flavor_name } metadata: { vnf_name: { get_param: vnf_name }, vnf_id: { get_param: vnf_id }, vf_module_id: { get_param: vf_module_id }}
194 vnf_id: PROVIDED_BY_ONAP
195 vnf_name: PROVIDED_BY_ONAP
196 vf_module_id: PROVIDED_BY_ONAP
197 dummy_name_0: dummy_1_0
198 dummy_image_name: dummy
199 dummy_flavor_name: dummy.default
204 For onboarding instructions please refer to steps 4-9 from the document
205 `here <https://wiki.onap.org/display/DW/vFWCL+instantiation%2C+testing%2C+and+debuging>`__.
207 Steps for installing KUD Cloud
208 ------------------------------
210 Follow the link to install KUD Kubernetes Deployment. KUD contains all the
211 packages required for running vfw use case.
213 Kubernetes Baremetal deployment instructions can be found here_
215 .. _here: https://wiki.onap.org/display/DW/Kubernetes+Baremetal+deployment+setup+instructions/
217 REGISTER KUD CLOUD REGION with K8s-Plugin
218 -----------------------------------------
220 You must declare the KUD as a new cloud region in ONAP thanks to ONAP multicloud
223 POST connectivity info
224 ~~~~~~~~~~~~~~~~~~~~~~
226 Create a the post.json file as follows:
231 "cloud-region" : "<name>", // Must be unique
232 "cloud-owner" : "<owner>",
233 "other-connectivity-list" : {
236 Then send the POST HTTP request as described below:
240 curl -i -F "metadata=<post.json;type=application/json" -F file=@
241 /home/ad_kkkamine/.kube/config -X POST http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info
243 GET Connectivity Info
244 ~~~~~~~~~~~~~~~~~~~~~~
248 curl -i -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info/{name}
251 DELETE Connectivity Info
252 ~~~~~~~~~~~~~~~~~~~~~~~~
256 curl -i -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info/{name}
259 UPDATE/PUT Connectivity Info
260 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
264 curl -i -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/connectivity-info/{name}
266 Register KUD Cloud region with AAI
267 ----------------------------------
269 A tenant must be added to the k8s cloud region.
270 The 'easy' way is to have the ESR information (in step 1 of cloud
271 registration) pointing to a real OpenStack tenant (e.g. the OOF tenant in
272 the lab where we tested).
274 This will cause multicloud to add the tenant to the k8s cloud region and
275 then, similarly to #10 in the documentation
276 `here <https://onap.readthedocs.io/en/casablanca/submodules/integration.git/docs/docs_vfwHPA.html#docs-vfw-hpa>`__,
277 the service-subscription can be added to that object.
279 **NOTE:** use same name cloud-region and cloud-owner name
281 An example is shown below for K8s cloud but following the steps 1,2,3
283 `here <https://onap.readthedocs.io/en/latest/submodules/multicloud/framework.git/docs/multicloud-plugin-windriver/UserGuide-MultiCloud-WindRiver-TitaniumCloud.html#tutorial-onboard-instance-of-wind-river-titanium-cloud>`__.
284 The sample input below is for k8s cloud type.
286 **Step 1**: Cloud Registration/ Create a cloud region to represent the instance
288 Note: the highlighted part of the body refers to an existing OpenStack
289 tenant (OOF in this case). It is provided as an illustration and must be adapted
290 according to your configuration.
294 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v13/cloud-infrastructure/cloud-regions/cloud-region/k8scloudowner4/k8sregionfour
296 "cloud-owner": "k8scloudowner4",
297 "cloud-region-id": "k8sregionfour",
299 "owner-defined-type": "t1",
300 "cloud-region-version": "1.0",
301 "complex-name": "clli1",
302 "cloud-zone": "CloudZone",
303 "sriov-automation": false,
304 "cloud-extra-info":"{\"openstack-region-id\":\"k8sregionthree\"}",
305 "esr-system-info-list": {
308 "esr-system-info-id": "55f97d59-6cc3-49df-8e69-926565f00066",
309 "service-url": "http://10.12.25.2:5000/v3",
311 "password": "onapdemo",
312 "system-type": "VIM",
313 "ssl-insecure": true,
314 "cloud-domain": "Default",
315 "default-tenant": "OOF",
316 "tenant-id": "6bbd2981b210461dbc8fe846df1a7808",
317 "system-status": "active"
323 **Step 2:** Add a complex to the cloud
325 Adding an already existing complex is enough. You do not need to create new ones.
329 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v13/cloud-infrastructure/cloud-regions/cloud-region/k8scloudowner4/k8sregionfour/relationship-list/relationship
331 "related-to": "complex",
332 "related-link": "/aai/v13/cloud-infrastructure/complexes/complex/clli1",
333 "relationship-data": [
335 "relationship-key": "complex.physical-location-id",
336 "relationship-value": "clli1"
341 **Step 3:** Trigger the Multicloud plugin registration process
345 POST http://{{MSB_IP}}:{{MSB_PORT}}/api/multicloud-titaniumcloud/v1/k8scloudowner4/k8sregionfour/registry
348 This step allws the registration of the K8S cloud with Multicloud. It also
349 reaches out and adds tenant information to the cloud (see example below, you
350 will see all kinds of flavor, image information that are associated with the OOF
353 If you did not follow the procedure described above then you will have to connect
354 on AAI point and manually add a tenant to the cloud region.
356 Once the tenant declared, you can add the service-subscription to it:
358 **Step 4:** Create a Service Type
362 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v13/service-design-and-creation/services/service/vfw-k8s
364 "service-description": "vfw-k8s",
365 "service-id": "vfw-k8s"
368 Add subscription service info to the service type of the customer.
372 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v16/business/customers/customer/Demonstration/service-subscriptions/service-subscription/vfw-k8s
374 "service-type": "vfw-k8s"
377 Add Service-Subscription to the tenant. The parameter resource-version is a
382 PUT https://{{AAI1_PUB_IP}}:{{AAI1_PUB_PORT}}/aai/v16/cloud-infrastructure/cloud-regions/cloud-region/k8scloudowner4/k8sregionfour/tenants/tenant/6bbd2981b210461dbc8fe846df1a7808?resource-version=1559345527327
384 "tenant-id": "6bbd2981b210461dbc8fe846df1a7808",
385 "tenant-name": "OOF",
386 "resource-version": "1559345527327",
387 "relationship-list": {
390 "related-to": "service-subscription",
391 "relationship-label": "org.onap.relationships.inventory.Uses",
392 "related-link": "/aai/v13/business/customers/customer/Demonstration/service-subscriptions/service-subscription/vfw-k8s",
393 "relationship-data": [
395 "relationship-key": "customer.global-customer-id",
396 "relationship-value": "Demonstration"
399 "relationship-key": "service-subscription.service-type",
400 "relationship-value": "vfw-k8s"
411 Once the cloud region is properly declared, it is possible to onboard the service
412 in the SDC and triggers a distribution from the SDC to the main ONAP components.
413 SO and other services are notified. A sdc listener is also getting the
414 distribution information in the multicloud sidecar.
415 When distribution happens it takes tar.gz file and uploads it to k8s plugin.
417 Create Profile Manually
418 -----------------------
420 K8s-plugin artifacts start in the form of Definitions. These are nothing
421 but Helm Charts wrapped with some metadata about the chart itself. Once
422 the Definitions are created, some profiles can be created. Finally it is
423 possible to customize the definition and instantiate it in the targeted
426 **NOTE:** Refer this link_ for complete API lists and documentation:
428 .. _link : https://wiki.onap.org/display/DW/MultiCloud+K8s-Plugin-service+API
430 The profile consists in:
432 * the **manifest.yaml**. It contains the details for the profile
433 * a **HELM** values override yaml file: It can have any name as long as it matches
434 the corresponding entry in the **manifest.yaml**
435 * Additional files organized in a folder structure: all these files should have
436 a corresponding entry in **manifest.yaml** file
438 Create a Profile Artifact
439 ~~~~~~~~~~~~~~~~~~~~~~~~~
443 > cd multicloud-k8s/kud/tests/vnfs/testrb/helm/profile
449 testfol/subdir/deployment.yaml
451 # Create profile tar.gz
453 > tar -cf profile.tar *
455 > mv profile.tar.gz ../
457 The manifest file contains the following parameters:
464 values: "values_override.yaml"
466 - filepath: testfol/subdir/deployment.yaml
467 chartpath: vault-consul-dev/templates/deployment.yaml
469 **Note:** values: "values\_override.yaml" can **be** empty **file** **if**
470 you are creating **a** dummy **profile**
472 **Note:** A dummy profile does not need any customization. The following is
473 optional in the manifest file.
478 - filepath: testfol/subdir/deployment.yaml
479 chartpath: vault-consul-dev/templates/deployment.yaml
481 The name of the Definition is retrived from SDC service distribution stage.
483 Retrieve the definition name
484 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
486 Execute the following command on the ONAP K8s Rancher host to read the definition
487 name and its version:
491 kubectl logs -n onap `kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep multicloud-k8s | head -1` -c multicloud-k8s
493 From the output read the name of the definition which is "rb-name" and
494 "rb-version" respectively
498 127.0.0.1 - - [15/Jul/2019:07:56:21 +0000] "POST /v1/rb/definition/test-rbdef/1/content HTTP/1.1"
503 With this information, it is possible to upload the profile with the following JSON data
508 "rb-name": "test-rbdef",
510 "profile-name": "p1",
511 "release-name": "r1", //If release-name is not provided, profile-name will be used
512 "namespace": "testnamespace1",
513 "kubernetes-version": "1.13.5"
521 curl -i -d @create_rbprofile.json -X POST http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile
525 UPLOAD artifact for Profile
526 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
530 curl -i --data-binary @profile.tar.gz -X POST http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile/p1/content
538 curl -i http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile
540 curl -i http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile/p1
548 curl -i -X DELETE http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/rb/definition/test-rbdef/1/profile/p1
554 Instantiation is done by SO. SO then talks to Multi Cloud-broker via MSB
555 and that in turn looks up the cloud region in AAI to find the endpoint.
556 If k8sregion one is properly registered in AAI (SO check),
557 then the broker will know that it needs to talk to k8s-plugin based on
558 the type of the registration.
560 Instantiate the created Profile via the following REST API
561 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
565 Using the following JSON:
567 "cloud-region": "kud",
568 "profile-name": "p1",
569 "rb-name":"test-rbdef",
575 **NOTE**: Make sure that the namespace is already created before instantiation.
577 Instantiate the profile with the ID provided above
579 Instantiate a Profile
580 ~~~~~~~~~~~~~~~~~~~~~
584 curl -d @create_rbinstance.json http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/instance
587 The command shall return the following JSON
595 "profile-name": "profile1",
596 "cloud-region": "kud",
597 "namespace": "testns",
618 "Version": "v1beta1",
619 "Kind": "StatefulSet"
621 "Name": "profile1-mongo"
626 Delete Instantiated Kubernetes resources
627 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
629 The **id** field from the returned JSON can be used to **DELETE** the
630 resources created in the previous step. This executes a Delete operation
631 using the Kubernetes API.
635 curl -X DELETE http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/instance/ZKMTSaxv
638 GET Instantiated Kubernetes resources
639 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
641 The **id** field from the returned JSON can be used to **GET** the
642 resources created in the previous step. This executes a get operation
643 using the Kubernetes API.
647 curl -X GET http://MSB_NODE_IP:30280/api/multicloud-k8s/v1/v1/instance/ZKMTSaxv
650 `*\ https://github.com/onap/oom/blob/master/kubernetes/multicloud/resources/config/provider-plugin.json <https://github.com/onap/oom/blob/master/kubernetes/multicloud/resources/config/provider-plugin.json>`__
652 Create User parameters
653 ~~~~~~~~~~~~~~~~~~~~~~
655 We need to create parameters that ultimately get translated as:
662 "attribute_name": "definition-name",
663 "attribute_value": "edgex"
666 "attribute_name": "definition-version",
667 "attribute_value": "v1"
670 "attribute_name": "profile-name",
671 "attribute_value": "profile1"