3 VF Module Scale Out Use Case
4 ----------------------------
8 - Heat templates directory: https://git.onap.org/demo/tree/heat/vLB_CDS?h=dublin
12 - DCAE blueprint: https://git.onap.org/integration/tree/docs/files/scaleout/k8s-tca-clamp-policy-05162019.yaml
13 - TOSCA model template: https://git.onap.org/integration/tree/docs/files/scaleout/service-Vloadbalancercds-template.yml
14 - Naming policy script: https://git.onap.org/integration/tree/docs/files/scaleout/push_naming_policy.sh
18 The scale out use case uses a VNF composed of three virtual functions. A traffic generator (vPacketGen), a load balancer (vLB), and a DNS (vDNS). Communication between the vPacketGen and the vLB, and the vLB and the vDNS occurs via two separate private networks. In addition, all virtual functions have an interface to the ONAP OAM private network, as shown in the topology below.
20 .. figure:: files/scaleout/topology.png
23 The vPacketGen issues DNS lookup queries that reach the DNS server via the vLB. vDNS replies reach the packet generator via the vLB as well. The vLB reports the average amount of traffic per vDNS instances over a given time interval (e.g. 10 seconds) to the DCAE collector via the ONAP OAM private network.
25 To run the use case, make sure that the security group in OpenStack has ingress/egress entries for protocol 47 (GRE). Users can test the VNF by running DNS queries from the vPakcketGen:
29 dig @vLoadBalancer_IP host1.dnsdemo.onap.org
31 The output below means that the vLB has been set up correctly, has forwarded the DNS queries to a vDNS instance, and the vPacketGen has received the vDNS reply message.
35 ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @192.168.9.111 host1.dnsdemo.onap.org
37 ;; global options: +cmd
39 ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31892
40 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2
41 ;; WARNING: recursion requested but not available
44 ; EDNS: version: 0, flags:; udp: 4096
46 ;host1.dnsdemo.onap.org. IN A
49 host1.dnsdemo.onap.org. 604800 IN A 10.0.100.101
52 dnsdemo.onap.org. 604800 IN NS dnsdemo.onap.org.
54 ;; ADDITIONAL SECTION:
55 dnsdemo.onap.org. 604800 IN A 10.0.100.100
58 ;; SERVER: 192.168.9.111#53(192.168.9.111)
59 ;; WHEN: Fri Nov 10 17:39:12 UTC 2017
63 The Scale Out Use Case
64 ~~~~~~~~~~~~~~~~~~~~~~
65 The Scale Out use case shows how users/network operators can add Virtual Network Function Components (VNFCs) as part of a VF Module that has been instantiated in the Service model, in order to increase capacity of the network. ONAP Dublin release supports scale out with manual trigger by directly calling SO APIs and closed-loop-enabled automation from Policy. For Dublin, the APPC controller is used to demonstrate accepting request from SO to execute the Scale Out operation. APPC can be used to scale different VNF types, not only the VNF described in this document.
70 For Dublin, the scale out use case integrates with the Controller Design Studio (CDS) ONAP component to automate the generation of cloud configuration at VNF instantiation time. Users can model this configuration at VNF design time and onboard the blueprint to CDS via the CDS GUI. The blueprint includes naming policies and network configuration details (e.g. IP address families, network names, etc.) that CDS will use during VNF instantiation to generate resource names and assign network configuration to VMs through the cloud orchestrator.
72 Please look at the CDS documentation for details about how to create configuration models, blueprints, and use the CDS tool: https://wiki.onap.org/display/DW/Modeling+Concepts. For running the use case, users can use the standard model package that CDS provides out of the box, which can be found here: https://wiki.onap.org/pages/viewpage.action?pageId=64007442
74 Once the configuration blueprint is uploaded to CDS, users can create and instantiate the service using SDC. For details about service design and creation, please refer to the SDC wiki page: https://wiki.onap.org/display/DW/Design
76 During the creation of the service, there are a few extra steps that need to be executed to make the VNF ready for scale out. These require users to login to the SDC Portal as service designer user (username: cs0008, password: demo123456!).
78 After importing the Vendor Software Package (VSP), as described in the SDC wiki page, users need to set property values in the Property Assignment window, as shown below:
80 .. figure:: files/scaleout/9.png
83 These properties include parameters in the Heat template (which will be overridden by CDS and then don't need to be changed) and other parameters that describe the VNF type or are used to link the service to the configuration in the CDS package.
85 Users can search for parameter names starting with "nf" to assign values that describe the VNF type, such as nf_type, nf_function, and nf_role. Users are free to choose the values they like. Users should also set "skip_post_instantiation" to "TRUE", as for Dublin CDS is not used for post-instantiation configuration.
87 .. figure:: files/scaleout/10.png
90 For CDS parameters, users can search for names starting with "sdnc". These parameters have to match the configuration blueprint in CDS. To use the standard blueprint shipped with CDS, please set the parameters as below. For further details, please refer to the CDS documentation.
92 .. figure:: files/scaleout/11.png
96 After importing the VSP, users need to onboard the DCAE blueprint and the Policy Model used to design closed loops in CLAMP. From the "Composition" tab in the service menu, select the artifact icon on the right, as shown below:
98 .. figure:: files/scaleout/1.png
101 Upload the DCAE blueprint linked at the top of the page using the pop-up window.
103 .. figure:: files/scaleout/2.png
106 The blueprint will appear in the artifacts section on the right.
108 .. figure:: files/scaleout/3.png
111 To attach a Policy Model to the service, open the Policy drop-down list on left.
113 .. figure:: files/scaleout/4.png
116 Then, add the TCA Policy.
118 .. figure:: files/scaleout/5.png
121 The Policy will be attached to the service.
123 .. figure:: files/scaleout/6.png
126 Finally, users need to provide the maximum number of instances that ONAP is allowed to create as part of the scale out use case by setting up deployment properties.
128 .. figure:: files/scaleout/7.png
131 This VNF only supports scaling the vDNS, so users should select the vDNS module from the right panel and then click the "max_vf_module_instance" link. The maximum number of instances to scale can be set to an arbitrary number higher than zero.
133 .. figure:: files/scaleout/8.png
136 At this point, users can complete the service creation by testing, accepting, and distributing the Service Models as described in the SDC wiki page.
138 In order to instantiate the VNF using CDS features, users need to deploy the naming policy that CDS uses for resource name generation to the Policy Engine. User can copy and run the script at the top of the page from any ONAP pod, for example Robot or Drools. The script uses the Policy endpoint defined in the Kubernetes domain, so the execution has to be triggered from some pod in the Kubernetes space.
142 kubectl exec -it dev-policy-drools-0
143 ./push_naming_policy.sh
146 Closed Loop Design from CLAMP
147 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
148 Once the service is distributed, users can design the closed loop from CLAMP, using the GUI at https://clamp.api.simpledemo.onap.org:30258/designer/index.html
150 Use the "Closed Loop" link to open a distributed model.
152 .. figure:: files/scaleout/12.png
155 Select the closed loop associated to the distributed service model.
157 .. figure:: files/scaleout/13.png
160 The closed loop main page for TCA microservices is shown below.
162 .. figure:: files/scaleout/14.png
165 Click on the TCA box to create a configuration policy. From the pop-up window, users need to click "Add item" to create a new policy and fill it in with specific information, as shown below.
167 .. figure:: files/scaleout/15.png
170 For this use case, the control loop schema type is "VM", while the event name has to match the event name reported in the VNF telemetry, which is "vLoadBalancer".
172 Once the policy item has been created, users can define a threshold that will be used at runtime to evaluate telemetry reported by the vLB. When the specified threshold is crossed, DCAE generates an ONSET event that will tell Policy Engine which closed loop to activate.
174 .. figure:: files/scaleout/16.png
177 After the configuration policy is created, users need to create the operational policy, which the Policy Engine uses to determine which actions and parameters should be used during closed loop.
179 .. figure:: files/scaleout/17.png
182 Select "VF Module Create" recipe and "SO" actor. The payload section is:
186 requestParameters: '{"usePreload":false,"userParams":[]}'
187 configurationParameters: '[{"ip-addr":"$.vf-module-topology.vf-module-parameters.param[17].value","oam-ip-addr":"$.vf-module-topology.vf-module-parameters.param[31].value"}]'
189 Policy Engine passes the payload to SO, which will then use it during VF module instantiation to resolve configuration parameters. The JSON path
193 "ip-addr":"$.vf-module-topology.vf-module-parameters.param[17].value"
195 indicates that resolution for parameter "ip-addr" is available at "$.vf-module-topology.vf-module-parameters.param[17].value" in the JSON object linked by the VF module self-link in AAI. For the vPacketGen/vLB/vDNS VNF, use the JSON paths provided in the example above.
197 The target type to select is VF module, as we are scaling a VF module. Please select the vDNS module as target resource ID.
199 .. figure:: files/scaleout/18.png
202 For what regards guard policies, either "Frequency Limiter", or "MinMax", or both can be used for the scale out use case. The example below shows the definition of a "Frequency Limiter" guard policy. Irrespective of the guard policy type, the policy name should be x.y.scaleout.
204 Once the operational policy design is completed, users can submit and then deploy the closed loop clicking the "Submit" and "Deploy" buttons, respectively, as shown below.
206 .. figure:: files/scaleout/20.png
209 At this point, the closed loop is deployed to Policy Engine and DCAE, and a new microservice will be deployed to the DCAE platform.
212 Creating a VNF Template with CDT
213 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
214 Before running scale out use case, the users need to create a VNF template using the Controller Design Tool (CDT), a design-time tool that allows users to create and on-board VNF templates into APPC. The template describes which control operation can be executed against the VNF (e.g. scale out, health check, modify configuration, etc.), the protocols that the VNF supports, port numbers, VNF APIs, and credentials for authentication. Being VNF agnostic, APPC uses these templates to "learn" about specific VNFs and the supported operations.
215 CDT requires two input:
217 1) the list of parameters that APPC will receive (ip-addr, oam-ip-addr, enabled in the example above);
219 2) the VNF API that APPC will use to reconfigure the VNF.
221 Below is an example of the parameters file (yaml format), which we call parameters.yaml:
254 Here is an example of API for the vLB VNF used for this use case. We name the file after the vnf-type contained in SDNC (i.e. Vloadbalancerms..vdns..module-3):
257 <vlb-business-vnf-onap-plugin xmlns="urn:opendaylight:params:xml:ns:yang:vlb-business-vnf-onap-plugin">
260 <ip-addr>${ip-addr}</ip-addr>
261 <oam-ip-addr>${oam-ip-addr}</oam-ip-addr>
262 <enabled>true</enabled>
265 </vlb-business-vnf-onap-plugin>
267 To create the VNF template in CDT, the following steps are required:
269 - Connect to the CDT GUI: http://ANY-K8S-IP:30289
270 - Click "My VNF" Tab. Create your user ID, if necessary
271 - Click "Create new VNF" entering the VNF type as reported in VID or AAI, e.g. vLoadBalancerMS/vLoadBalancerMS 0
272 - Select "ConfigScaleOut" action
273 - Create a new template identifier using the VNF type name in service model as template name, e.g. Vloadbalancerms..vdns..module-3
274 - Select protocol (Netconf-XML), VNF username (admin), and VNF port number (2831 for NETCONF)
275 - Click "Parameter Definition" Tab and upload the parameters (.yaml) file
276 - Click "Template Tab" and upload API template (.yaml) file
277 - Click "Reference Data" Tab
278 - Click "Save All to APPC"
280 For health check operation, we just need to specify the protocol, the port number and username of the VNF (REST, 8183, and "admin" respectively, in the case of vLB/vDNS) and the API. For the vLB/vDNS, the API is:
283 restconf/operational/health-vnf-onap-plugin:health-vnf-onap-plugin-state/health-check
285 Note that we don't need to create a VNF template for health check, so the "Template" flag can be set to "N". Again, the user has to click "Save All to APPC" to update the APPC database.
286 At this time, CDT doesn't allow users to provide VNF password from the GUI. To update the VNF password we need to log into the APPC Maria DB container and change the password manually:
289 mysql -u sdnctl -p (type "gamma" when password is prompted)
291 UPDATE DEVICE_AUTHENTICATION SET PASSWORD='admin' WHERE
292 VNF_TYPE='vLoadBalancerMS/vLoadBalancerMS 0'; (use your VNF type)
295 Setting the Controller Type in SO Database
296 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
297 Users need to specify which controller to use for the scale out use case. For Dublin, the supported controller is APPC. Users need to create an association between the controller and the VNF type in the SO database.
301 - Connect to one of the replicas of the MariaDB database
306 mysql -ucataloguser -pcatalog123
308 - Use catalogdb databalse
314 - Create an association between APPC and the VNF type, for example:
318 INSER INTO controller_selection_reference (`VNF_TYPE`, `CONTROLLER_NAME`, `ACTION_CATEGORY`) VALUES ('<VNF Type>', 'APPC', 'ConfigScaleOut');
319 INSER INTO controller_selection_reference (`VNF_TYPE`, `CONTROLLER_NAME`, `ACTION_CATEGORY`) VALUES ('<VNF Type>', 'APPC', 'HealthCheck');
321 SO has a default entry for VNF type "vLoadBalancerMS/vLoadBalancerMS 0"
326 After the VNF has been instantiated using the CDS configuration blueprint, user should manually configure the vLB to open a connection towards the vDNS. At this time, the use case doesn't support automated post-instantiation configuration with CDS. Note that this step is NOT required during scale out operations, as VNF reconfiguration will be triggered by SO and executed by APPC. To change the state of the vLB, the users can run the following REST call, replacing the IP addresses in the VNF endpoint and JSON object to match the private IP addresses of their vDNS instance:
331 http://10.12.5.78:8183/restconf/config/vlb-business-vnf-onap-plugin:vlb-business-vnf-onap-plugin/vdns-instances/vdns-instance/192.168.10.59 \
332 -H 'Accept: application/json' \
333 -H 'Content-Type: application/json' \
334 -H 'Postman-Token: a708b064-adb1-4804-89a7-ee604f5fe76f' \
335 -H 'cache-control: no-cache' \
339 "ip-addr": "192.168.10.59",
340 "oam-ip-addr": "10.0.101.49",
345 At this point, the VNF is fully set up. To allow automated scale out via closed loop, the users need to inventory the VNF resources in AAI. This is done by running the heatbridge python script in /root/oom/kubernetes/robot in the Rancher VM in the Kubernetes cluster:
349 ./demo-k8s.ete onap heatbridge <vLB stack_name in OpenStack> <service_instance_id> vLB vlb_onap_private_ip_0
351 Heatbridge is needed for control loops because DCAE and Policy runs queries against AAI using vServer names as key.
353 For scale out with manual trigger, VID is not supported at this time. Users can run the use case by directly calling SO APIs:
358 http://10.12.5.86:30277/onap/so/infra/serviceInstantiation/v7/serviceInstances/7d3ca782-c486-44b3-9fe5-39f322d8ee80/vnfs/9d33cf2d-d6aa-4b9e-a311-460a6be5a7de/vfModules/scaleOut \
359 -H 'Accept: application/json' \
360 -H 'Authorization: Basic SW5mcmFQb3J0YWxDbGllbnQ6cGFzc3dvcmQxJA==' \
361 -H 'Cache-Control: no-cache' \
362 -H 'Connection: keep-alive' \
363 -H 'Content-Type: application/json' \
364 -H 'Host: 10.12.5.86:30277' \
365 -H 'Postman-Token: 12f2601a-4eb2-402c-a51a-f29502359501,9befda68-b2c9-4e7a-90ca-1be9c24ef664' \
366 -H 'User-Agent: PostmanRuntime/7.15.0' \
367 -H 'accept-encoding: gzip, deflate' \
368 -H 'cache-control: no-cache' \
369 -H 'content-length: 2422' \
370 -H 'cookie: JSESSIONID=B3BA24216367F9D39E3DF5E8CBA4BC64' \
371 -b JSESSIONID=B3BA24216367F9D39E3DF5E8CBA4BC64 \
375 "modelCustomizationName": "VdnsloadbalancerCds..vdns..module-3",
376 "modelCustomizationId": "ec2f6466-a786-41f9-98f3-86506ceb57aa",
377 "modelInvariantId": "8e134fbd-d6fe-4b0a-b4da-286c69dfed2f",
378 "modelVersionId": "297c4829-a412-4db2-bcf4-8f8ab8890772",
379 "modelName": "VdnsloadbalancerCds..vdns..module-3",
380 "modelType": "vfModule",
383 "cloudConfiguration": {
384 "lcpCloudRegionId": "RegionOne",
385 "tenantId": "d570c718cbc545029f40e50b75eb13df",
386 "cloudOwner": "CloudOwner"
389 "instanceName": "RegionOne_ONAP-NF_20190613T023006695Z_1",
391 "suppressRollback": false,
392 "requestorId": "demo"
394 "requestParameters": {
397 "relatedInstanceList": [
400 "instanceId": "7d3ca782-c486-44b3-9fe5-39f322d8ee80",
402 "modelType": "service",
403 "modelInvariantId": "a158d0c9-7de4-4011-a838-f1fb8fa26be8",
404 "modelVersionId": "a68d8e71-206b-4ed7-a759-533a4473304b",
405 "modelName": "vDNSLoadBalancerService_CDS",
406 "modelVersion": "2.0"
412 "instanceId": "9d33cf2d-d6aa-4b9e-a311-460a6be5a7de",
415 "modelInvariantId": "7cc46834-962b-463a-93b8-8c88d45c4fb1",
416 "modelVersionId": "94cb4ca9-7084-4236-869f-9ba114245e41",
417 "modelName": "vDNSLOADBALANCER_CDS",
418 "modelVersion": "3.0",
419 "modelCustomizationId": "69a4ebc7-0200-435b-930a-3cb247d7a3fd"
424 "configurationParameters": [
426 "ip-addr": "$.vf-module-topology.vf-module-parameters.param[17].value",
427 "oam-ip-addr": "$.vf-module-topology.vf-module-parameters.param[31].value"
433 To fill in the JSON object, users can refer to the Service Model TOSCA template at the top of the page. The template contains all the model (invariant/version/customization) IDs of service, VNF, and VF modules that the input request to SO needs.
436 Test Status and Plans
437 ~~~~~~~~~~~~~~~~~~~~~
438 Dublin Scale Out test cases can be found here: https://wiki.onap.org/pages/viewpage.action?pageId=59966105
440 Known Issues and Resolutions
441 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
442 1) When running closed loop-enabled scale out, the closed loop designed in CLAMP conflicts with the default closed loop defined for the old vLB/vDNS use case
444 Resolution: Change TCA configuration for the old vLB/vDNS use case
446 - Connect to Consul: http://<ANY K8S VM IP ADDRESS>:30270 and click on "Key/Value" → "dcae-tca-analytics"
447 - Change "eventName" in the vLB default policy to something different, for example "vLB" instead of the default value "vLoadBalancer"
448 - Change "subscriberConsumerGroup" in the TCA configuration to something different, for example "OpenDCAE-c13" instead of the default value "OpenDCAE-c12"
449 - Click "UPDATE" to upload the new TCA configuration