1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
4 ===================================================
5 Container based network service/function deployment
6 ===================================================
7 https://wiki.onap.org/pages/viewpage.action?pageId=16007890
9 This proposal is to implement PoC in Beijing release(R-2) in order to
10 get experience/feedback for future progress.
15 The current ONAP supports only VM based cloud infrastructure for VNF.
16 On the other hand, in the industry container technology is getting more
17 momentum. Increasing VNF density on each node and latency
18 requirements are driving container based VNFs. This project enhances
19 ONAP to support VNFs as containers in addition to VNFs as VMs.
21 It is beneficial to support for multiple container orchestration technologies
22 as cloud infrastructure:
23 * Allow VNFs to run within container technology and also allow closed
24 feedback loop same to VM based VIM. e.g. openstack.
25 * Support for co-existence of VNF VMs and VNF containers
26 * Add container orchestration technology in addition to the
27 traditional VM-based VIM or environment managed by ONAP.
28 * Support for uniform network connectivity among VMs and containers.
30 NOTE: This is different from OOM project `OOM`_. their scope is to
31 deploy ONAP itself on k8s. Our scope is to deploy/manage VNFs on
32 container/container orchestration engine(coe). The first target is
33 k8s. Other CoE will also be addressed if someone steps up to support it.
39 Scope for Beijing release(R-2)
40 ------------------------------
42 * First baby step is to support containers in a Kubernetes cluster via a
43 Multicloud SBI /K8S Plugin
44 (other COE's(Container Orchestration Engine) are out of Beijing scope.
45 They are future scope.)
46 * Minimal implementation with zero impact on MVP of Multicloud Beijing work
49 * Sample VNFs(vFW and vDNS)
50 (vCPE use case is post Beijing release)
51 Both vFW and vDNS are targeted. Since custom TOSCA node definitions
52 are used (please refer to tosca section below), new TOSCA templates
53 are needed for them. (In future, post-Beijing, this will be revised
54 to share common TOSCA template.)
58 In Beijing release, several design aspects are compromised to re-use
59 the existing component/work flow with zero impact primarily because
60 the long term solution is not ready to use. It's acceptable this effort
61 in Beijing release is PoC or experimental to demonstrate functionality
62 and to get experience for post-Beijing design/architecture.
63 For example, the use of CSAR/new tosca node definitions are to re-use
64 the existing code(i.e. Amsteldam release). After Beijing release, those
65 will be revised for long term solution along with related topics. e.g.
66 model driven API, modeling based on Beijing experience. Once we
67 figured out what multicloud COE API should look like and what adapters
68 in other projects(SO, APP-C) are needed(or not needed) in long term,
69 the inter-project discussion (mainly with SO, APP-C) will start in
74 * Register/unregister k8s cluster instances which are already deployed.
75 dynamic deployment of k8s is out of scope. It is assumed that admin knows
76 all the necessary parameters.
77 * onboard VNFD/NSD to use container
78 * Instantiate / de-instantiate containerized VNFs through K8S Plugin
80 * Vnf configuration with sample VNFs(vFW, vDNS) with the existing configuration
81 interface. (no change to the existing configuration interface)
87 REST API Impact and Base URL
88 ----------------------------
90 Similar to other plugins(e.g. openstack plugin), k8s plugin has
91 its own API endpoint and base URL so that it doesn't affect other
92 multicloud northbound API.
94 Base URL for kubernets plugin:
96 https://msb.onap.org:80/api/multicloud/v0/
98 NOTE: Each multicloud plugin has its own API endpoint(ip address).
99 So the plugin is distinguished by endpoint IP address with MSB.
100 "multicloud-kubernetes" name space for MSB is used.
101 NOTE: each COE support has its own API end point and name space.
102 their name spaces will be "multicloud-<coe name>". With model driven
103 API, we will have API agnostic to COE. in that case the name space
104 "multicloud-coe" will be used.
108 In ONAP, cloud-id is the format of <cloudOwner>_<cloudRegion>
109 Since k8s doesn't have notion of region, cloud admin will assign
110 unique string it as cloudRegion and it will be used as a part of cloud-id.
112 APIs for VNF Lifecycle Management
113 ---------------------------------
115 * PATH: /<cloud-id>/proxy/<resources>
116 * METHOD: All methods
118 Northbound components, e.g. APP-C, use these APIs for lifecycle management of
119 VNF resources within the Kubernetes cluster, e.g. pods. In essence, these APIs
120 provide simple proxy (or passthrough) functions with authorization adjustment
121 to Kubernetes API Server so that the relevant lifecycle management operations
122 are actually achieved by Kubernetes cluster itself. In another word, these API
123 requests are proxied to "{kubernetes api prefix}/<resources>" within Kubernetes
124 cluster without any changes to http/https request body.
125 the API end point is stored in AA&I and the API consumer will get it from
128 For details of Kubernetes API, please refer to
129 https://kubernetes.io/docs/reference/api-overview/
131 NOTE: kubernetes doesn't have concept of region and tenant at this moment.
132 So region and tenant_id isn't in path.
133 NOTE: VF-C is ETSI NFV orchestrater.(NFV-O) In Beijing release, this isn't
134 addressed because container is out of scope of ETSI NFV at the time of
135 writing. Post-Beijing, this will be given consideration. First target
136 is APP-C as it's easier.
138 API for VNF Deployment
139 ----------------------
141 * PATH: /<cloud-id>/package
143 media type of Content-Type and/or filename of Contest-Disposition are used
144 to specify package type.
146 As private media type, application/onap-multicloud-<coe name>-<type> is used.
147 More concretely for Beijing release the following media types are used.
148 * Content-Type: application/onap-multicloud-kubernetes-csar
149 * Content-Type: application/onap-multicloud-kubernetes-helm
150 As supplement, filename is also used to guess media type. As http header type
151 Contest-Disposition is used to pass filename.
152 * Content-Disposition: attachment; filename="fname.tgz"
153 first media type is tried and then filename is tried. If both are present
156 This API provides northbound components, e.g. SO, with the function of
157 deploying containerized VNF package into Kubernetes cluster. The VNF package
158 is delivered as payload of HTTP request body in the API call. The VNF package
159 could be a CSAR or Helm Charts.
161 CSAR deployment package will include a yaml deployment file and other artifacts.
162 This approach would work for simple VNFs consisting of single PODs.
164 For VNFs comprising of multiple PODs which are dependent on each other, Helm
165 based approach would be used. The VNF package would be described as a Helm
166 package consisting of a set of Helm charts and k8s yamls for each constituent
167 service that is part of the VNF.
169 There would be no change required in the Northboud API from MultiCloud for
170 either CSAR package or Helm package or any other package in the future. SO calls
171 this MultiVIM Northbound API and sends the k8s package (e.g. csar, or tgz)
172 as payload. k8s Plugin will distinguish package types based on its suffix
173 and interact with k8s cluster appropriately:
175 * For CSAR: k8s yaml file will be extracted from CSAR. k8s REST API server
176 will be called to create k8s resources (e.g. pods), which is equivalent to
177 "kubectl create -f <file.yaml>". The TOSCA file in CSAR is expected to include
178 onap.multicloud.container.kubernetes.proxy.nodes.resources_yaml
179 node which is explained below. In another word, Kubernetes yaml is stored as
180 artifact in CSAR. it is extracted and then it is fed to k8s API.
182 * For TGZ: call Tiller API (gRPC-based) and pass through the Helm package
184 The Kubernetes API Server (RESTful) or Helm Tiller Server (gRPC) URLs are
185 configured for k8s Plugin when the Kubernetes cluster is created and Helm
188 With this single API for package, when we need to add new package
189 support in the future, no extra code in SO is needed.
194 swagger.json for kubernetes API definitions
197 returns swagger.json definitions of k8s API similar to other multicloud plugins
199 Internal APIs for Implementations
200 ---------------------------------
202 Some internal APIs may be needed by the implementation details of above
203 northbound APIs. For example, when implementing VNF Deployment API above,
204 we may need internal APIs to assist calling Helm Tiller Server or Kubernetes
205 API Server, e.g. similar to "kubectl create -f xxx.yaml".
207 The internal API, if needed, will be handled in implementation, which is out
208 of scope of this section of the document.
213 In this section test play is discussed. In Beijing cycle, test is minimal
214 or stretched goal because the effort in Beijing is PoC/experimental
215 to get experience. the following class of test would be planned as
221 ** communication to backend(K8S API server, helm tiller server)
222 * CSIT as end-to-end test
225 Register/Unregister Kubernetes Cluster Instance
226 ===============================================
228 This is done via A&AI ESR `ESR`_ to follow the way of the existing
229 multicloud. some attributes, e.g. region id, don't make sense for
230 k8s. In that case predefined value, e.g. 'default', are used.
231 The info for basic authentication, i.e. the pair of (username, password),
232 against kuberenetes API is registered and stored in A&AI.
234 NOTE: ESR will call registry API when register a new VIM(k8s). we need to
235 make sure that we have this API in this plugin and give them response.
237 NOTE: HPA(kubernetes cluster features/capabilities) is out of scope
238 for Beijing Assumption K8s cluster instance is already
239 pre-build/deployed Dynamic instantiation is out of scope(for Beijing)
241 attributes for A&AI ESR
242 -----------------------
244 This subsection describes how attributes for VIM registration are specified.
245 For actual definitions, please refer to `ESR`_
246 Some attributes doesn't apply to kubernetes so that such attributes will
247 be left unspecified if it's optional or define pre-defined constants if
250 URI /api/aai-esr-server/v1/vims
255 ------------------ ---------- ------- ----------------------------------------
256 Attribute Qualifier Content Description
257 ================== ========== ======= ========================================
258 cloudOwner M String any string as cloud owner
259 ------------------ ---------- ------- ----------------------------------------
260 cloudRegionId M String e.g. "kubernetes-<N>" as it doesn't apply
261 to k8s. Cloud admin assigns unique id.
262 ------------------ ---------- ------- ----------------------------------------
263 cloudType M String "kubernetes". new type
264 ------------------ ---------- ------- ----------------------------------------
265 cloudRegionVersion M String kubernetes version. "v1.9", "v1.8" ...
266 ------------------ ---------- ------- ----------------------------------------
267 ownerDefinedType O String None. (not specified)
268 ------------------ ---------- ------- ----------------------------------------
269 cloudZone O String None. (not speicfied)
270 as kubernetes doesn't have notion of
272 ------------------ ---------- ------- ----------------------------------------
273 complexName O String None. (not specified)
274 as kubernetes doesn't have notion of
276 ------------------ ---------- ------- ----------------------------------------
277 cloudExtraInfo O String json string(dictionary) for necessary
278 info. For now "{}" empty dictionary.
279 For helm support, URL for tiller server
281 ------------------ ---------- ------- ----------------------------------------
282 vimAuthInfos M [Obj] Auth information of Cloud
283 list of authInfoItem which is described
285 ================== ========== ======= ========================================
287 There are several constraints/assumptions on cloudOwner and
288 cloudRegionId. `cloud-region`_ . For k8s, cloudRegionId is (ab)used to
289 specify k8s cluster instance. ONAP admin has to assign unique id for
290 cloudRegionId as id for k8s cluster instance.
292 NOTE: complexName: this will be revised post-Beijing. "complex" is used to
293 specify (latitude, longitude) of a data center location for the purpose of
294 homing optimization. If those values can be obtained somehow, this should
299 Basic authentication is used for k8s api server.
301 -------------- --------- ------- -------------------------------------------
302 Attribute Qualifier Content Description
303 ============== ========= ======= ===========================================
304 cloudDomain M String "kubernetes" as this doesn't apply.
305 -------------- --------- ------- -------------------------------------------
306 userName M String User name
307 -------------- --------- ------- -------------------------------------------
308 password M String Password
309 -------------- --------- ------- -------------------------------------------
310 authUrl M String URL for kubernetes API server
311 -------------- --------- ------- -------------------------------------------
312 sslCacert O String ca file content if enabled ssl on
313 kubernetes API server
314 -------------- --------- ------- -------------------------------------------
315 sslInsecure O Boolean Whether to verify VIM's certificate
316 ============== ========= ======= ===========================================
318 NOTE: For some issues `issue23`_, ESR should provide authenticating by
319 bearer token for Kubernetes cluster if possible beside basic authentication.
320 Those extra value will be stored in cloudExtraInfo. This is stretched goal.
323 On boarding/packaging/instantiation
324 ===================================
326 We shouldn't change the current existing work flow.
327 In short term: Use additional node type/capability types etc.
328 In longer term way: Follow ONAP community directoin. At the moment, work
329 with TOSCA community to add additional node type to express k8s.
331 NOTE: this packaging is temporally work around until ONAP modelling
332 and multicloud model driven API are available. Post Beijing release
333 packaging will be revised to follow ONAP modeling and multicloud model
336 Packaging and on-boarding
337 -------------------------
339 Reuse CASR so that the existing work flow doesn't need change. For
340 Beijing CSAR is used with its own TOSCA node definition. In longer
341 term, once multicloud project has model driven API, it will be followed
342 to align with modeling and SO.
344 TOSCA node definitions
345 -----------------------
347 Introduce new nodes to wrap k8s ingredients(k8s yaml, helm etc.) These
348 TOSCA node definitions are short term work around to re-use the existing
349 component/workflow until model driven API is defined/implemented.
350 For Beijing, human will write this TOSCA by hands for PoC. Post Beijing,
351 packaging needs to be revised to align with modeling and SO. Also SDC,
352 VNF-SDK need to be addressed for creation.
354 * onap.multicloud.nodes.kubernetes.proxy
360 onap.multicloud.container.kubernetes.proxy.nodes.resources_yaml:
369 Paths to kubernetes yaml file
371 For VNFs that are packages as Helm package there would be only one
372 TOSCA node in the TOSCA template which would have reference to the
375 * onap.multicloud.nodes.kubernetes.helm
381 onap.multicloud.container.kubernetes.helm.nodes.helm_package:
390 Paths to Helm package file
392 This TOSCA node definitions wrap kubernetes yaml file or helm chart.
393 cloudify.nodes.Kubernetes isn't reused in order to avoid definition conflict.
398 SO ARIA adaptor can be used. (with twist to have SO to talk to
399 multicloud k8s plugin instead of ARIA) Instantiation so that SO
400 can talk to multicloud k8s plugin.
401 NOTE: This is temporally work around for Beijing release. Post Beijing, this
407 With Amsteldam Release, SO has ARIA adaptor which talks to ARIA orchestrator.
408 https://wiki.onap.org/download/attachments/16002054/Model%20Driven%20Service%20Orchestration%20-%20SO%20State%20of%20the%20Union.pptx
410 The work flow looks like follows::
412 user request to instantiate VNF
414 +--------------|-------+
417 | +------------------+ |
418 | | SO: ARIA adaptor | |
419 | +------------+-----+ |
420 +--------------|-------+
423 +--------------|---------+
426 | +--------------------+ |
427 | | multicloud plugin | | template as TOSCA artifact is
428 | +------------+-------+ | extracted and build requests to
429 +--------------|---------+ multicloud
432 +--------------|-------+
435 | +------------------+ |
436 | | openstack plugin | |
437 | +------------+-----+ |
438 +--------------|-------+
442 +----------------------+
444 +----------------------+
447 This will be twisted by configuration so that SO can talks to
448 multicloud k8s plugin::
450 user request to instantiate VNF
452 +--------------|-------+
455 | +------------------+ |
456 | | SO: ARIA adaptor | | configuration is twisted to call
457 | +------------+-----+ | multicloud k8s API
458 +--------------|-------+
461 +--------------|-------+
464 | +------------------+ | handle CSAR or TGZ (Helm Charts) file
465 | | k8s plugin | | e.g. extract k8s yaml from CSAR, and
466 | +------------+-----+ | pass through requests to k8s/Helm API
467 +--------------|-------+
471 +----------------------+
472 | k8s/Helm API server |
473 +----------------------+
476 NOTE: In this work flow. only the northbound deployment API endpoint is needed
477 for VNF deployment. LCM APIs are only needed for lifecycle management. Other
478 internal APIs, e.g. k8s YAML API may be needed only for internal implementation.
480 SO ARIA multicloud plugin needs to be twisted to call k8s plugin.
482 The strategy is to keep the existing design of ONAP or to follow
484 The key point of The interaction between SO and multicloud is
485 * SO decomposes VNFD/NSD into single atomic resource
486 (e.g. VNF-C corresponding to single VM or single container/pod)
487 and send requests to create each resources via deployment API.
488 * multicloud accepts each request for single atomic resource and
489 create single resource(e.g. VM or container/pod)
490 * multicloud doesn't do resource decomposition. The decomposition is task
493 API work flow example and k8s API
494 ---------------------------------
495 * register k8s cluster to A&AI ESR
496 <cloud-id> is obtained
497 * ONAP north bound components generates a TOSCA template targeted for k8s.
498 * SO calls Multicloud deployment API and passes the entire BluePrint(as CSAR or
499 TGZ) to k8s plugin, e.g.:
500 POST https://msb.onap.org:80/api/multicloud/v0/<cloud-id>/package
501 * k8s plugin handles the CSAR or TGZ accordingly and talks to k8s API Server
502 or Helm Tiller Server to deploy containerized VNF
503 POST <k8s api server>://api/v1/namespaces/{namespace}/pods
504 to create pods. then <pod id> is obtained
505 * DELETE https://msb.onap.org:80/api/multicloud/v0/<cloud-id>/proxy/api/v1/namespaces/{namespace}/pods/<pod id>
507 * to execute script inside pod, the following URL can be used.
508 POST /api/v1/namespaces/{namespace}/pods/{name}/exec
511 Affected Projects and impact
512 ============================
516 new type to represent k8s/container for cloud infrastructure will
517 be introduced as work around. Post Beijing official value will be
518 discussed for inclusion.
522 Policy matching is done by OOF.
523 For Beijing. Enhancement to policy is stretched goal.
524 Decomposing service design(NSD, VNFD) from VNF package is done by SO
529 ARIA adaptor is re-used with config tweak to avoid modification
533 new k8s plugin will be introduced. The details are discussed in this
534 documentation you're reading right now.
537 Kubernetes cluster authentication
538 =================================
539 For details of k8s authentication, please refer to
540 https://kubernetes.io/docs/admin/authentication
542 Because Kubernetes cluster installation is not mentioned, we should
543 treat all users as normal users when authenticate to
544 Kubernetes VIM. There are several ways to authenticate Kubernetes
545 cluster. For Beijing release, basic authentication will be supported.
546 username and password are stored in ESR.
551 Past presentations/proposals
552 ----------------------------
553 .. _Munish proposal: https://schd.ws/hosted_files/onapbeijing2017/dd/Management%20of%20Cloud%20Native%20VNFs%20with%20ONAP%20PA5.pptx
554 .. _Isaku proposal:https://schd.ws/hosted_files/onapbeijing2017/9d/onap-kubernetes-arch-design-proposal.pdf
555 .. _Bin Hu proposal:https://wiki.onap.org/download/attachments/16007890/ONAP-SantaClara-BinHu-final.pdf?version=1&modificationDate=1513558701000&api=v2
559 .. _ESR: Extenral System Register https://wiki.onap.org/pages/viewpage.action?pageId=11930343#A&AI:ExternalSystemOperationAPIDefinition-VIM
560 .. _AAI: Active and Available Inventory https://wiki.onap.org/display/DW/Active+and+Available+Inventory+Project
561 .. _OOM: ONAP Operations Manager https://wiki.onap.org/display/DW/ONAP+Operations+Manager+Project
562 .. _ONAPREST: RESTful API Design Specification https://wiki.onap.org/display/DW/RESTful+API+Design+Specification
566 .. _kubernetes-python-client: Kubernetes python client https://github.com/kubernetes-client/python
568 .. _issue23: https://github.com/kubernetes/kubeadm/issues/23
572 .. _cloud-region: How to add a new cloud region and some thoughts https://wiki.onap.org/download/attachments/25429038/HowToAddNewCloudRegionAndThoughts.pdf
577 * Isaku Yamahata <isaku.yamahata@intel.com> <isaku.yamahata@gmail.com>
578 * Bin Hu <bh526r@att.com>
579 * Munish Agarwal <munish.agarwal@ericsson.com>
580 * Phuoc Hoang <phuoc.hc@dcn.ssu.ac.kr>
585 This section is informative. This is out of Beijing scope and will be
586 revised after Beijing.
587 The purpose of this appendix is to help readers to understand this
588 proposal by giving future direction and considerations.
589 At some point, this appendix will be separated out into its own document
590 for long-term right design.
592 Model driven API and kubernetes model
593 -------------------------------------
594 Currently the discussion on model driver API is on going. Once it's usable,
595 it will be followed and the above experimental API/code will be revised.
597 The eventual work flow looks like as follows::
599 user request to instantiate VNF/NS
602 +----------------------+ +-----+
603 | SO |-------->| OOF | <--- policy to use
604 | |<--------| | CoE instead of VM
605 | | +-----+ from A&AI
606 | +------------------+ |
607 | | SO: adaptor for | | SO decomposes VNFD/NSD into atomic
608 | | multicloud model | | resources(VDUs for VNF-C) with asking OOF
609 | | driven API | | for placement. then SO builds up
610 | +------------+-----+ | requests to multicoud for instantiation.
611 +--------------|-------+
614 +--------------|-------+
615 | multicloud | | So multicloud accepts request for single
616 | V | resource of VDU which corresponds to
617 | +------------------+ | VNF-C. which is mapped to single
618 | | model driven API | | container/pod. multicloud doesn't
619 | +------------+-----+ | decompose VDU into multiple containers.
620 | | | CoE doesn't change such work flow.
622 | +------------------+ |
623 | | k8s plugin | | convert request(VDU of VNF-C) into
624 | +------------+-----+ | kubernetes
625 +--------------|-------+
629 +----------------------+
631 +----------------------+
634 Modeling/TOSCA to kubernetes conversion
635 ---------------------------------------
636 In this section, conversion from TOSCA to kubernetes is discussed
637 so that reader can get idea for future direction.
639 Once ONAP information/data model is usable, similar conversion is possible.
640 The followings are only examples. More node definitions would be considered
643 TOSCA node definition k8s resource
644 ============================ ================================
645 tosca.nodes.Compute (bare)single pod
646 vcpu, memory -> k8s resource
647 ---------------------------- --------------------------------
648 tosca.nodes.nfv.VDU.Compute (bare)single pod
653 This is just to show idea.
654 This example is very early phase and there are hard-coded values.
663 type: tosca.nodes.Compute
665 # Host container properties
671 # Guest Operating System properties
674 # host Operating System image properties
684 $ PYTHONPATH=. python -m tosca_translator.shell -d --debug --template-file tosca_translator/tests/data/tosca_helloworld.yaml
685 api_version: apps/v1beta1
688 labels: {name: my_server}
693 labels: {name: my_server}
699 limits: {cpu: 2, ephemeral-storage: 10 GB, memory: 512 MB}
700 requests: {cpu: 2, ephemeral-storage: 10 GB, memory: 512 MB}