target/
logs/*.log
*.pyc
+*.swp
# Test related files
multivimbroker/.coverage
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. Copyright (c) 2017-2018 VMware, Inc.
+
+=================
+Event/Alert/Metrics Federation
+=================
+
+As a cloud mediation layer, Multicloud could be invoked by many projects, through this feature, Multicloud will provide
+VM status/events check and also can customize the type of event which user would like to receive. There are some
+kinds of VM status can be chosen: DELETE, PAUSE, POWER_OFF, REBUILD，SHUT_DOWN, SOFT_DELETE, etc.. In VMware VIO Plugin,
+once any change of VM status is detected of a given type, Multicloud will catch the event and throw it to DMaaP.
+Other projects can try this way of getting VM status messages in the future. Also, for other Multicloud plugin providers,
+due to some issues, there will be rest apis for them to grab the VM status messages.
+
+The APP-C won't be impacted since APP-C can still call the existing API which implemented in Amsterdam Release
+ and the API is an existing API
+
+Use Cases
+===================
+
+In VIO, one typical use case is to allow VIO users to fetch messages from DMaaP, this will provide a easier way for fetching status of
+VMs, it may drastically reduce the time of close loop, for other Multicloud plugin providers, Multicloud will provide a set of
+rest apis to get them
+
+
+Proposed change
+===================
+
+In VIO plugin:
+
+The proposed change will include two parts: * listener: to listen the events of the status change of VM, for others it
+will have rest apis to get the messages * publisher: to throw the event to DMaaP.The message we try to send is something like this:
+{
+ "state_description": "powering-off",
+ "availability_zone": "nova",
+ "terminated_at": "",
+ "ephemeral_gb": 0,
+ "instance_type_id": 5,
+ "deleted_at": "",
+ "reservation_id": "r-pvx3l6s2",
+ "memory_mb": 2048,
+ "display_name": "VM1",
+ "hostname": "vm1",
+ "state": "active",
+ "progress": "",
+ "launched_at": "2018-03-07T05:59:46.000000",
+ "metadata": {},
+ "node": "domain-c202.22bfc05c-da55-4ba6-ba93-08d9a067138e",
+ "ramdisk_id": "",
+ "access_ip_v6": null,
+ "disk_gb": 20,
+ "access_ip_v4": null,
+ "kernel_id": "",
+ "host": "compute01",
+ "user_id": "aa90efa5c84c4084b39094da952e0bd1",
+ "image_ref_url": "http://10.154.9.172:9292/images/207b9b7c-9450-4a95-852b-0d6d41f35d24",
+ "cell_name": "",
+ "root_gb": 20,
+ "tenant_id": "943ecb804cdf4103976b8a02cea12fdb",
+ "created_at": "2018-03-07 05:58:01+00:00",
+ "instance_id": "4f398943-7d39-4119-8058-74768d6dfa52",
+ "instance_type": "m1.small",
+ "vcpus": 1,
+ "image_meta": {
+ "is_copying": "1",
+ "container_format": "bare",
+ "min_ram": "0",
+ "vmware_disktype": "streamOptimized",
+ "disk_format": "vmdk",
+ "source_type": "url",
+ "image_url": "https://cloud-images.ubuntu.com/releases/14.04/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img",
+ "vmware_adaptertype": "lsiLogic",
+ "min_disk": "20",
+ "base_image_ref": "207b9b7c-9450-4a95-852b-0d6d41f35d24"
+ },
+ "architecture": null,
+ "os_type": null,
+ "instance_flavor_id": "2"
+}
+
+The eventual work flow looks like as follows:
+
+ +------------------+
+ | |
+ | Multicloud |
+ | Broker |
+ | |
+ +---------+--------+
+ |
+ |
+ V
+ +-----------------------+ +------------------+
+ | Multicloud VIO Plugin |----------->| Dmaap |
+ | | Event | |
+ +--------|-----^--------+ +------------------+
+ Oslo | |
+ Listener | |
+ V |
+ +----------------------+
+ | VIO |
+ +----------------------+
+
+
+In Other Plugins:
+
+Since the security rules of VIMs and network connectivity issues, other multicloud plugins won't be suitable for the
+oslo notification listener, so we will provide rest apis for them, the specific implementation will be dicided by them
+
+Input of <vim_id>/check_vim_status will be
+
+::
+ {
+ "states": array // the set of VIM status which user wants to get
+ }
+
+Output of check_vim_status will be
+
+::
+ {
+ "state_description": string // VIM's state
+ "launched_at": string // time of state change
+ }
+
+The work flow looks like as follows:
+
+ +------------------+
+ | |
+ | Multicloud |
+ | Broker |
+ | |
+ +---------+--------+
+ |
+ |
+ V
+ +-----------------------+
+ | Multicloud Plugins |
+ | |
+ +--------|-----^--------+
+ polling | |
+ or other way | |
+ V |
+ +----------------------+
+ | Openstack |
+ +----------------------+
--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+.. Copyright (c) 2017-2018 VMware, Inc.
+
+
+=================
+Image Service
+=================
+
+Because Multicloud provides a cloud mediation layer supporting multiple clouds. It's necessary to
+introduces some function enhancements in it. Image Service could let user upload/download images
+in a convinient way just by using Multicloud.
+
+
+Problem Description
+===================
+
+The original functions which Multicloud possesses are to use urls to upload images, while in this
+spec we intend to upload images as raw file which means it has to store a copy in Multicloud then
+upload the images to the backend openstack. So this spec is to extend multicloud to support
+download/upload images as raw file rather than a through a url
+
+
+Use Cases
+===================
+
+One typical use case is to allow users to upload/download images by Multicloud
+
+
+Proposed change
+===================
+
+The proposed change mainly means introducing glance python APIs to enable multicloud support openstack
+image service. This feature needs two changes: Upload API to import an image to backend OpenStack
+and the image that just imported can be queried from MultiCloud. Download API to download an image
+from backend Openstack and the image can be downloaded from MultiCloud.
+
+The eventual work flow looks like as follows:
+
+ user request to upload image
+ |
+ V
+ +------------------+
+ | |
+ | image file(iso, |
+ | vmdk... ) |
+ | |
+ +---------+--------+
+ |
+ |
+ |
+ +-----------|----------+
+ | multicloud| |
+ | V |
+ | +------------------+ |
+ | | image service API| |
+ | +---------+--------+ |
+ +-----------|----------+
+ | glance
+ |
+ V
+ +----------------------+
+ | openstack |
+ +----------------------+
+
+The APIs look like this:
+
+upload:
+
+Input of /{vimid}/{tenantid}/images/file will be
+
+::
+ required: image file
+ {
+ "imageType": string, // image type: ami, ari, aki, vhd, vhdx, vmdk, raw, qcow2, vdi, iso
+ "containerFormat": string, // image container format: ami, ari, aki, bare, ovf, ova, docker
+ "visibility": string, // public, private, shared, or community
+ "properties": arrary // list of properties
+ }
+
+Output of upload_image will be
+
+::
+ "responses": {
+ "201": {
+ "description": "upload successfully",
+ },
+ "404": {
+ "description": "the vim id or tenant UUID is wrong"
+ },
+ "500": {
+ "description": "the vim image is not accessable"
+ }
+
+download:
+
+Input of /{vimid}/{tenantid}/images/file/{imageid} will be
+
+::
+ {
+ "imagepath": string, // the path of the downloaded image
+ "properties": arrary // list of properties
+ }
+
+Output of download_image will be
+
+::
+ "responses": {
+ "200": {
+ "description": "download successfully",
+ },
+ "404": {
+ "description": "the vim id or tenant UUID is wrong"
+ },
+ "500": {
+ "description": "the vim image is not accessable"
+ }
--- /dev/null
+..
+ This work is licensed under a Creative Commons Attribution 4.0
+ International License.
+
+===============================================
+Parallelism improvement of Multi Cloud Services
+===============================================
+
+
+Problem Description
+===================
+
+Multi-Cloud runs Django by using Django's built-in webserver currently.
+According to Django Document[Django_Document]_, this mode should not be used
+in production. This mode has not gone through security audits or performance
+tests, and should only be used in development. From test on local computer,
+this mode can only handle ONE API request at one time. This can not meet the
+performance requirement.
+
+.. [Django_Document] https://docs.djangoproject.com/en/dev/ref/django-admin/#runserver
+
+Although security and scalability might be improved as the side effect of
+resolving the performance issue, this spec will only focus on how to improve
+the parallelism(performance) of current MultiCloud API framework.
+
+Possible Solutions
+==================
+
+Solution 1
+----------
+
+Django is a mature framework. And it has its own way to improve parallelism.
+Instead of running Django's build-in webserver, Django APP can be deployed in
+some dedicated web server. Django’s primary deployment platform is WSGI[django_deploy]_,
+the Python standard for web servers and applications.
+
+.. [django_deploy] https://docs.djangoproject.com/en/2.0/howto/deployment/wsgi/
+
+
+But on the other side, Danjgo is very huge. And Django is a black box if one
+doesn't have good knowledge of it. Adding feature based on Django may be
+time-consuming. For example, the unit test[unit_test]_ of Multi-Cloud can't use
+regular python test library because of Django. The unit test has to base on
+Django's test framework. When we want to improve the parallelism of Multi-Cloud
+services, we need to find out how Django can implement it, instead of using some
+common method.
+
+.. [unit_test] https://gerrit.onap.org/r/#/c/8909/
+
+Besides, Django's code pattern is too much like web code. And, most famous use
+cases of Django are web UI. Current code of Multi-Cloud puts many logic in
+files named `views.py`, but actually there is no view to expose. It is confusing.
+
+The benefit of this solution is that most current code needs no change.
+
+Solution 2
+----------
+
+Given the fact that Django has shortcomings to move on, this solution propose
+to use a alternative framework. Eventlet[Eventlet]_ with Pecan[Pecan]_ will be the
+idea web framework in this case, because it is lightweight, lean and widely
+used.
+
+.. [Eventlet] http://eventlet.net/doc/modules/wsgi.html
+
+.. [Pecan] https://pecan.readthedocs.io/en/latest/
+
+For example, most OpenStack projects use such framework. This framework is so
+thin that it can provide flexibility for future architecture design.
+
+However, it needs to change existing code of API exposing.
+
+
+Performance Test Comparison
+===========================
+
+Test Environment
+----------------
+
+Apache Benchmark is used as test tool. It is shipped with Ubuntu, if you
+don’t find it, just run “sudo apt install -y apache2-utils”
+
+2 Virtual Machine with Ubuntu1604. Virtual Machines are hosted in a multi-core
+hardware server. One VM is for Apache Benchmark. This VM is 1 CPU core, 8G mem.
+The other VM is for Multicloud. The VM is 4 CPU core, 6G mem.
+
+Test Command
+~~~~~~~~~~~~
+
+`ab -n <num of total requests> -c <concurrency level> http://<IP:port>/api/multicloud/v0/vim_types`
+
+Test result
+-----------
+
+It should be noted that data may vary in different test run, but overall result is
+similar as below.
+
+100 requests, concurrency level 1
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Command: `ab -n 100 -c 1 http://<IP:port>/api/multicloud/v0/vim_types`
+Result:
+ Django runserver: total takes 0.512 seconds, all requests success
+ Django+uwsgi: totally takes 0.671 seconds, all requests success.
+ Pecan+eventlet: totally takes 0.149 seconds, all requests success.
+
+10000 requests, concurrency level 100
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Command: `ab -n 10000 -c 100 http://<IP:port>/api/multicloud/v0/vim_types`
+Result:
+ Django runserver: total takes 85.326 seconds, all requests success
+ Django+uwsgi: totally takes 3.808 seconds, all requests success.
+ Pecan+eventlet: totally takes 3.181 seconds, all requests success.
+
+100000 requests, concurrency level 1000
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Command: `ab -n 100000 -c 1000 http://<IP:port>/api/multicloud/v0/vim_types`
+Result:
+ Django runserver: Apache Benchmark quit because it reports timeout after
+ running a random portion of all requests.
+ Django+uwsgi: totally takes 37.316 seconds, about 32% requests fail. I see
+ some error says that tcp socket open too many.
+ Pecan+eventlet: totally takes 35.315 seconds, all requests success.
+
+Proposed Change
+===============
+
+Given the test result above, this spec proposes to use solution 2. Based on
+the consideration of Elastic API exposure[jira_workitem]_, Multi-Cloud will
+provide a new way to expose its API. That is to say, existing code of API
+exposing needs rewrite in [jira_workitem]_. So the disadvantage of solution
+2 doesn't exist.
+
+.. [jira_workitem] https://jira.onap.org/browse/MULTICLOUD-152
+
+To define a clear scope of this spec, VoLTE is the use case that will be used
+to perform test to this spec. All functionality that VoLTE needed should be
+implemented in this spec and [jira_workitem]_.
+
+Backward compatibility
+----------------------
+
+This spec will NOT change current API. This spec will NOT replace the current
+API framework in R2, nor will switch to new API framework in R2. Instead,
+this spec will provide a configuration option, named `web_framework`, to make
+sure use case and functionalities not be broken. Default value of the
+configuration will BE `django`, which will still run current Django API
+framework. An alternative value is `pecan`, which will run the API framework
+proposed in this spec. So users don't care about the change won't be
+affected.
+
+WSGI Server
+-----------
+
+No matter what API framework will be used, a WSGI Server needs to be provided.
+This spec will use Eventlet WSGI server. API framework will be run as an
+application in WSGI server.
+
+Multi processes framework
+-------------------------
+
+This spec proposes to run Multi-Cloud API server in multiple processes mode.
+Multi-process can provide parallel API handlers. So, when multiple API
+requests come to Multi-Cloud, they can be handled simultaneously. On the other
+hand, different processes can effectively isolate different API request. So
+that, one API request will not affect another.
+
+Managing multiple processes could be overwhelming difficult and sometimes
+dangerous. Some mature library could be used to reduce related work here, for
+example oslo.service[oslo_service]_. Since oslo is used by all OpenStack
+projects for many releases, and oslo project is actively updated, it can be
+seen as a stable library.
+
+.. [oslo_service] https://github.com/openstack/oslo.service
+
+Number of processes
+~~~~~~~~~~~~~~~~~~~
+
+To best utilize multi-core CPU, the number of processes will be set to the
+number of CPU cores by default.
+
+Shared socket file
+~~~~~~~~~~~~~~~~~~
+
+To make multiple processes work together and provide a unified port number,
+multiple processes need to share a socket file. To achieve this, a bootstrap
+process will be started and will initialize the socket file. Other processes
+can be forked from this bootstrap process.
+
+Work Items
+==========
+
+#. Add WSGI server.
+#. Run Pecan application in WSGI server.
+#. Add multiple processes support.
+#. Update deploy script to support new API framework.
+
--- /dev/null
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from pecan import rest
+
+from multivimbroker.api_v2.api_router import v0_controller
+
+
+class MultiCloudController(rest.RestController):
+ v0 = v0_controller.V0_Controller()
+
+
+class APIController(rest.RestController):
+ multicloud = MultiCloudController()
+
+
+class RootController(object):
+ api = APIController()
--- /dev/null
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import pecan
+
+from multivimbroker.pub import exceptions
+from multivimbroker.pub.utils import restcall
+from multivimbroker.pub.utils import syscomm
+from multivimbroker.swagger import utils
+
+
+logger = logging.getLogger(__name__)
+
+# TODO: Move to a constant file.
+REGISTRY_URI = "registry"
+UNREGISTRY_URI = ""
+IDENTITY_URI = "identity/v3"
+IDENTITY_AUTH_URI = "identity/v3/auth/tokens"
+
+
+class V0_Controller(object):
+
+ @pecan.expose('json')
+ def vim_types(self):
+ return syscomm.getVIMTypes()
+
+ @pecan.expose('json', route="swagger.json")
+ def swagger_json(self):
+ return utils.get_swagger_json_data()
+
+ def _filter_illegal_uri(self, uri, method):
+ """
+ Filter unsupported actions, so they can be stopped at begginning.
+ """
+
+ if uri == REGISTRY_URI and method != "POST":
+ pecan.abort(405)
+
+ if uri == UNREGISTRY_URI and method != "DELETE":
+ pecan.abort(405)
+
+ if (uri in (IDENTITY_URI, IDENTITY_AUTH_URI) and
+ method not in ("POST", "GET")):
+ pecan.abort(405)
+
+ @pecan.expose()
+ def _route(self, remainder, request):
+ uri = "/".join(remainder[1:])
+ method = request.method
+ self._filter_illegal_uri(uri, method)
+
+ return self.forwarder, remainder
+
+ @pecan.expose('json')
+ def forwarder(self, *remainder, **kwargs):
+ """ Forward any requests that don't have a specific match """
+
+ # TODO(xiaohhui): Add validator for vim_id.
+ vim_id = remainder[0]
+ request = pecan.request
+ try:
+ vim_url = syscomm.getMultivimDriver(vim_id,
+ full_path=request.path)
+
+ # NOTE: Not sure headers should be set here. According to original
+ # code, headers are discarded.
+ retcode, content, status_code, resp = restcall.req_by_msb(
+ vim_url, request.method, content=request.body)
+ except exceptions.NotFound as e:
+ pecan.abort(404, detail=str(e))
+ except Exception as e:
+ pecan.abort(500, detail=str(e))
+
+ if retcode:
+ # Execptions are handled within req_by_msb
+ logger.error("Status code is %s, detail is %s.",
+ status_code, content)
+ response = pecan.Response(body=content, status=status_code)
+
+ for k in syscomm.getHeadersKeys(resp):
+ response.headers[k] = resp[k]
+
+ return response
--- /dev/null
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import pecan
+
+
+def setup_app(config=None):
+ app_conf = {
+ 'root': "multivimbroker.api_v2.api_router.root.RootController",
+ 'modules': ["multivimbroker.api_v2"],
+ 'debug': True,
+ # NOTE: By default, guess_content_type_from_ext is True, and Pecan will
+ # strip the file extension from url. For example, ../../swagger.json
+ # will look like ../../swagger to Pecan API router. This makes other
+ # url like ../../swagger.txt get the same API route. Set this to False
+ # to do strict url mapping.
+ 'guess_content_type_from_ext': False
+ }
+ app = pecan.make_app(
+ app_conf.pop('root'),
+ **app_conf
+ )
+
+ return app
--- /dev/null
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+from oslo_concurrency import processutils
+from oslo_config import cfg
+from oslo_service import service
+from oslo_service import wsgi
+
+from multivimbroker.api_v2 import app
+
+
+CONF = cfg.CONF
+
+
+class WSGIService(service.ServiceBase):
+ """Provides ability to launch API from wsgi app."""
+
+ def __init__(self):
+ self.app = app.setup_app()
+
+ self.workers = processutils.get_worker_count()
+
+ self.server = wsgi.Server(
+ CONF,
+ "multivimbroker",
+ self.app,
+ # TODO(xiaohhui): these should be configurable.
+ host="0.0.0.0",
+ port="9002",
+ use_ssl=False
+ )
+
+ def start(self):
+ self.server.start()
+
+ def stop(self):
+ self.server.stop()
+
+ def wait(self):
+ self.server.wait()
+
+ def reset(self):
+ self.server.reset()
from django.conf.urls import url
from rest_framework.urlpatterns import format_suffix_patterns
+from multivimbroker.forwarder.views import CheckCapacity
from multivimbroker.forwarder.views import Extension
from multivimbroker.forwarder.views import Forward
from multivimbroker.forwarder.views import Identity
urlpatterns = [
url(r'^api/multicloud/v0/vim_types$',
VIMTypes.as_view()),
+ url(r'^api/multicloud/v0/check_vim_capacity$',
+ CheckCapacity.as_view()),
url(r'^api/multicloud/v0/(?P<vimid>[0-9a-zA-Z_-]+)/identity/v3$',
Identity.as_view()),
url(r'^api/multicloud/v0/(?P<vimid>[0-9a-zA-Z_-]+)/identity/v3'
# See the License for the specific language governing permissions and
# limitations under the License.
-import os
import json
from rest_framework.views import APIView
from rest_framework.views import Response
from rest_framework.views import status
from multivimbroker.forwarder.base import BaseHandler
-
-#
+from multivimbroker.pub.utils.syscomm import originHeaders
+from multivimbroker.pub.utils import syscomm
class BaseServer(BaseHandler, APIView):
def get(self, request, vimid):
- return self.send(vimid, request.get_full_path(), request.body, "GET")
+ return self.send(vimid, request.get_full_path(), request.body, "GET",
+ headers=originHeaders(request))
def post(self, request, vimid):
- return self.send(vimid, request.get_full_path(), request.body, "POST")
+ return self.send(vimid, request.get_full_path(), request.body, "POST",
+ headers=originHeaders(request))
class Registry(BaseServer):
def post(self, request, vimid):
- return self.send(vimid, request.get_full_path(), request.body, "POST")
+ return self.send(vimid, request.get_full_path(), request.body, "POST",
+ headers=originHeaders(request))
class UnRegistry(BaseServer):
def delete(self, request, vimid):
return self.send(vimid, request.get_full_path(), request.body,
- "DELETE")
+ "DELETE", headers=originHeaders(request))
class Extension(BaseServer):
def get(self, request, vimid):
- return self.send(vimid, request.get_full_path(), request.body, "GET")
+ return self.send(vimid, request.get_full_path(), request.body, "GET",
+ headers=originHeaders(request))
class VIMTypes(BaseServer):
def get(self, request):
- # Fix here unless we have plugin registry
- json_file = os.path.join(os.path.dirname(__file__),
- '../pub/config/provider-plugin.json')
- with open(json_file, "r") as f:
- plugins = json.load(f)
- ret = []
- for k, v in plugins.items():
- item = {}
- item["vim_type"] = v.get("vim_type")
- item["versions"] = [k for k in v.get('versions', {})]
- ret.append(item)
+ return Response(data=syscomm.getVIMTypes(), status=status.HTTP_200_OK)
+
+
+class CheckCapacity(BaseServer):
+
+ def post(self, request):
+ try:
+ body = json.loads(request.body)
+ except ValueError as e:
+ return Response(
+ data={'error': 'Invalidate request body %s.' % e},
+ status=status.HTTP_400_BAD_REQUEST)
+
+ ret = {"VIMs": []}
+ newbody = {
+ "vCPU": body.get("vCPU", 0),
+ "Memory": body.get("Memory", 0),
+ "Storage": body.get("Storage", 0)
+ }
+ for vim in body.get("VIMs", []):
+ url = request.get_full_path().replace(
+ "check_vim_capacity", "%s/capacity_check" % vim)
+ resp = self.send(vim, url, newbody, "POST")
+ if resp.status_code != status.HTTP_200_OK:
+ continue
+ try:
+ resp_body = json.loads(resp.body)
+ except ValueError:
+ continue
+ if not resp_body.get("result", False):
+ continue
+ ret['VIMs'].append(vim)
return Response(data=ret, status=status.HTTP_200_OK)
--- /dev/null
+# Copyright (c) 2017-2018 VMware, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+
+
+import uuid
+from onaplogging.mdcContext import MDC
+from multivimbroker.pub.config.config import SERVICE_NAME
+from multivimbroker.pub.config.config import FORWARDED_FOR_FIELDS
+
+
+class LogContextMiddleware(object):
+
+ # the last IP behind multiple proxies, if no exist proxies
+ # get local host ip.
+ def _getLastIp(self, request):
+
+ ip = ""
+ try:
+ for field in FORWARDED_FOR_FIELDS:
+ if field in request.META:
+ if ',' in request.META[field]:
+ parts = request.META[field].split(',')
+ ip = parts[-1].strip().split(":")[0]
+ else:
+ ip = request.META[field].split(":")[0]
+
+ if ip == "":
+ ip = request.META.get("HTTP_HOST").split(":")[0]
+
+ except Exception:
+ pass
+
+ return ip
+
+ def process_request(self, request):
+
+ # Fetch TRANSACTIONID Id and pass to plugin server
+ ReqeustID = request.META.get("HTTP_X_TRANSACTIONID", None)
+ if ReqeustID is None:
+ ReqeustID = uuid.uuid3(uuid.NAMESPACE_URL, SERVICE_NAME)
+ request.META["HTTP_X_TRANSACTIONID"] = ReqeustID
+ MDC.put("requestID", ReqeustID)
+ # generate the unique id
+ InovocationID = uuid.uuid3(uuid.NAMESPACE_DNS, SERVICE_NAME)
+ MDC.put("invocationID", InovocationID)
+ MDC.put("serviceName", SERVICE_NAME)
+ # access ip
+ MDC.put("serviceIP", self._getLastIp(request))
+
+ return None
+
+ def process_response(self, request, response):
+
+ MDC.clear()
+ return response
AAI_ADDR = "aai.api.simpledemo.openecomp.org"
AAI_PORT = "8443"
AAI_SERVICE_URL = 'https://%s:%s/aai' % (AAI_ADDR, AAI_PORT)
-AAI_SCHEMA_VERSION = "v11"
+AAI_SCHEMA_VERSION = "v13"
AAI_USERNAME = 'AAI'
AAI_PASSWORD = 'AAI'
+# [MDC]
+SERVICE_NAME = "multicloud-broker"
+FORWARDED_FOR_FIELDS = ["HTTP_X_FORWARDED_FOR", "HTTP_X_FORWARDED_HOST",
+ "HTTP_X_FORWARDED_SERVER"]
+
# [IMAGE LOCAL PATH]
ROOT_PATH = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-# [register]
-REG_TO_MSB_WHEN_START = False
-REG_TO_MSB_REG_URL = "/api/microservices/v1/services"
-REG_TO_MSB_REG_PARAM = {
- "serviceName": "multicloud",
- "version": "v0",
- "url": "/api/multicloud/v0",
- "protocol": "REST",
- "visualRange": "1",
- "nodes": [{
- "ip": "127.0.0.1",
- "port": "9001",
- "ttl": 0
- }]
-}
class: "logging.handlers.RotatingFileHandler"
filename: "/var/log/onap/multicloud/multivimbroker/multivimbroker.log"
formatter: "mdcFormat"
- maxBytes: 1024*1024*50
+ maxBytes: 52428800
backupCount: 10
formatters:
standard:
- format: "%(asctime)s:[%(name)s]:[%(filename)s]-[%(lineno)d] [%(levelname)s]:%(message)s"
+ format: "%(asctime)s|||||%(name)s||%(thread)||%(funcName)s||%(levelname)s||%(message)s"
mdcFormat:
- format: "%(asctime)s:[%(name)s]:[%(filename)s]-[%(lineno)d] [%(levelname)s]:[%(mdc)s]: %(message)s"
- mdcfmt: "{requestID}"
+ format: "%(asctime)s|||||%(name)s||%(thread)s||%(funcName)s||%(levelname)s||%(message)s||||%(mdc)s \t"
+ mdcfmt: "{requestID} {invocationID} {serviceName} {serviceIP}"
datefmt: "%Y-%m-%d %H:%M:%S"
(): onaplogging.mdcformatter.MDCFormatter
return [header for header in response.keys() if header not in hopbyhop]
+# trim out 'HTTP_' prefix part and replace "_" wiht "-".
+def originHeaders(request):
+ regex = re.compile('^HTTP_')
+ return dict((regex.sub('', header).replace("_", "-"), value)
+ for (header, value) in request.META.items()
+ if header.startswith('HTTP_'))
+
+
def findMultivimDriver(vim=None):
json_file = os.path.join(os.path.dirname(__file__),
'../config/provider-plugin.json')
vim = get_vim_by_id(vimid)
multclouddriver = findMultivimDriver(vim=vim)
return re.sub(multcloud, multclouddriver, full_path)
+
+
+def getVIMTypes():
+ # Fix here unless we have plugin registry
+ json_file = os.path.join(os.path.dirname(__file__),
+ '../config/provider-plugin.json')
+ with open(json_file, "r") as f:
+ plugins = json.load(f)
+ ret = []
+ for k, v in plugins.items():
+ item = {}
+ item["vim_type"] = v.get("vim_type")
+ item["versions"] = [k for k in v.get('versions', {})]
+ ret.append(item)
+
+ return ret
--- /dev/null
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import eventlet
+eventlet.monkey_patch()
+
+from oslo_config import cfg # noqa
+from oslo_service import service # noqa
+import sys # noqa
+
+from multivimbroker.api_v2 import service as api_service # noqa
+
+
+def main():
+ try:
+ api_server = api_service.WSGIService()
+ launcher = service.launch(cfg.CONF,
+ api_server,
+ workers=api_server.workers)
+ launcher.wait()
+ except RuntimeError as excp:
+ sys.stderr.write("ERROR: %s\n" % excp)
+ sys.exit(1)
+
+
+if __name__ == '__main__':
+ main()
SECRET_KEY = '3o-wney!99y)^h3v)0$j16l9=fdjxcb+a8g+q3tfbahcnu2b0o'
# SECURITY WARNING: don't run with debug turned on in production!
-DEBUG = True
+# DEBUG = True
-ALLOWED_HOSTS = []
+ALLOWED_HOSTS = ['*']
# Application definition
'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
+ 'multivimbroker.middleware.LogContextMiddleware',
]
ROOT_URLCONF = 'multivimbroker.urls'
if 'test' in sys.argv:
from multivimbroker.pub.config import config
- config.REG_TO_MSB_WHEN_START = False
DATABASES = {}
DATABASES['default'] = {
'ENGINE': 'django.db.backends.sqlite3',
--- /dev/null
+# Copyright (c) 2017 Wind River Systems, Inc.
+# Copyright (c) 2017-2018 VMware, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+
+import json
+import os
+
+
+def get_swagger_json_data():
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.flavor.swagger.json')
+ f = open(json_file)
+ json_data = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.image.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.network.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.subnet.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.server.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.volume.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.vport.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.tenant.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.host.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.limit.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+ json_file = os.path.join(os.path.dirname(__file__),
+ 'multivim.identity.swagger.json')
+ f = open(json_file)
+ json_data_temp = json.JSONDecoder().decode(f.read())
+ f.close()
+ json_data["paths"].update(json_data_temp["paths"])
+ json_data["definitions"].update(json_data_temp["definitions"])
+
+ return json_data
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-import json
-import logging
-import os
-# import traceback
# from rest_framework import status
from rest_framework.response import Response
from rest_framework.views import APIView
-# from multivimbroker.pub.exceptions import VimBrokerException
-
-logger = logging.getLogger(__name__)
+from multivimbroker.swagger import utils
class SwaggerJsonView(APIView):
def get(self, request):
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.flavor.swagger.json')
- f = open(json_file)
- json_data = json.JSONDecoder().decode(f.read())
- f.close()
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.image.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.network.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.subnet.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.server.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.volume.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.vport.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.tenant.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.host.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.limit.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- #
- json_file = os.path.join(os.path.dirname(__file__),
- 'multivim.identity.swagger.json')
- f = open(json_file)
- json_data_temp = json.JSONDecoder().decode(f.read())
- f.close()
- json_data["paths"].update(json_data_temp["paths"])
- json_data["definitions"].update(json_data_temp["definitions"])
- return Response(json_data)
+ return Response(utils.get_swagger_json_data())
--- /dev/null
+# Copyright (c) 2017-2018 VMware, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+
+import mock
+import unittest
+
+from rest_framework import status
+
+from multivimbroker.forwarder.views import CheckCapacity
+
+
+class CheckCapacityTest(unittest.TestCase):
+
+ def setUp(self):
+ self.view = CheckCapacity()
+ super(CheckCapacityTest, self).setUp()
+
+ def tearDown(self):
+ pass
+
+ def test_check_capacity_success(self):
+ req = mock.Mock()
+ req.body = """
+ {
+ "vCPU": 1,
+ "Memory": 1,
+ "Storage": 500,
+ "VIMs": ["openstack_RegionOne"]
+ }"""
+ req.get_full_path.return_value = ("http://msb.onap.org/api/multicloud"
+ "/v0/check_vim_capacity")
+ with mock.patch.object(self.view, "send") as send:
+ plugin_resp = mock.Mock()
+ plugin_resp.body = """{
+ "result": true
+ }"""
+ plugin_resp.status_code = status.HTTP_200_OK
+ send.return_value = plugin_resp
+
+ resp = self.view.post(req)
+ expect_body = {
+ "VIMs": ["openstack_RegionOne"]
+ }
+ self.assertEqual(status.HTTP_200_OK, resp.status_code)
+ self.assertDictEqual(expect_body, resp.data)
+
+ def test_check_capacity_no_suitable_vim(self):
+ req = mock.Mock()
+ req.body = """
+ {
+ "vCPU": 1,
+ "Memory": 1,
+ "Storage": 500,
+ "VIMs": ["openstack_RegionOne"]
+ }"""
+ req.get_full_path.return_value = ("http://msb.onap.org/api/multicloud"
+ "/v0/check_vim_capacity")
+ with mock.patch.object(self.view, "send") as send:
+ plugin_resp = mock.Mock()
+ plugin_resp.body = """{
+ "result": false
+ }"""
+ plugin_resp.status_code = status.HTTP_200_OK
+ send.return_value = plugin_resp
+
+ resp = self.view.post(req)
+ expect_body = {
+ "VIMs": []
+ }
+ self.assertEqual(status.HTTP_200_OK, resp.status_code)
+ self.assertDictEqual(expect_body, resp.data)
+
+ def test_check_capacity_invalid_input(self):
+ req = mock.Mock()
+ req.body = "hello world"
+ req.get_full_path.return_value = ("http://msb.onap.org/api/multicloud"
+ "/v0/check_vim_capacity")
+ expect_body = {
+ "error": ("Invalidate request body "
+ "No JSON object could be decoded.")
+ }
+ resp = self.view.post(req)
+ self.assertEqual(status.HTTP_400_BAD_REQUEST, resp.status_code)
+ self.assertDictEqual(expect_body, resp.data)
--- /dev/null
+# Copyright (c) 2017-2018 VMware, Inc.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+
+import unittest
+
+from multivimbroker.pub.utils import restcall
+
+
+class TestRestCall(unittest.TestCase):
+
+ def test_combine_url(self):
+ url = ["http://a.com/test/", "http://a.com/test/",
+ "http://a.com/test"]
+ res = ["/resource", "resource", "/resource"]
+ expected = "http://a.com/test/resource"
+ for i in range(len(url)):
+ self.assertEqual(expected, restcall.combine_url(url[i], res[i]))
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-import json
import mock
import unittest
-from multivimbroker.pub.config import config
-from multivimbroker import urls
+from rest_framework.views import status
+
+from multivimbroker.forwarder.views import VIMTypes
class TestUrls(unittest.TestCase):
+ def setUp(self):
+ self.view = VIMTypes()
- def test_request_msb(self):
- with mock.patch("multivimbroker.pub.utils.restcall."
- "req_by_msb") as req_by_msb:
- urls.req_msb(True)
- req_by_msb.assert_called_once_with(
- config.REG_TO_MSB_REG_URL, "POST",
- json.JSONEncoder().encode(config.REG_TO_MSB_REG_PARAM))
+ def test_vim_types_success(self):
+ resp = self.view.get(mock.Mock())
+ self.assertEqual(status.HTTP_200_OK, resp.status_code)
+ self.assertEqual(2, len(resp.data))
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
from django.conf.urls import include, url
-import json
-
-from multivimbroker.pub.config import config
-
urlpatterns = [
url(r'^', include('multivimbroker.swagger.urls')),
url(r'^', include('multivimbroker.forwarder.urls')),
]
-
-
-def req_msb(request_when_start):
- # regist to MSB when startup
- if request_when_start:
- from multivimbroker.pub.utils.restcall import req_by_msb
- req_by_msb(config.REG_TO_MSB_REG_URL, "POST",
- json.JSONEncoder().encode(config.REG_TO_MSB_REG_PARAM))
-
-
-req_msb(config.REG_TO_MSB_WHEN_START)
unittest_xml_reporting==1.12.0
# for onap logging
-onappylog>=1.0.5
\ No newline at end of file
+onappylog>=1.0.6
+
+# for pecan framework
+pecan>=1.2.1
+oslo.concurrency>=3.21.0
+oslo.config>=4.11.0
+oslo.service>=1.25.0
+eventlet>=0.20.0
+
+# uwsgi for parallel processing
+uwsgi
+
mkdir -p $logDir
fi
-nohup python manage.py runserver 0.0.0.0:9001 2>&1 &
+if [ "$WEB_FRAMEWORK" == "pecan" ]
+then
+ python multivimbroker/scripts/api.py
+else
+ # nohup python manage.py runserver 0.0.0.0:9001 2>&1 &
+ nohup uwsgi --http :9001 --module multivimbroker.wsgi --master --processes 4 &
-while [ ! -f $logDir/multivimbroker.log ]; do
- sleep 1
-done
+ while [ ! -f $logDir/multivimbroker.log ]; do
+ sleep 1
+ done
-tail -F $logDir/multivimbroker.log
+ tail -F $logDir/multivimbroker.log
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-ps auxww | grep 'manage.py runserver 0.0.0.0:9001' | awk '{print $2}' | xargs kill -9
+# ps auxww | grep 'manage.py runserver 0.0.0.0:9001' | awk '{print $2}' | xargs kill -9
+ps auxww |grep 'uwsgi --http :9001 --module multivimbroker.wsgi --master' |awk '{print $2}' |xargs kill -9
<parent>
<groupId>org.onap.oparent</groupId>
<artifactId>oparent</artifactId>
- <version>0.1.1</version>
+ <version>1.1.0</version>
<relativePath>../oparent</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Code Review / multicloud / framework.git / commitdiff