1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. _clamp-acm_architecture-label:
5 TOSCA Defined Automation Compositions: Architecture and Design
6 ##############################################################
12 The idea of using automation compositions to automatically (or autonomously) perform network management
13 has been the subject of much research in the Network Management research community, see
14 :download:`this paper <files/Acms.pdf>` for some background. However, it is only with
15 the advent of ONAP that we have a platform that supports automation compositions for network management.
16 Before ONAP, Automation Compositions have been implemented by hard-coding components together and hard
17 coding logic into components. ONAP has taken a step forward towards automatic implementation
18 of Automation Compositions by allowing parameterization of Automation Compositions that work on the premise that
19 the Automation Compositions use a set of analytic, policy, and control components connected together in
22 The goal of the work is to extend and enhance the current ONAP Automation Composition support to provide
23 a complete open-source framework for Automation Compositions. This will enhance the current support to
24 provide TOSCA based Automation Composition definition and development, commissioning and run-time management.
25 The participants that comprise a Automation Composition and the metadata needed to link the participants
26 together to create a Automation Composition are specified in a standardized way using the `OASIS TOSCA
27 modelling language <http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/>`_. The TOSCA
28 description is then used to commission, instantiate, and manage the Automation Compositions in the run
31 .. image:: images/01-acm-overview.png
36 This section describes the terminology used in the system.
38 1.1 Automation Composition Terminology
39 --------------------------------------
41 **Automation Composition Type:** A definition of a Automation Composition in the TOSCA language. This definition describes
42 a certain type of a automation composition. The life cycle of instances of a Automation Composition Type are managed
45 **Automation Composition Instance:** An instance of a Automation Composition Type. The life cycle of a Automation Composition
46 Instance is managed by CLAMP. A Automation Composition Instance is a set of executing elements on which
47 Life Cycle Management (LCM) is executed collectively. For example, a set of microservices may be
48 spawned and executed together to deliver a service. This collection of services is a automation composition.
50 **Automation Composition Element Type:** A definition of a Automation Composition Element in the TOSCA language. This
51 definition describes a certain type of Automation Composition Element for a automation composition in a Automation
54 **Automation Composition Element Instance:** A single entity executing on a participant, with its Life Cycle
55 being managed as part of the overall automation composition. For example, a single microservice that is
56 executing as one microservice in a service.
58 **CLAMP Automation Composition Runtime:** The CLAMP server that holds Automation Composition Type definitions and manages
59 the life cycle of Automation Composition Instances and their Automation Composition Elements in cooperation with
63 1.2 Participant Terminology
64 ---------------------------
66 **Participant Type:** Definition of a type of system or framework that can take part in control
67 loops and a definition of the capabilities of that participant type. A participant advertises
68 its type to the CLAMP Automation Composition Runtime.
70 **Participant:** A system or framework that takes part in automation compositions by executing Automation Composition
71 Elements in cooperation with the CLAMP Automation Composition Runtime. A participant chooses to partake
72 in automation compositions, to manage Automation Composition Elements for CLAMP, and to receive, send and act on
73 LCM messages for the CLAMP runtime.
75 1.3 Terminology for Properties
76 ------------------------------
78 **Common Properties:** Properties that apply to all Automation Composition Instances of a certain Automation
79 Composition Type and are specified when a Automation Composition Type is commissioned.
81 **Instance Specific Properties:** Properties that must be specified for each Automation Composition Instance
82 and are specified when a Automation Composition Instance is Initialized.
84 1.4 Concepts and their relationships
85 ------------------------------------
87 The UML diagram below shows the concepts described in the terminology sections above and how
88 they are interrelated.
90 .. image:: images/02-acm-concepts.png
92 The Automation Composition Definition concepts describe the types of things that are in the system. These
93 concepts are defined at design time and are passed to the runtime in a TOSCA document. The
94 concepts in the Automation Composition Runtime are created by the runtime part of the system using the
95 definitions created at design time.
102 We consider the capabilities of Automation Compositions at Design Time and Run Time.
104 At Design Time, three capabilities are supported:
106 #. **Automation Composition Element Definition Specification.** This capability allows users to define Automation
107 Composition Element Types and the metadata that can be used on and configured on a Automation Composition Element
108 Type. Users also define the Participant Type that will run the Automation Composition Element when it is
109 taking part in in a automation composition. The post condition of an execution of this capability is that
110 metadata for a Automation Composition Element Type is defined in the Automation Composition Design Time Catalogue.
112 #. **Automation Composition Element Definition Onboarding.** This capability allows external users and systems
113 (such as SDC or DCAE-MOD) to define the metadata that can be used on and configured on a Automation
114 Composition Element Type and to define the Participant Type that will run the Automation Composition Element when
115 it is taking part in in a automation composition. The post condition of an execution of this capability
116 is that metadata for a Automation Composition Element Type is defined in the Automation Composition Design Time
119 #. **Automation Composition Type Definition.** This capability allows users and other systems to create Automation
120 Composition Type definitions by specifying a set of Automation Composition Element Definitions from those that
121 are available in the Automation Composition Design Time Catalogue. These Automation Composition Elements will
122 work together to form Automation Compositions. In an execution of this capability, a user specifies the
123 metadata for the Automation Composition and specifies the set of Automation Composition Elements and their Participant
124 Types. The user also selects the correct metadata sets for each participant in the Automation Composition
125 Type and defines the overall Automation Composition Type metadata. The user also specifies the Common
126 Property Types that apply to all instances of a automation composition type and the Instance Specific
127 Property Types that apply to individual instances of a Automation Composition Type. The post condition for
128 an execution of this capability is a Automation Composition definition in TOSCA stored in the Automation Composition
129 Design Time Catalogue.
132 Once a Automation Composition Definition is commissioned to the Automation Composition Runtime and has been
133 stored in the Run Time Inventory, it cannot be further edited unless it is decommissioned.
136 At Run Time, the following participant related capabilities are supported:
138 #. **System Pre-Configuration.** This capability allows participants to register and deregister
139 with CLAMP. Participants explicitly register with CLAMP when they start. Automation Composition Priming
140 is performed on each participant once it registers. The post condition for an execution of this
141 capability is that a participant becomes available (registration) or is no longer available
142 (deregistration) for participation in a automation composition.
144 #. **Automation Composition Priming on Participants.** A participant is primed to support a Automation Composition Type.
145 Priming a participant means that the definition of a automation composition and the values of Common
146 Property Types that apply to all instances of a automation composition type on a participant are sent
147 to a participant. The participant can then take whatever actions it need to do to support
148 the automation composition type in question. Automation Composition Priming takes place at participant
149 registration and at Automation Composition Commissioning. The post condition for an execution of this
150 capability is that all participants in this automation composition type are commissioned, that is they
151 are prepared to run instances of their Automation Composition Element types.
154 At Run Time, the following Automation Composition Life Cycle management capabilities are supported:
156 #. **Automation Composition Commissioning:** This capability allows version controlled Automation Composition Type
157 definitions to be taken from the Automation Composition Design Time Catalogue and be placed in the
158 Commissioned Automation Composition Inventory. It also allows the values of Common Property Types
159 that apply to all instances of a Automation Composition Type to be set. Further, the Automation Composition
160 Type is primed on all concerned participants. The post condition for an execution of this
161 capability is that the Automation Composition Type definition is in the Commissioned Automation Composition
162 Inventory and the Automation Composition Type is primed on concerned participants.
164 #. **Automation Composition Instance Life Cycle Management:** This capability allows a Automation Composition
165 Instance to have its life cycle managed.
167 #. **Automation Composition Instance Creation:** This capability allows a Automation Composition Instance to be
168 created. The Automation Composition Type definition is read from the Commissioned Automation Composition
169 Inventory and values are assigned to the Instance Specific Property Types defined for
170 instances of the Automation Composition Type in the same manner as the existing CLAMP client does.
171 A Automation Composition Instance that has been created but has not yet been instantiated on
172 participants is in state UNINITIALIZED. In this state, the Instance Specific Property Type
173 values can be revised and updated as often as the user requires. The post condition for an
174 execution of this capability is that the Automation Composition instance is created in the
175 Instantiated Automation Composition Inventory but has not been instantiated on Participants.
177 #. **Automation Composition Instance Update on Participants:** Once the user is happy with the property
178 values, the Automation Composition Instance is updated on participants and the Automation Composition Elements
179 for this Automation Composition Instance are initialized or updated by participants using the control
180 loop metadata. The post condition for an execution of this capability is that the Automation
181 Composition instance is updated on Participants.
183 #. **Automation Composition State Change:** The user can now order the participants to change the state
184 of the Automation Composition Instance. If the Automation Composition is set to state RUNNING, each participant
185 begins accepting and processing automation composition events and the Automation Composition Instance is set
186 to state RUNNING in the Instantiated Automation Composition inventory. The post condition for an
187 execution of this capability is that the Automation Composition instance state is changed on
190 #. **Automation Composition Instance Monitoring:** This capability allows Automation Composition Instances to be
191 monitored. Users can check the status of Participants, Automation Composition Instances, and Automation
192 Composition Elements. Participants report their overall status and the status of Automation Composition
193 Elements they are running periodically to CLAMP. Clamp aggregates these status reports
194 into an aggregated Automation Composition Instance status record, which is available for monitoring.
195 The post condition for an execution of this capability is that Automation Composition Instances are
198 #. **Automation Composition Instance Supervision:** This capability allows Automation Composition Instances to be
199 supervised. The CLAMP runtime expects participants to report on Automation Composition Elements
200 periodically. The CLAMP runtime checks that periodic reports are received and that each
201 Automation Composition Element is in the state it should be in. If reports are missed or if a
202 Automation Composition Element is in an incorrect state, remedial action is taken and notifications
203 are issued. The post condition for an execution of this capability is that Automation Composition
204 Instances are being supervised by the CLAMP runtime.
206 #. **Automation Composition Instance Removal from Participants:** A user can order the removal of a Automation
207 Composition Instance from participants. The post condition for an execution of this capability is
208 that the Automation Composition instance is removed from Participants.
210 #. **Automation Composition Instance Deletion:** A user can order the removal of a Automation Composition Instance
211 from the CLAMP runtime. Automation Composition Instances that are instantiated on participants cannot
212 be removed from the CLAMP runtime. The post condition for an execution of this capability
213 is that the Automation Composition instance is removed from Instantiated Automation Composition Inventory.
215 #. **Automation Composition Decommissioning:** This capability allows version controlled Automation Composition Type
216 definitions to be removed from the Commissioned Automation Composition Inventory. A Automation Composition
217 Definition that has instances in the Instantiated Automation Composition Inventory cannot be removed.
218 The post condition for an execution of this capability is that the Automation Composition Type
219 definition removed from the Commissioned Automation Composition Inventory.
222 The system dialogues for run time capabilities are described in detail on the
223 :ref:`System Level Dialogues <system-level-label>` page.
225 .. _acm-instance-states:
227 2.1 Automation Composition Instance States
228 ------------------------------------------
230 When a automation composition definition has been commissioned, instances of the automation composition can be
231 created, updated, and deleted. The system manages the lifecycle of automation compositions and control
232 loop elements following the state transition diagram below.
234 .. image:: images/03-acm-instance-states.png
236 3 Overall Target Architecture
237 =============================
239 The diagram below shows an overview of the architecture of TOSCA based Automation Composition
242 .. image:: images/04-overview.png
244 Following the ONAP Reference Architecture, the architecture has a Design Time part and
247 The Design Time part of the architecture allows a user to specify metadata for participants.
248 It also allows users to compose automation compositions. The Design Time Catalogue contains the metadata
249 primitives and automation composition definition primitives for composition of automation compositions. As shown
250 in the figure above, the Design Time component provides a system where Automation Compositions can be
251 designed and defined in metadata. This means that a Automation Composition can have any arbitrary
252 structure and the Automation Composition developers can use whatever analytic, policy, or control
253 participants they like to implement their Automation Composition. At composition time, the user
254 parameterises the Automation Composition and stores it in the design time catalogue. This catalogue
255 contains the primitive metadata for any participants that can be used to compose a Automation
256 Composition. A Automation Composition SDK is used to compose a Automation Composition by aggregating the metadata for
257 the participants chosen to be used in a Automation Composition and by constructing the references between
258 the participants. The architecture of the Automation Composition Design Time part will be elaborated in
261 Composed Automation Compositions are commissioned on the run time part of the system, where they are
262 stored in the Commissioned Automation Composition inventory and are available for instantiation. The
263 Commissioning component provides a CRUD REST interface for Automation Composition Types, and implements
264 CRUD of Automation Composition Types. Commissioning also implements validation and persistence of incoming
265 Automation Composition Types. It also guarantees the integrity of updates and deletions of Automation Composition
266 Types, such as performing updates in accordance with semantic versioning rules and ensuring that
267 deletions are not allowed on Automation Composition Types that have instances defined.
269 The Instantiation component manages the Life Cycle Management of Automation Composition Instances and
270 their Automation Composition Elements. It publishes a REST interface that is used to create Automation Composition
271 Instances and set values for Common and Instance Specific properties. This REST interface is
272 public and is used by the CLAMP GUI. It may also be used by any other client via the public
273 REST interface. the REST interface also allows the state of Automation Composition Instances to be changed.
274 A user can change the state of Automation Composition Instances as described in the state transition
275 diagram shown in section 2 above. The Instantiation component issues update and state change
276 messages via DMaaP to participants so that they can update and manage the state of the Automation
277 Composition Elements they are responsible for. The Instantiation component also implements persistence
278 of Automation Composition Instances, automation composition elements, and their state changes.
280 The Monitoring component reads updates sent by participants. Participants report on the
281 state of their Automation Composition Elements periodically and in response to a message they have
282 received from the Instantiation component. The Monitoring component reads the contents of
283 the participant messages and persists their state updates and statistics records. It also
284 publishes a REST interface that publishes the current state of all Participants, Automation
285 Composition Instances and their Automation Composition Elements, as well as publishing Participant and
286 Automation Composition statistics.
288 The Supervision component is responsible for checking that Automation Composition Instances are correctly
289 instantiated and are in the correct state (UNINITIALIZED/READY/RUNNING). It also handles
290 timeouts and on state changes to Automation Composition Instances, and retries and rolls back state
291 changes where state changes failed.
293 A Participant is an executing component that partakes in automation compositions. More explicitly, a
294 Participant is something that implements the Participant Instantiation and Participant
295 Monitoring messaging protocol over DMaaP for Life Cycle management of Automation Composition Elements.
296 A Participant runs Automation Composition Elements and manages and reports on their life cycle
297 following the instructions it gets from the CLAMP runtime in messages delivered over DMaaP.
299 In the figure above, five participants are shown. A Configuration Persistence Participant
300 manages Automation Composition Elements that interact with the `ONAP Configuration Persistence Service
301 <https://docs.onap.org/projects/onap-cps/en/latest/overview.html>`_
302 to store common data. The DCAE Participant runs Automation Composition Elements that manage DCAE
303 microservices. The Kubernetes Participant hosts the Automation Composition Elements that are managing
304 the life cycle of microservices in automation compositions that are in a Kubernetes ecosystem. The
305 Policy Participant handles the Automation Composition Elements that interact with the Policy Framework
306 to manage policies for automation compositions. A Automation Participant such as the CDS Participant
307 runs Automation Composition Elements that load metadata and configure controllers so that they can
308 partake in automation compositions. Any third party Existing System Participant can be developed to
309 run Automation Composition Elements that interact with any existing system (such as an operator's
310 analytic, machine learning, or artificial intelligence system) so that those systems can
311 partake in automation compositions.
313 4. Other Considerations
314 =======================
316 .. _management-acm-instance-configs:
318 4.1 Management of Automation Composition Instance Configurations
319 ----------------------------------------------------------------
321 In order to keep management of versions of the configuration of automation composition instances
322 straightforward and easy to implement, the following version management scheme using
323 semantic versioning is implemented. Each configuration of a Automation Composition Instance and
324 configuration of a Automation Composition Element has a semantic version with 3 digits indicating
325 the **major.minor.patch** number of the version.
328 A **configuration** means a full set of parameter values for a Automation Composition Instance.
330 .. image:: images/05-upgrade-states.png
334 #. A Automation Composition or Automation Composition Element in state **RUNNING** can be changed to a higher patch
335 level or rolled back to a lower patch level. This means that hot changes that do not
336 impact the structure of a Automation Composition or its elements can be executed.
338 #. A Automation Composition or Automation Composition Element in state **PASSIVE** can be changed to a higher
339 minor/patch level or rolled back to a lower minor/patch level. This means that structural
340 changes to Automation Composition Elements that do not impact the Automation Composition as a whole can be
341 executed by taking the automation composition to state **PASSIVE**.
343 #. A Automation Composition or Automation Composition Element in state **UNINITIALIZED** can be changed to a higher
344 major/minor/patch level or rolled back to a lower major/minor/patch level. This means
345 that where the structure of the entire automation composition is changed, the automation composition must
346 be uninitialized and reinitialized.
348 #. If a Automation Composition Element has a **minor** version change, then its Automation Composition Instance
349 must have at least a **minor** version change.
351 #. If a Automation Composition Element has a **major** version change, then its Automation Composition Instance
352 must have a **major** version change.
357 The system is designed to be inherently scalable. The CLAMP runtime is stateless, all state
358 is preserved in the Instantiated Automation Composition inventory in the database. When the user
359 requests an operation such as an instantiation, activation, passivation, or an uninitialization
360 on a Automation Composition Instance, the CLAMP runtime broadcasts the request to participants over
361 DMaaP and saves details of the request to the database. The CLAMP runtime does not directly
362 wait for responses to requests.
364 When a request is broadcast on DMaaP, the request is asynchronously picked up by participants
365 of the types required for the Automation Composition Instance and those participants manage the life
366 cycle of its automation composition elements. Periodically, each participant reports back on the status
367 of operations it has picked up for the Automation Composition Elements it controls, together with
368 statistics on the Automation Composition Elements over DMaaP. On reception of these participant messages,
369 the CLAMP runtime stores this information to its database.
371 The participant to use on a automation composition can be selected from the registered participants
372 in either of two ways:
374 **Runtime-side Selection:** The CLAMP runtime selects a suitable participant from the list of
375 participants and sends the participant ID that should be used in the Participant Update message.
376 In this case, the CLAMP runtime decides on which participant will run the Automation Composition Element
377 based on a suitable algorithm. Algorithms could be round robin based or load based.
379 **Participant-side Selection:** The CLAMP runtime sends a list of Participant IDs that may be used
380 in the Participant Update message. In this case, the candidate participants decide among
381 themselves which participant should host the Automation Composition Element.
383 This approach makes it easy to scale Automation Composition life cycle management. As Automation Composition
384 Instance counts increase, more than one CLAMP runtime can be deployed and REST/supervision
385 operations on Automation Composition Instances can run in parallel. The number of participants can
386 scale because an asynchronous broadcast mechanism is used for runtime-participant communication
387 and there is no direct connection or communication channel between participants and CLAMP
388 runtime servers. Participant state, Automation Composition Instance state, and Automation Composition Element
389 state is held in the database, so any CLAMP runtime server can handle operations for any
390 participant. Because many participants of a particular type can be deployed and participant
391 instances can load balance automation composition element instances for different Automation Composition Instances
392 of many types across themselves using a mechanism such as a Kubernetes cluster.
395 4.3 Sandboxing and API Gateway Support
396 --------------------------------------
398 At runtime, interaction between ONAP platform services and application microservices are
399 relatively unconstrained, so interactions between Automation Composition Elements for a given Automation
400 Composition Instance remain relatively unconstrained. A
401 `proposal to support access-controlled access to and between ONAP services
402 <https://wiki.onap.org/pages/viewpage.action?pageId=103417456>`_
403 will improve this. This can be complemented by intercepting and controlling services
404 accesses between Automation Composition Elements for Automation Composition Instances for some/all Automation
407 API gateways such as `Kong <https://konghq.com/kong/>`_ have emerged as a useful technology
408 for exposing and controlling service endpoint access for applications and services. When a
409 Automation Composition Type is onboarded, or when Automation Composition Instances are created in the Participants,
410 CLAMP can configure service endpoints between Automation Composition Elements to redirect through an
413 Authentication and access-control rules can then be dynamically configured at the API gateway
414 to support constrained access between Automation Composition Elements and Automation Composition Instances.
416 The diagram below shows the approach for configuring API Gateway access at Automation Composition
417 Instance and Automation Composition Element level.
419 .. image:: images/06-api-gateway-sandbox.png
421 At design time, the Automation Composition type definition specifies the type of API gateway configuration
422 that should be supported at Automation Composition and Automation Composition Element levels.
424 At runtime, the CLAMP can configure the API gateway to enable (or deny) interactions between
425 Automation Composition Instances and individually for each Automation Composition Element. All service-level
426 interactions in/out of a Automation Composition Element, except that to/from the API Gateway, can be
427 blocked by networking policies, thus sandboxing a Automation Composition Element and an entire Automation
428 Composition Instance if desired. Therefore, a Automation Composition Element will only have access to the APIs
429 that are configured and enabled for the Automation Composition Element/Instance in the API gateway.
431 For some Automation Composition Element Types the Participant can assist with service endpoint
432 reconfiguration, service request/response redirection to/from the API Gateway, or
433 annotation of requests/responses.
435 Once the Automation Composition instance is instantiated on participants, the participants configure
436 the API gateway with the Automation Composition Instance level configuration and with the specific
437 configuration for their Automation Composition Element.
439 Monitoring and logging of the use of the API gateway may also be provided. Information and
440 statistics on API gateway use can be read from the API gateway and passed back in monitoring
441 messages to the CLAMP runtime.
443 Additional isolation and execution-environment sandboxing can be supported depending on the
444 Automation Composition Element Type. For example: ONAP policies for given Automation Composition Instances/Types
445 can be executed in a dedicated PDP engine instances; DCAE or K8S-hosted services can executed
446 in isolated namespaces or in dedicated workers/clusters; etc..
452 The APIs and Protocols used by CLAMP for Automation Compositions are described on the pages below:
454 #. :ref:`System Level Dialogues <system-level-label>`
455 #. :ref:`The CLAMP Automation Composition Participant Protocol <acm-participant-protocol-label>`
456 #. :ref:`REST APIs for CLAMP Automation Compositions <acm-rest-apis-label>`
459 6 Design and Implementation
460 ===========================
462 The design and implementation of TOSCA Automation Compositions in CLAMP is described for each executable entity on the pages below:
464 #. :ref:`The CLAMP Automation Composition Runtime Server <clamp-runtime-acm>`
465 #. :ref:`CLAMP Automation Composition Participants <clamp-acm-participants>`
466 #. :ref:`Managing Automation Compositions using The CLAMP GUI <clamp-gui-acm>`