1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
3 .. _clamp-acm_architecture-label:
5 Automation Composition Management: Architecture and Design
6 ##########################################################
11 To get a feature, service, or capability working in modern networks is not straightforward.
12 It is not as simple as deploying a microservice or running a workflow. Our features, services, and capabilities are
13 now typically delivered using loose compositions of microservices, rules, algorithms, configurations, and workflows.
14 Of course, we use workflows and deploy microservices, but how do we keep track of what workflow activated which service
15 or what microservice instance enables a given capability. We must be able to deploy, keep track of, amend, and remove
16 the compositions that combine to give us our features, services and capabilities, that is we must manage those compositions.
18 .. image:: images/feature-compositions.png
20 Consider Features A, B and C in the diagram above.
22 Feature A is realised as an Analytic Microservice, but it also requires counters to be configured in a collection
23 service to enable its input stream of data. It also requires two policies to be present, and its result
24 requires an Ansible playbook to be present.
26 Feature B is realised as an AI microservice, which is triggered by a set of triggers that are configured in the
27 persistence service. The AI algorithm in Feature B triggers a workflow in the workflow service
29 Feature C is realised as two microservices, an analytic microservice and a Machine Learning microservice.
30 The feature also requires that certain counters are collected and certain Netconf configurations are enabled.
32 All three features are realised as Automation Compositions, as shown in the diagram below.
34 .. image:: images/feature-automation-compositions.png
36 The ability to deploy features in a scalable, flexible and loosely coupled microservice architecture is of course
37 a major step forward from layered architectures of the past. However, managing at "Feature" level in such architectures
38 does present challenges. For example, to manage the three running instances of Features A to C above,
39 9 separate elements must be kept track of. There is nothing in the deployed system to sat what element is
40 related to what other element, and what element are working together to realise a feature.
42 Automation Composition Management (ACM) is a framework that supports Life Cycle Management of Automation Compositions.
43 It supports deployment, monitoring, update and removal of Automation Compositions en-bloc, allowing users
44 to manage their features, services, and capabilities as single logical units.
49 The idea of using automation compositions to automatically (or autonomously) perform network management
50 has been the subject of much research in the Network Management research community, see
51 :download:`this paper <files/Acms.pdf>` for some background. However, it is only with
52 the advent of ONAP that we have a platform that supports automation compositions for network management.
53 Before ONAP, Automation Compositions have been implemented by hard-coding components together and hard
54 coding logic into components. ONAP has taken a step forward towards automatic implementation
55 of Automation Compositions by allowing parameterization of Automation Compositions that work on the premise that
56 the Automation Compositions use a set of analytic, policy, and control components connected together in
59 The goal of the work is to extend and enhance the current ONAP Automation Composition support to provide
60 a complete open-source framework for Automation Compositions. This will enhance the current support to
61 provide TOSCA based Automation Composition definition and development, commissioning and run-time management.
62 The participants that comprise a Automation Composition and the metadata needed to link the participants
63 together to create a Automation Composition are specified in a standardized way using the `OASIS TOSCA
64 modelling language <http://docs.oasis-open.org/tosca/TOSCA-Simple-Profile-YAML/>`_. The TOSCA
65 description is then used to commission, instantiate, and manage the Automation Compositions in the run
68 .. image:: images/01-acm-overview.png
73 This section describes the terminology used in the system.
75 2.1 Automation Composition Terminology
76 --------------------------------------
78 **Automation Composition Type:** A definition of a Automation Composition in the TOSCA language. This definition describes
79 a certain type of a automation composition. The life cycle of instances of a Automation Composition Type are managed
82 **Automation Composition Instance:** An instance of a Automation Composition Type. The life cycle of a Automation Composition
83 Instance is managed by ACM. An Automation Composition Instance is a set of executing elements on which
84 Life Cycle Management (LCM) is executed collectively. For example, a set of microservices may be
85 spawned and executed together to deliver a service. This collection of services is a automation composition.
87 **Automation Composition Element Type:** A definition of a Automation Composition Element in the TOSCA language. This
88 definition describes a certain type of Automation Composition Element for a automation composition in a Automation
91 **Automation Composition Element Instance:** A single entity executing on a participant, with its Life Cycle
92 being managed as part of the overall automation composition. For example, a single microservice that is
93 executing as one microservice in a service.
95 **Automation Composition Runtime:** The ACM Runtime server that holds Automation Composition Type definitions and manages
96 the life cycle of Automation Composition Instances and their Automation Composition Elements in cooperation with
100 2.2 Participant Terminology
101 ---------------------------
103 **Participant Type:** Definition of a type of system or framework that can take part in control
104 loops and a definition of the capabilities of that participant type. A participant advertises
105 its type to the CLAMP Automation Composition Runtime.
107 **Participant:** A system or framework that takes part in automation compositions by executing Automation Composition
108 Elements in cooperation with the Automation Composition Runtime. A participant chooses to partake
109 in automation compositions, to manage Automation Composition Elements for ACM, and to receive, send and act on
110 LCM messages for the ACM runtime.
112 2.3 Terminology for Properties
113 ------------------------------
115 **Common Properties:** Properties that apply to all Automation Composition Instances of a certain Automation
116 Composition Type and are specified when a Automation Composition Type is commissioned.
118 **Instance Specific Properties:** Properties that must be specified for each Automation Composition Instance
119 and are specified when a Automation Composition Instance is Initialized.
121 2.4 Concepts and their relationships
122 ------------------------------------
124 The UML diagram below shows the concepts described in the terminology sections above and how
125 they are interrelated.
127 .. image:: images/02-acm-concepts.png
129 The Automation Composition Definition concepts describe the types of things that are in the system. These
130 concepts are defined at design time and are passed to the runtime in a TOSCA document. The
131 concepts in the Automation Composition Runtime are created by the runtime part of the system using the
132 definitions created at design time.
134 .. _acm-capabilities:
139 We consider the capabilities of Automation Compositions at Design Time and Run Time.
141 At Design Time, three capabilities are supported:
143 #. **Automation Composition Element Definition Specification.** This capability allows users to define Automation
144 Composition Element Types and the metadata that can be used on and configured on a Automation Composition Element
145 Type. Users also define the Participant Type that will run the Automation Composition Element when it is
146 taking part in in a automation composition. The post condition of an execution of this capability is that
147 metadata for a Automation Composition Element Type is defined in the Automation Composition Design Time Catalogue.
149 #. **Automation Composition Element Definition Onboarding.** This capability allows external users and systems
150 (such as SDC or DCAE-MOD) to define the metadata that can be used on and configured on a Automation
151 Composition Element Type and to define the Participant Type that will run the Automation Composition Element when
152 it is taking part in in a automation composition. The post condition of an execution of this capability
153 is that metadata for a Automation Composition Element Type is defined in the Automation Composition Design Time
156 #. **Automation Composition Type Definition.** This capability allows users and other systems to create Automation
157 Composition Type definitions by specifying a set of Automation Composition Element Definitions from those that
158 are available in the Automation Composition Design Time Catalogue. These Automation Composition Elements will
159 work together to form Automation Compositions. In an execution of this capability, a user specifies the
160 metadata for the Automation Composition and specifies the set of Automation Composition Elements and their Participant
161 Types. The user also selects the correct metadata sets for each participant in the Automation Composition
162 Type and defines the overall Automation Composition Type metadata. The user also specifies the Common
163 Property Types that apply to all instances of a automation composition type and the Instance Specific
164 Property Types that apply to individual instances of a Automation Composition Type. The post condition for
165 an execution of this capability is a Automation Composition definition in TOSCA stored in the Automation Composition
166 Design Time Catalogue.
169 Once a Automation Composition Definition is commissioned to the Automation Composition Runtime and has been
170 stored in the Run Time Inventory, it cannot be further edited unless it is decommissioned.
173 At Run Time, the following participant related capabilities are supported:
175 #. **System Pre-Configuration.** This capability allows participants to register and deregister with ACM-R.
176 Participants explicitly register with ACM-R when they start. The post condition for an execution of this capability
177 is that a participant becomes available (registration) or is no longer available (deregistration) for
178 participation in an Automation Composition.
180 At Run Time, the following Automation Composition Life Cycle management capabilities are supported:
182 #. **Automation Composition Commissioning:** This capability allows version controlled Automation Composition Type
183 definitions to be taken from the Automation Composition Design Time Catalogue and be placed in the
184 Commissioned Automation Composition Inventory. It also allows the values of Common Property Types
185 that apply to all instances of a Automation Composition Type to be set. The post condition for an execution of this
186 capability is that the Automation Composition Type definition is in the Commissioned Automation Composition
189 #. **Automation Composition Priming on Participants.** The Priming operation sends Automation Composition Types
190 and common property values to participants for each Automation Composition Element Type in the Automation
191 Composition Type. The participant can then take whatever actions it need to do to support
192 the automation composition type in question. Automation Composition Priming is triggered by user interaction
193 with the ACM-R Rest API. The post condition for an execution of this capability is that the AC definitions are
194 primed on all participants, that is they are prepared to run instances of their Automation Composition Element types.
195 Automation composition definitions cannot be primed until they are commissioned.
197 #. **Automation Composition Instance Life Cycle Management:** This capability allows a Automation Composition
198 Instance to have its life cycle managed.
200 #. **Automation Composition Instance Creation:** This capability allows an Automation Composition Instance to be
201 created. The Automation Composition Type definition is read from the Commissioned Automation Composition Inventory
202 and values are assigned to the Instance Specific Property Types defined for instances of the Automation Composition
203 Type by the ACM client. An Automation Composition Instance that has been created but has not yet been deployed
204 on participants is in deploy state UNDEPLOYED and lock state LOCKED. In this state, the Instance Specific Property
205 Type values can be revised and updated as often as the user requires. The post condition for an execution of this
206 capability is that the Automation Composition instance is created in the Instance Automation Composition Inventory but
207 has not been deployed on Participants.
209 #. **Automation Composition Instance Update on Participants:** Once the user is happy with the property
210 values, the Automation Composition Instance is updated on participants and the Automation Composition Elements
211 for this Automation Composition Instance are deployed or updated by participants using the acm metadata.
212 The post condition for an execution of this capability is that the Automation Composition instance is updated
215 #. **Automation Composition State Change:** The user can now order the participants to change the state
216 of the Automation Composition Instance. If the Automation Composition is set to deploy state DEPLOYED and
217 lock state UNLOCKED, each participant begins accepting and processing automation composition events and the
218 Automation Composition Instance is set to state DEPLOYED/UNLOCKED in the Instantiated Automation Composition
219 inventory. The post condition for an execution of this capability is that the Automation Composition instance
220 state is changed on participants.
222 #. **Automation Composition Instance Monitoring:** This capability allows Automation Composition Instances to be
223 monitored. Users can check the status of Participants, Automation Composition Instances, and Automation
224 Composition Elements. Participants report their overall status and the status of Automation Composition
225 Elements they are running periodically to ACM-R. ACM aggregates these status reports
226 into an aggregated Automation Composition Instance status record, which is available for monitoring.
227 The post condition for an execution of this capability is that Automation Composition Instances are
230 #. **Automation Composition Instance Supervision:** This capability allows Automation Composition Instances to be
231 supervised. The ACM runtime expects participants to report on Automation Composition Elements
232 periodically. The ACM runtime checks that periodic reports are received and that each
233 Automation Composition Element is in the state it should be in. If reports are missed or if a
234 Automation Composition Element is in an incorrect state, remedial action is taken and notifications
235 are issued. The post condition for an execution of this capability is that Automation Composition
236 Instances are being supervised by the ACM runtime.
238 #. **Automation Composition Instance Removal from Participants:** A user can order the removal of a Automation
239 Composition Instance from participants. The post condition for an execution of this capability is
240 that the Automation Composition instance is removed from Participants.
242 #. **Automation Composition Instance Deletion:** A user can order the removal of a Automation Composition Instance
243 from the ACM runtime and participants. Automation Composition Instances that are DEPLOYED/UNLOCKED on participants cannot
244 be removed from the ACM runtime. Each participant begins accepting and processing automation composition events removing the
245 Automation Composition Instance from the inventory. The post condition for an execution of this capability
246 is that the Automation Composition instance is removed from Instance Automation Composition Inventory
247 in ACM runtime and participants.
249 #. **Automation Composition Depriming on Participants.** The Depriming operation removes Automation Composition Types
250 and common property values from participants for each Automation Composition Element Type in the Automation
251 Composition Type. Automation Composition Depriming is triggered by user interaction
252 with the ACM-R Rest API. The post condition for an execution of this capability is that the AC definitions are
253 deprimed on all participants, that is they are no longer prepared to run instances of their Automation Composition
254 Element types. Automation composition definitions cannot be deprimed on a participant until the instances for that
255 participant are deleted.
257 #. **Automation Composition Decommissioning:** This capability allows version controlled Automation Composition Type
258 definitions to be removed from the Commissioned Automation Composition Inventory. An Automation Composition
259 Definition that has instances in the Instantiated Automation Composition Inventory or has not been deprimed on
260 participants cannot be removed. The post condition for an execution of this capability is that the Automation
261 Composition Type definition removed from the Commissioned Automation Composition Inventory.
264 The system dialogues for run time capabilities are described in detail on the
265 :ref:`System Level Dialogues <system-level-label>` page.
267 .. _acm-instance-states:
269 3.1 Automation Composition Instance States
270 ------------------------------------------
272 When an automation composition definition has been commissioned and primed, instances of the automation composition can be
273 created, updated, and deleted. The system manages the lifecycle of automation compositions and ac elements
274 following the state transition diagram below.
276 .. image:: images/03-acm-instance-states.png
278 4 Overall Target Architecture
279 =============================
281 The diagram below shows an overview of the architecture of TOSCA based Automation Composition
284 .. image:: images/04-overview.png
286 Following the ONAP Reference Architecture, the architecture has a Design Time part and
289 The Design Time part of the architecture allows a user to specify metadata for participants.
290 It also allows users to compose automation compositions. The Design Time Catalogue contains the metadata
291 primitives and automation composition definition primitives for composition of automation compositions. As shown
292 in the figure above, the Design Time component provides a system where Automation Compositions can be
293 designed and defined in metadata. This means that a Automation Composition can have any arbitrary
294 structure and the Automation Composition developers can use whatever analytic, policy, or control
295 participants they like to implement their Automation Composition. At composition time, the user
296 parameterizes the Automation Composition and stores it in the design time catalogue. This catalogue
297 contains the primitive metadata for any participants that can be used to compose a Automation
298 Composition. A Automation Composition SDK is used to compose a Automation Composition by aggregating the metadata for
299 the participants chosen to be used in a Automation Composition and by constructing the references between
300 the participants. The architecture of the Automation Composition Design Time part will be elaborated in
303 Composed Automation Compositions are commissioned on the run time part of the system, where they are
304 stored in the Commissioned Automation Composition inventory and are available for instantiation. The
305 Commissioning component provides a CRUD REST interface for Automation Composition Types, and implements
306 CRUD of Automation Composition Types. Commissioning also implements validation and persistence of incoming
307 Automation Composition Types. It also guarantees the integrity of updates and deletions of Automation Composition
308 Types, such as performing updates in accordance with semantic versioning rules and ensuring that
309 deletions are not allowed on Automation Composition Types that have instances defined.
311 The Instantiation component manages the Life Cycle Management of Automation Composition Instances and
312 their Automation Composition Elements. It publishes a REST interface that is used to create Automation Composition
313 Instances and set values for Common and Instance Specific properties. This REST interface is
314 public and is used by the ACM GUI. It may also be used by any other client via the public
315 REST interface. The REST interface also allows the state of Automation Composition Instances to be changed.
316 A user can change the state of Automation Composition Instances as described in the state transition
317 diagram shown in section 2 above. The Instantiation component issues update and state change
318 messages via DMaaP/Kafka to participants so that they can update and manage the state of the Automation
319 Composition Elements they are responsible for. The Instantiation component also implements persistence
320 of Automation Composition Instances, automation composition elements, and their state changes.
322 The Monitoring component reads updates sent by participants. Participants report on the
323 state of their Automation Composition Elements periodically and in response to a message they have
324 received from the Instantiation component. The Monitoring component reads the contents of
325 the participant messages and persists their state updates and statistics records. It also
326 publishes a REST interface that publishes the current state of all Participants, Automation
327 Composition Instances and their Automation Composition Elements, as well as publishing Participant and
328 Automation Composition statistics.
330 The Supervision component is responsible for checking that Automation Composition Instances are correctly
331 instantiated and are in the correct state (UNDEPLOYED/DEPLOYED/LOCKED/UNLOCKED). It also handles
332 timeouts and on state changes to Automation Composition Instances, and retries and rolls back state
333 changes where state changes failed.
335 A Participant is an executing component that partakes in automation compositions. More explicitly, a
336 Participant is something that implements the Participant Instantiation and Participant
337 Monitoring messaging protocol over DMaaP/Kafka for Life Cycle management of Automation Composition Elements.
338 A Participant runs Automation Composition Elements and manages and reports on their life cycle
339 following the instructions it gets from the ACM runtime in messages delivered over DMaaP/Kafka.
341 In the figure above, five participants are shown. A Configuration Persistence Participant
342 manages Automation Composition Elements that interact with the `ONAP Configuration Persistence Service
343 <https://docs.onap.org/projects/onap-cps/en/latest/overview.html>`_
344 to store common data. The DCAE Participant runs Automation Composition Elements that manage DCAE
345 microservices. The Kubernetes Participant hosts the Automation Composition Elements that are managing
346 the life cycle of microservices in automation compositions that are in a Kubernetes ecosystem. The
347 Policy Participant handles the Automation Composition Elements that interact with the Policy Framework
348 to manage policies for automation compositions. A Automation Participant such as the CDS Participant
349 runs Automation Composition Elements that load metadata and configure controllers so that they can
350 partake in automation compositions. Any third party Existing System Participant can be developed to
351 run Automation Composition Elements that interact with any existing system (such as an operator's
352 analytic, machine learning, or artificial intelligence system) so that those systems can
353 partake in automation compositions.
355 5. Other Considerations
356 =======================
358 .. _management-acm-instance-configs:
360 5.1 Management of Automation Composition Instance Configurations
361 ----------------------------------------------------------------
363 In order to keep management of versions of the configuration of automation composition instances
364 straightforward and easy to implement, the following version management scheme using
365 semantic versioning is implemented. Each configuration of a Automation Composition Instance and
366 configuration of a Automation Composition Element has a semantic version with 3 digits indicating
367 the **major.minor.patch** number of the version.
370 A **configuration** means a full set of parameter values for a Automation Composition Instance.
372 .. image:: images/05-upgrade-states.png
376 #. An Automation Composition or Automation Composition Element in state **UNDEPLOYED** can be changed to a higher
377 major/minor/patch level or rolled back to a lower major/minor/patch level. This means
378 that where the structure of the entire automation composition is changed, the automation composition must
379 be undeployed and redeployed.
381 #. If a Automation Composition Element has a **minor** version change, then its Automation Composition Instance
382 must have at least a **minor** version change.
384 #. If a Automation Composition Element has a **major** version change, then its Automation Composition Instance
385 must have a **major** version change.
390 The system is designed to be inherently scalable. The ACM runtime is stateless, all state
391 is preserved in the Instantiated Automation Composition inventory in the database. When the user
392 requests an operation such as an undeploy, deploy, lock, or an unlock
393 on a Automation Composition Instance, the ACM runtime broadcasts the request to participants over
394 DMaaP/Kafka and saves details of the request to the database. The ACM runtime does not directly
395 wait for responses to requests.
397 When a request is broadcast on DMaaP/Kafka, the request is asynchronously picked up by participants
398 of the types required for the Automation Composition Instance and those participants manage the life
399 cycle of its automation composition elements. Periodically, each participant reports back on the status
400 of operations it has picked up for the Automation Composition Elements it controls, together with
401 statistics on the Automation Composition Elements over DMaaP/Kafka. On reception of these participant messages,
402 the ACM runtime stores this information to its database.
404 The participant to use on a automation composition can be selected from the registered participants
405 in either of two ways:
407 **Runtime-side Selection:** The ACM runtime selects a suitable participant from the list of
408 participants and sends the participant ID that should be used in the Participant Update message.
409 In this case, the CLAMP runtime decides on which participant will run the Automation Composition Element
410 based on a suitable algorithm. Algorithms could be round robin based or load based.
412 **Participant-side Selection:** The ACM runtime sends a list of Participant IDs that may be used
413 in the Participant Update message. In this case, the candidate participants decide among
414 themselves which participant should host the Automation Composition Element.
416 This approach makes it easy to scale Automation Composition life cycle management. As Automation Composition
417 Instance counts increase, more than one ACM runtime can be deployed and REST/supervision
418 operations on Automation Composition Instances can run in parallel. The number of participants can
419 scale because an asynchronous broadcast mechanism is used for runtime-participant communication
420 and there is no direct connection or communication channel between participants and ACM
421 runtime servers. Participant state, Automation Composition Instance state, and Automation Composition Element
422 state is held in the database, so any CLAMP runtime server can handle operations for any
423 participant. Because many participants of a particular type can be deployed and participant
424 instances can load balance automation composition element instances for different Automation Composition Instances
425 of many types across themselves using a mechanism such as a Kubernetes cluster.
428 5.3 Sandboxing and API Gateway Support
429 --------------------------------------
431 At runtime, interaction between ONAP platform services and application microservices are
432 relatively unconstrained, so interactions between Automation Composition Elements for a given Automation
433 Composition Instance remain relatively unconstrained. A
434 `proposal to support access-controlled access to and between ONAP services
435 <https://wiki.onap.org/pages/viewpage.action?pageId=103417456>`_
436 will improve this. This can be complemented by intercepting and controlling services
437 accesses between Automation Composition Elements for Automation Composition Instances for some/all Automation
440 API gateways such as `Kong <https://konghq.com/kong/>`_ have emerged as a useful technology
441 for exposing and controlling service endpoint access for applications and services. When a
442 Automation Composition Type is onboarded, or when Automation Composition Instances are created in the Participants,
443 CLAMP can configure service endpoints between Automation Composition Elements to redirect through an
446 Authentication and access-control rules can then be dynamically configured at the API gateway
447 to support constrained access between Automation Composition Elements and Automation Composition Instances.
449 The diagram below shows the approach for configuring API Gateway access at Automation Composition
450 Instance and Automation Composition Element level.
452 .. image:: images/06-api-gateway-sandbox.png
454 At design time, the Automation Composition type definition specifies the type of API gateway configuration
455 that should be supported at Automation Composition and Automation Composition Element levels.
457 At runtime, the ACM-R can configure the API gateway to enable (or deny) interactions between
458 Automation Composition Instances and individually for each Automation Composition Element. All service-level
459 interactions in/out of a Automation Composition Element, except that to/from the API Gateway, can be
460 blocked by networking policies, thus sandboxing a Automation Composition Element and an entire Automation
461 Composition Instance if desired. Therefore, a Automation Composition Element will only have access to the APIs
462 that are configured and enabled for the Automation Composition Element/Instance in the API gateway.
464 For some Automation Composition Element Types the Participant can assist with service endpoint
465 reconfiguration, service request/response redirection to/from the API Gateway, or
466 annotation of requests/responses.
468 Once the Automation Composition instance is instantiated on participants, the participants configure
469 the API gateway with the Automation Composition Instance level configuration and with the specific
470 configuration for their Automation Composition Element.
472 Monitoring and logging of the use of the API gateway may also be provided. Information and
473 statistics on API gateway use can be read from the API gateway and passed back in monitoring
474 messages to the ACM runtime.
476 Additional isolation and execution-environment sandboxing can be supported depending on the
477 Automation Composition Element Type. For example: ONAP policies for given Automation Composition Instances/Types
478 can be executed in a dedicated PDP engine instances; DCAE or K8S-hosted services can executed
479 in isolated namespaces or in dedicated workers/clusters; etc..
485 The APIs and Protocols used by CLAMP for Automation Compositions are described on the pages below:
487 #. :ref:`System Level Dialogues <system-level-label>`
488 #. :ref:`The CLAMP Automation Composition Participant Protocol <acm-participant-protocol-label>`
489 #. :ref:`REST APIs for CLAMP Automation Compositions <acm-rest-apis-label>`
492 7 Design and Implementation
493 ===========================
495 The design and implementation of TOSCA Automation Compositions in CLAMP is described for each executable entity on the pages below:
497 #. :ref:`The CLAMP Automation Composition Runtime Server <clamp-runtime-acm>`
498 #. :ref:`CLAMP Automation Composition Participants <clamp-acm-participants>`
499 #. :ref:`Managing Automation Compositions using The CLAMP GUI <clamp-gui-acm>`