docs/sections/services/ves-http/stnd-defined-validation.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 StndDefined Events Collection Mechanism
   5 =======================================
   6
   7 Description
   8 -----------
   9
  10 The target of that development was to allow collection of events defined by standards organizations using VES Collector,
  11 and providing them for consumption by analytics applications running on top of DCAE platform. The following features
  12 have been implemented:
  13
  14 - Event routing, based on a new CommonHeader field "stndDefinedNamespace"
  15 - Standards-organization defined events can be included using a dedicated stndDefinedFields.data property
  16 - Standards-defined events can be validated using openAPI descriptions provided by standards organizations, and indicated in stndDefinedFields.schemaReference
  17
  18 StndDefined properties
  19 ----------------------
  20
  21 There are 5 additional properties related to stndDefined validation in collector.properties file.
  22
  23 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  24 | Name                                         | Description                                                                    | Example                                                                                              |
  25 +==============================================+================================================================================+======================================================================================================+
  26 | collector.externalSchema.checkflag           | Flag is responsible for turning on/off stndDefined data validation.            | -1 or 1                                                                                              |
  27 |                                              | By default this flag is set to 1, which means that the validation is enabled.  |                                                                                                      |
  28 |                                              | In case flag is set to -1, validation is disabled.                             |                                                                                                      |
  29 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  30 | collector.externalSchema.mappingFileLocation | This should be a local filesystem path to file with mappings of public URLs    | /opt/app/VESCollector/etc/externalRepo/schema-map.json                                               |
  31 |                                              | to local URLs.                                                                 |                                                                                                      |
  32 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  33 | collector.externalSchema.schemasLocation     | Schemas location is a directory context for localURL paths set in mapping file.| /opt/app/VESCollector/etc/externalRepo/ ,                                                            |
  34 |                                              | Result path of schema is collector.externalSchema.schemasLocation + localURL.  | when first mapping from example mapping file below this table is taken, validator will look for      |
  35 |                                              | This path is not related to mapping file path and may point to any location.   | schema under the path:                                                                               |
  36 |                                              |                                                                                | /opt/app/VESCollector/etc/externalRepo/3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml    |
  37 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  38 | event.externalSchema.schemaRefPath           | This is an internal path from validated JSON. It should define which field     | $.event.stndDefinedFields.schemaReference                                                            |
  39 |                                              | will be taken as public schema reference, which is later mapped.               |                                                                                                      |
  40 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  41 | event.externalSchema.stndDefinedDataPath     | This is internal path from validated JSON.                                     | $.event.stndDefinedFields.data                                                                       |
  42 |                                              | It should define which field will be validated.                                |                                                                                                      |
  43 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  44
  45 Format of the schema mapping file is a JSON file with list of mappings, as shown in the example below.
  46
  47 .. code-block:: json
  48
  49     [
  50       {
  51         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/faultMnS.yaml",
  52         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/faultMnS.yaml"
  53       },
  54       {
  55         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/heartbeatNtf.yaml",
  56         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/heartbeatNtf.yaml"
  57       },
  58       {
  59         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml",
  60         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml"
  61       },
  62       {
  63         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/provMnS.yaml",
  64         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/provMnS.yaml"
  65       }
  66     ]
  67
  68 External schema config maps
  69 ---------------------------
  70
  71 The mapping and schema files content can be changed by editing a proper config map.
  72
  73
  74 +----------------------------------------------+-----------------------------------------------------------------------------------------------------+
  75 | Config map name                              | Description                                                                                         |
  76 +==============================================+=====================================================================================================+
  77 | dcae-external-repo-configmap-schema-map      | Defines a content of the /opt/app/VESCollector/etc/externalRepo/schema-map.json file.               |
  78 +----------------------------------------------+-----------------------------------------------------------------------------------------------------+
  79 | dcae-external-repo-configmap-sa88-rel16      | Defines a content of schemas stored in the /opt/app/VESCollector/etc/externalRepo folder.           |
  80 +----------------------------------------------+-----------------------------------------------------------------------------------------------------+
  81
  82 Config maps are defined in the `OOM <https://gerrit.onap.org/r/gitweb?p=oom.git;a=tree;f=kubernetes/dcaegen2-services/resources/external>`_ repository
  83 and are installed with dcaegen2-services module.
  84
  85
  86 Properties configuration via Helm chart overrides
  87 -------------------------------------------------
  88
  89 Collector.properties content may be overridden when deploying VES Collector via Helm chart. In case of deploying VES using Helm chart,
  90 a config map "dcae-ves-collector-application-config-configmap" with the application_config.yaml file is created. The application_config.yaml
  91 contains properties, that override values from Collector.properties. In order to change any value, it is sufficient to edit the application_config.yaml
  92 in the config map. The VES application frequently reads the configMap content and applies configuration changes.
  93
  94 The content of "dcae-ves-collector-application-config-configmap" is defined in the values.yaml of the dcae-ves-collector chart
  95 and is installed with dcaegen2-services module.
  96
  97 The following table shows stndDefined related properties added to VES Collector Helm chart. These properties
  98 represent fields from collector.properties file, but also contain configuration of DMaaP topic URLs used for stndDefined
  99 events routing.
 100
 101 **NOTE**: Keep in mind that some properties may use relative path. It is relative to default VES Collector context which
 102 is: */opt/app/VESCollector/*. Final path of etc. *collector.externalSchema.schemasLocation* will be:
 103 */opt/app/VESCollector/etc/externalRepo/*. Setting absolute path to these properties is also acceptable and won't
 104 generate error.
 105
 106 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 107 | Property name                                                               | Type    | Default value                                                                                                 |
 108 +=============================================================================+=========+===============================================================================================================+
 109 | collector.externalSchema.checkflag                                          | Integer | 1                                                                                                             |
 110 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 111 | collector.externalSchema.mappingFileLocation                                | String  | ./etc/externalRepo/schema-map.json                                                                            |
 112 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 113 | collector.externalSchema.schemasLocation                                    | String  | ./etc/externalRepo/                                                                                           |
 114 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 115 | event.externalSchema.schemaRefPath                                          | String  | $.event.stndDefinedFields.schemaReference                                                                     |
 116 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 117 | event.externalSchema.stndDefinedDataPath                                    | String  | $.event.stndDefinedFields.data                                                                                |
 118 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 119 | streams_publishes.ves-3gpp-fault-supervision.dmaap_info.topic_url           | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
 120 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 121 | streams_publishes.ves-3gpp-provisioning.dmaap_info.topic_url                | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
 122 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 123 | streams_publishes.ves-3gpp-heartbeat.dmaap_info.topic_url                   | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
 124 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 125 | streams_publishes.ves-3gpp-performance-assurance.dmaap_info.topic_url       | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
 126 +-----------------------------------------------------------------------------+---------+---------------------------------------------------------------------------------------------------------------+
 127
 128 Validation overview
 129 -------------------
 130
 131 This mechanism can be used to validate any JSON content incoming as JsonNode using OpenAPI standardized schemas.
 132 During validation externally located schemas are mapped to local schema files.
 133
 134 Validated JSON must have one field that will refer to an external schema, which will be mapped to local file and then
 135 validation of any chosen part of JSON is executed using local schema.
 136
 137 StndDefined validation is integrated with the event collecting functionality available under the endpoint
 138 */eventListener/v7*. Process of event collecting includes steps in the following order:
 139
 140 1. General event validation (1st stage validation)
 141 2. Event transformation
 142 3. **StndDefined event validation** (2nd stage validation)
 143 4. Event routing to DMaaP
 144
 145 Mapping file is cached on stndDefined validator creation, so it's not read every time validation is performed.
 146 Schemas' content couldn't be cached due to an external library restrictions (OpenAPI4j).
 147
 148 The value of the 'stndDefinedNamespace' field in any incoming stndDefined JSON event is used to match the topic from
 149 property *collector.dmaap.streamid*.
 150
 151 Requirements for stndDefined validation
 152 ---------------------------------------
 153
 154 To run stndDefined validation, both *collector.schema.checkflag* and *collector.externalSchema.checkflag* must be set to 1.
 155
 156 Despite the flag set, the validation will not start when:
 157
 158 - Domain of the incoming event is not 'stndDefined'.
 159 - General event validation (1st stage) failed.
 160 - Field of event referenced under the property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*):
 161     - Has an empty value.
 162     - Does not exist in the incoming event.
 163
 164 Validation scenarios
 165 --------------------
 166
 167 Positive scenario, which returns 202 Accepted HTTP code after successful stndDefined validation:
 168
 169 - *collector.schema.checkflag* and *collector.externalSchema.checkflag* is set to 1
 170 - Mapping file has valid format
 171 - Schema file mapped from referenced in the event is valid
 172 - The incoming event is valid against schema
 173
 174 Below are scenarios when, the stndDefined validation will end with failure and return 400 Bad Request HTTP code:
 175
 176 - One of stndDefined data fields has wrong type or value
 177 - StndDefined data has an empty body or is missing required field
 178 - Field of event referenced under the property event.externalSchema.schemaRefPath has publicURL which is not mapped in the schemas mappings
 179 - Field defining public schema in event (by default */event/stndDefinedFields/schemaReference*) after "#" has non existing reference in schema file
 180
 181 Schema repository description
 182 -----------------------------
 183
 184 Schemas and mapping file location might be configured to any local directory through properties in collector.properties
 185 as described in 'StndDefined properties' section.
 186
 187 By default schemas repository is located under */opt/app/VESCollector/etc/externalSchema* directory, as well as schemas mapping file called
 188 *schema-map.json*. Every organisation which adds or mounts external schemas should store them in folder named by
 189 organisation name. Further folders structure may be whatever as long as schemas are correctly referenced in the mapping
 190 file.
 191
 192 Sample directory tree of */opt/app/VESCollector/etc* directory:
 193
 194 .. code-block:: text
 195
 196     /opt/app/VESCollector/etc
 197     ├── ...
 198     └── externalRepo
 199         ├── schema-map.json
 200         └── 3gpp
 201             └── rep
 202                 └── sa5
 203                     └── MnS
 204                         └── blob
 205                             └── SA88-Rel16
 206                                 └── OpenAPI
 207                                     ├── faultMnS.yaml
 208                                     ├── heartbeatNtf.yaml
 209                                     ├── PerDataFileReportMnS.yaml
 210                                     └── provMnS.yaml
 211
 212 Routing of stndDefined domain events
 213 ------------------------------------
 214
 215 All events, except those with 'stndDefined' domain, are routed to DMaaP topics based on domain value. Events with
 216 'stndDefined' domain are sent to proper topic basing on field 'stndDefinedNamespace'.
 217
 218 This is the only difference from standard event routing, specific for 'stndDefined' domain. As in every other event
 219 routing value is being mapped for specific DMaaP stream. Stream ID to DMaaP channels mappings are located in
 220 */opt/app/VESCollector/etc/collector.properties* file under property *collector.dmaap.streamid*. Channels descriptions are in
 221 */opt/app/VESCollector/etc/DmaapConfig.json*, where destination DMaaP topics are selected.
 222
 223 With stndDefined domain managment 4 new mappings were added. Their routing has been described in the table below:
 224
 225     +---------------------------+--------------------------------+------------------------------------------------------+
 226     | Stream ID                 | Channel                        | DMaaP Stream                                         |
 227     +===========================+================================+======================================================+
 228     | 3GPP-FaultSupervision     | ves-3gpp-fault-supervision     | unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
 229     +---------------------------+--------------------------------+------------------------------------------------------+
 230     | 3GPP-Heartbeat            | ves-3gpp-heartbeat             | unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
 231     +---------------------------+--------------------------------+------------------------------------------------------+
 232     | 3GPP-Provisioning         | ves-3gpp-provisioning          | unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
 233     +---------------------------+--------------------------------+------------------------------------------------------+
 234     | 3GPP-PerformanceAssurance | ves-3gpp-performance-assurance | unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
 235     +---------------------------+--------------------------------+------------------------------------------------------+
 236
 237
 238 Error scenarios behaviour
 239 -------------------------
 240
 241 There are few error scenarios described in 'Validation scenarios' section. This section will describe user point of view
 242 of VES Collector behaviour when they happen. Messages returned as HTTP response contain data described below for each
 243 scenario.
 244
 245 1. StndDefined fields validation related errors
 246
 247 1.1. Schema file referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) not present in the schema repository.
 248
 249     +---------------------+------------------------------------------------------------------+
 250     | Property Name       | Property Description                                             |
 251     +=====================+==================================================================+
 252     | MessageId           | SVC2004                                                          |
 253     +---------------------+------------------------------------------------------------------+
 254     | Text                | "Invalid input value for %1 %2: %3"                              |
 255     +---------------------+------------------------------------------------------------------+
 256     | Variables           | %1 - "attribute"                                                 |
 257     |                     | %2 - "event.stndDefinedFields.schemaReference"                   |
 258     |                     | %3 - "Referred external schema not present in schema repository" |
 259     +---------------------+------------------------------------------------------------------+
 260     | HTTP status code(s) | 400 Bad request                                                  |
 261     +---------------------+------------------------------------------------------------------+
 262
 263 1.2. File referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) exists, but internal reference (part of URL after #) is incorrect.
 264
 265     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 266     | Property Name       | Property Description                                                                                                              |
 267     +=====================+===================================================================================================================================+
 268     | MessageId           | SVC2000                                                                                                                           |
 269     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 270     | Text                | The following service error occurred: %1. Error code is %2                                                                        |
 271     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 272     | Variables           | %1 - "event.stndDefinedFields.schemaReference value does not correspond to any external event schema file in externalSchema repo" |
 273     |                     | %2 - "400"                                                                                                                        |
 274     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 275     | HTTP status code(s) | 400 Bad request                                                                                                                   |
 276     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 277
 278 1.3. StndDefined validation executed, but event contents do not validate with referenced schema.
 279
 280     +---------------------+---------------------------------------------------------------------------------------------+
 281     | Property Name       | Property Description                                                                        |
 282     +=====================+=============================================================================================+
 283     | MessageId           | SVC2000                                                                                     |
 284     +---------------------+---------------------------------------------------------------------------------------------+
 285     | Text                | The following service error occurred: %1. Error code is %2                                  |
 286     +---------------------+---------------------------------------------------------------------------------------------+
 287     | Variables           | %1 - "event.stndDefinedFields.data invalid against event.stndDefinedFields.schemaReference" |
 288     |                     | %2 - "400"                                                                                  |
 289     +---------------------+---------------------------------------------------------------------------------------------+
 290     | HTTP status code(s) | 400 Bad request                                                                             |
 291     +---------------------+---------------------------------------------------------------------------------------------+
 292
 293 2. Problems with routing of stndDefined domain.
 294
 295 2.1. StndDefinedNamespace field not received in the incoming event.
 296
 297     +---------------------+-----------------------------------------------------+
 298     | Property Name       | Property Description                                |
 299     +=====================+=====================================================+
 300     | MessageId           | SVC2006                                             |
 301     +---------------------+-----------------------------------------------------+
 302     | Text                | Mandatory input %1 %2 is missing from request       |
 303     +---------------------+-----------------------------------------------------+
 304     | Variables           | %1 - "attribute"                                    |
 305     |                     | %2 - "event.commonEventHeader.stndDefinedNamespace" |
 306     +---------------------+-----------------------------------------------------+
 307     | HTTP status code(s) | 400 Bad Request                                     |
 308     +---------------------+-----------------------------------------------------+
 309
 310 2.2. StndDefinedNamespace field present, but value is empty.
 311
 312     +---------------------+-----------------------------------------------------+
 313     | Property Name       | Property Description                                |
 314     +=====================+=====================================================+
 315     | MessageId           | SVC2006                                             |
 316     +---------------------+-----------------------------------------------------+
 317     | Text                | Mandatory input %1 %2 is empty in request           |
 318     +---------------------+-----------------------------------------------------+
 319     | Variables           | %1 - "attribute"                                    |
 320     |                     | %2 - "event.commonEventHeader.stndDefinedNamespace" |
 321     +---------------------+-----------------------------------------------------+
 322     | HTTP status code(s) | 400 Bad Request                                     |
 323     +---------------------+-----------------------------------------------------+
 324
 325 2.3. StndDefinedNamespace field present, but value doesn't match any stream ID mapping.
 326
 327     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 328     | Property Name       | Property Description                                                                                                                      |
 329     +=====================+===========================================================================================================================================+
 330     | MessageId           | SVC2004                                                                                                                                   |
 331     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 332     | Text                | "Invalid input value for %1 %2: %3"                                                                                                       |
 333     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 334     | Variables           | %1 - "attribute"                                                                                                                          |
 335     |                     | %2 - "event.commonEventHeader.stndDefinedNamespace"                                                                                       |
 336     |                     | %3 - "stndDefinedNamespace received not present in VES Collector routing configuration. Unable to route event to appropriate DMaaP topic" |
 337     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 338     | HTTP status code(s) | 400 Bad request                                                                                                                           |
 339     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+