docs/sections/services/ves-http/stnd-defined-validation.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 StndDefined Events Collection Mechanism
   5 =======================================
   6
   7 Description
   8 -----------
   9
  10 The target of that development was to allow collection of events defined by standards organizations using VES Collector,
  11 and providing them for consumption by analytics applications running on top of DCAE platform. The following features
  12 have been implemented:
  13
  14 - Event routing, based on a new CommonHeader field "stndDefinedNamespace"
  15 - Standards-organization defined events can be included using a dedicated stndDefinedFields.data property
  16 - Standards-defined events can be validated using openAPI descriptions provided by standards organizations, and indicated in stndDefinedFields.schemaReference
  17
  18 StndDefined properties
  19 ----------------------
  20
  21 There are 5 additional properties related to stndDefined validation in collector.properties file.
  22
  23 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  24 | Name                                         | Description                                                                    | Example                                                                                              |
  25 +==============================================+================================================================================+======================================================================================================+
  26 | collector.externalSchema.checkflag           | Flag is responsible for turning on/off stndDefined data validation.            | -1 or 1                                                                                              |
  27 |                                              | By default this flag is set to 1, which means that the validation is enabled.  |                                                                                                      |
  28 |                                              | In case flag is set to -1, validation is disabled.                             |                                                                                                      |
  29 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  30 | collector.externalSchema.mappingFileLocation | This should be a local filesystem path to file with mappings of public URLs    | /opt/app/VESCollector/etc/externalRepo/schema-map.json                                               |
  31 |                                              | to local URLs.                                                                 |                                                                                                      |
  32 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  33 | collector.externalSchema.schemasLocation     | Schemas location is a directory context for localURL paths set in mapping file.| /opt/app/VESCollector/etc/externalRepo/ ,                                                            |
  34 |                                              | Result path of schema is collector.externalSchema.schemasLocation + localURL.  | when first mapping from example mapping file below this table is taken, validator will look for      |
  35 |                                              | This path is not related to mapping file path and may point to any location.   | schema under the path:                                                                               |
  36 |                                              |                                                                                | /opt/app/VESCollector/etc/externalRepo/3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml    |
  37 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  38 | event.externalSchema.schemaRefPath           | This is an internal path from validated JSON. It should define which field     | $.event.stndDefinedFields.schemaReference                                                            |
  39 |                                              | will be taken as public schema reference, which is later mapped.               |                                                                                                      |
  40 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  41 | event.externalSchema.stndDefinedDataPath     | This is internal path from validated JSON.                                     | $.event.stndDefinedFields.data                                                                       |
  42 |                                              | It should define which field will be validated.                                |                                                                                                      |
  43 +----------------------------------------------+--------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
  44
  45 Format of the schema mapping file is a JSON file with list of mappings, as shown in the example below.
  46
  47 .. code-block:: json
  48
  49     [
  50       {
  51         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/faultMnS.yaml",
  52         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/faultMnS.yaml"
  53       },
  54       {
  55         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/heartbeatNtf.yaml",
  56         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/heartbeatNtf.yaml"
  57       },
  58       {
  59         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml",
  60         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml"
  61       },
  62       {
  63         "publicURL": "https://forge.3gpp.org/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/provMnS.yaml",
  64         "localURL": "3gpp/rep/sa5/MnS/blob/SA88-Rel16/OpenAPI/provMnS.yaml"
  65       }
  66     ]
  67
  68 Properties configuration via Cloudify
  69 -------------------------------------
  70
  71 Collector.properties content may be overridden when deploying VES Collector via Cloudify. To keep VES settings
  72 consistent listed above properties has been updated in the VES Collector Cloudify blueprint (in blueprints/k8s-ves.yaml
  73 file under dcaegen2/platform/blueprints project) and in componentspec file (in dpo/spec/vescollector-componentspec.json
  74 file in VES project) which may be used for generation of VES Collector Cloudify blueprints in some scenarios.
  75
  76 The following table shows new stndDefined related properties added to VES Collector Cloudify blueprint. These properties
  77 represent fields from collector.properties file, but also contain configuration of DMaaP topic URLs used for stndDefined
  78 events routing. It has been specified in the table which of these properties may be configured via inputs during
  79 blueprint deployment.
  80
  81 **NOTE**: Keep in mind that some properties may use relative path. It is relative to default VES Collector context which
  82 is: */opt/app/VESCollector/*. Final path of etc. *collector.externalSchema.schemasLocation* will be:
  83 */opt/app/VESCollector/etc/externalRepo/*. Setting absolute path to these properties is also acceptable and won't
  84 generate error.
  85
  86 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
  87 | Property name                                | Input? | Type    | Default value                                                                                                 |
  88 +==============================================+========+=========+===============================================================================================================+
  89 | collector.externalSchema.checkflag           | Yes    | Integer | 1                                                                                                             |
  90 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
  91 | collector.externalSchema.mappingFileLocation | Yes    | String  | ./etc/externalRepo/schema-map.json                                                                            |
  92 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
  93 | collector.externalSchema.schemasLocation     | Yes    | String  | ./etc/externalRepo/                                                                                           |
  94 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
  95 | event.externalSchema.schemaRefPath           | No     | String  | $.event.stndDefinedFields.schemaReference                                                                     |
  96 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
  97 | event.externalSchema.stndDefinedDataPath     | No     | String  | $.event.stndDefinedFields.data                                                                                |
  98 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
  99 | ves_3gpp_fault_supervision_publish_url       | Yes    | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
 100 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
 101 | ves_3gpp_provisioning_publish_url            | Yes    | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
 102 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
 103 | ves_3gpp_hearbeat_publish_url                | Yes    | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
 104 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
 105 | ves_3gpp_performance_assurance_publish_url   | Yes    | String  | http://message-router.onap.svc.cluster.local:3904/events/unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
 106 +----------------------------------------------+--------+---------+---------------------------------------------------------------------------------------------------------------+
 107
 108 Validation overview
 109 -------------------
 110
 111 This mechanism can be used to validate any JSON content incoming as JsonNode using OpenAPI standardized schemas.
 112 During validation externally located schemas are mapped to local schema files.
 113
 114 Validated JSON must have one field that will refer to an external schema, which will be mapped to local file and then
 115 validation of any chosen part of JSON is executed using local schema.
 116
 117 StndDefined validation is integrated with the event collecting functionality available under the endpoint
 118 */eventListener/v7*. Process of event collecting includes steps in the following order:
 119
 120 1. General event validation (1st stage validation)
 121 2. Event transformation
 122 3. **StndDefined event validation** (2nd stage validation)
 123 4. Event routing to DMaaP
 124
 125 Mapping file is cached on stndDefined validator creation, so it's not read every time validation is performed.
 126 Schemas' content couldn't be cached due to an external library restrictions (OpenAPI4j).
 127
 128 The value of the 'stndDefinedNamespace' field in any incoming stndDefined JSON event is used to match the topic from
 129 property *collector.dmaap.streamid*.
 130
 131 Requirements for stndDefined validation
 132 ---------------------------------------
 133
 134 To run stndDefined validation, both *collector.schema.checkflag* and *collector.externalSchema.checkflag* must be set to 1.
 135
 136 Despite the flag set, the validation will not start when:
 137
 138 - Domain of the incoming event is not 'stndDefined'.
 139 - General event validation (1st stage) failed.
 140 - Field of event referenced under the property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*):
 141     - Has an empty value.
 142     - Does not exist in the incoming event.
 143
 144 Validation scenarios
 145 --------------------
 146
 147 Positive scenario, which returns 202 Accepted HTTP code after successful stndDefined validation:
 148
 149 - *collector.schema.checkflag* and *collector.externalSchema.checkflag* is set to 1
 150 - Mapping file has valid format
 151 - Schema file mapped from referenced in the event is valid
 152 - The incoming event is valid against schema
 153
 154 Below are scenarios when, the stndDefined validation will end with failure and return 400 Bad Request HTTP code:
 155
 156 - One of stndDefined data fields has wrong type or value
 157 - StndDefined data has an empty body or is missing required field
 158 - Field of event referenced under the property event.externalSchema.schemaRefPath has publicURL which is not mapped in the schemas mappings
 159 - Field defining public schema in event (by default */event/stndDefinedFields/schemaReference*) after "#" has non existing reference in schema file
 160
 161 Schema repository description
 162 -----------------------------
 163
 164 Schemas and mapping file location might be configured to any local directory through properties in collector.properties
 165 as described in 'StndDefined properties' section.
 166
 167 By default schemas repository is located under */opt/app/VESCollector/etc/externalSchema* directory, as well as schemas mapping file called
 168 *schema-map.json*. Every organisation which adds or mounts external schemas should store them in folder named by
 169 organisation name. Further folders structure may be whatever as long as schemas are correctly referenced in the mapping
 170 file.
 171
 172 Sample directory tree of */opt/app/VESCollector/etc* directory:
 173
 174 .. code-block:: text
 175
 176     /opt/app/VESCollector/etc
 177     ├── ...
 178     └── externalRepo
 179         ├── schema-map.json
 180         └── 3gpp
 181             └── rep
 182                 └── sa5
 183                     └── MnS
 184                         └── blob
 185                             └── SA88-Rel16
 186                                 └── OpenAPI
 187                                     ├── faultMnS.yaml
 188                                     ├── heartbeatNtf.yaml
 189                                     ├── PerDataFileReportMnS.yaml
 190                                     └── provMnS.yaml
 191
 192 Routing of stndDefined domain events
 193 ------------------------------------
 194
 195 All events, except those with 'stndDefined' domain, are routed to DMaaP topics based on domain value. Events with
 196 'stndDefined' domain are sent to proper topic basing on field 'stndDefinedNamespace'.
 197
 198 This is the only difference from standard event routing, specific for 'stndDefined' domain. As in every other event
 199 routing value is being mapped for specific DMaaP stream. Stream ID to DMaaP channels mappings are located in
 200 */opt/app/VESCollector/etc/collector.properties* file under property *collector.dmaap.streamid*. Channels descriptions are in
 201 */opt/app/VESCollector/etc/DmaapConfig.json*, where destination DMaaP topics are selected.
 202
 203 With stndDefined domain managment 4 new mappings were added. Their routing has been described in the table below:
 204
 205     +---------------------------+--------------------------------+------------------------------------------------------+
 206     | Stream ID                 | Channel                        | DMaaP Stream                                         |
 207     +===========================+================================+======================================================+
 208     | 3GPP-FaultSupervision     | ves-3gpp-fault-supervision     | unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
 209     +---------------------------+--------------------------------+------------------------------------------------------+
 210     | 3GPP-Heartbeat            | ves-3gpp-heartbeat             | unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
 211     +---------------------------+--------------------------------+------------------------------------------------------+
 212     | 3GPP-Provisioning         | ves-3gpp-provisioning          | unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
 213     +---------------------------+--------------------------------+------------------------------------------------------+
 214     | 3GPP-PerformanceAssurance | ves-3gpp-performance-assurance | unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
 215     +---------------------------+--------------------------------+------------------------------------------------------+
 216
 217
 218 Error scenarios behaviour
 219 -------------------------
 220
 221 There are few error scenarios described in 'Validation scenarios' section. This section will describe user point of view
 222 of VES Collector behaviour when they happen. Messages returned as HTTP response contain data described below for each
 223 scenario.
 224
 225 1. StndDefined fields validation related errors
 226
 227 1.1. Schema file referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) not present in the schema repository.
 228
 229     +---------------------+------------------------------------------------------------------+
 230     | Property Name       | Property Description                                             |
 231     +=====================+==================================================================+
 232     | MessageId           | SVC2004                                                          |
 233     +---------------------+------------------------------------------------------------------+
 234     | Text                | "Invalid input value for %1 %2: %3"                              |
 235     +---------------------+------------------------------------------------------------------+
 236     | Variables           | %1 – “attribute”                                                 |
 237     |                     | %2 – "event.stndDefinedFields.schemaReference"                   |
 238     |                     | %3 – "Referred external schema not present in schema repository" |
 239     +---------------------+------------------------------------------------------------------+
 240     | HTTP status code(s) | 400 Bad request                                                  |
 241     +---------------------+------------------------------------------------------------------+
 242
 243 1.2. File referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) exists, but internal reference (part of URL after #) is incorrect.
 244
 245     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 246     | Property Name       | Property Description                                                                                                              |
 247     +=====================+===================================================================================================================================+
 248     | MessageId           | SVC2000                                                                                                                           |
 249     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 250     | Text                | The following service error occurred: %1. Error code is %2                                                                        |
 251     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 252     | Variables           | %1 - "event.stndDefinedFields.schemaReference value does not correspond to any external event schema file in externalSchema repo" |
 253     |                     | %2 - "400"                                                                                                                        |
 254     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 255     | HTTP status code(s) | 400 Bad request                                                                                                                   |
 256     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 257
 258 1.3. StndDefined validation executed, but event contents do not validate with referenced schema.
 259
 260     +---------------------+---------------------------------------------------------------------------------------------+
 261     | Property Name       | Property Description                                                                        |
 262     +=====================+=============================================================================================+
 263     | MessageId           | SVC2000                                                                                     |
 264     +---------------------+---------------------------------------------------------------------------------------------+
 265     | Text                | The following service error occurred: %1. Error code is %2                                  |
 266     +---------------------+---------------------------------------------------------------------------------------------+
 267     | Variables           | %1 - "event.stndDefinedFields.data invalid against event.stndDefinedFields.schemaReference" |
 268     |                     | %2 - "400"                                                                                  |
 269     +---------------------+---------------------------------------------------------------------------------------------+
 270     | HTTP status code(s) | 400 Bad request                                                                             |
 271     +---------------------+---------------------------------------------------------------------------------------------+
 272
 273 2. Problems with routing of stndDefined domain.
 274
 275 2.1. StndDefinedNamespace field not received in the incoming event.
 276
 277     +---------------------+-----------------------------------------------------+
 278     | Property Name       | Property Description                                |
 279     +=====================+=====================================================+
 280     | MessageId           | SVC2006                                             |
 281     +---------------------+-----------------------------------------------------+
 282     | Text                | Mandatory input %1 %2 is missing from request       |
 283     +---------------------+-----------------------------------------------------+
 284     | Variables           | %1 – “attribute”                                    |
 285     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
 286     +---------------------+-----------------------------------------------------+
 287     | HTTP status code(s) | 400 Bad Request                                     |
 288     +---------------------+-----------------------------------------------------+
 289
 290 2.2. StndDefinedNamespace field present, but value is empty.
 291
 292     +---------------------+-----------------------------------------------------+
 293     | Property Name       | Property Description                                |
 294     +=====================+=====================================================+
 295     | MessageId           | SVC2006                                             |
 296     +---------------------+-----------------------------------------------------+
 297     | Text                | Mandatory input %1 %2 is empty in request           |
 298     +---------------------+-----------------------------------------------------+
 299     | Variables           | %1 – “attribute”                                    |
 300     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
 301     +---------------------+-----------------------------------------------------+
 302     | HTTP status code(s) | 400 Bad Request                                     |
 303     +---------------------+-----------------------------------------------------+
 304
 305 2.3. StndDefinedNamespace field present, but value doesn't match any stream ID mapping.
 306
 307     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 308     | Property Name       | Property Description                                                                                                                      |
 309     +=====================+===========================================================================================================================================+
 310     | MessageId           | SVC2004                                                                                                                                   |
 311     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 312     | Text                | "Invalid input value for %1 %2: %3"                                                                                                       |
 313     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 314     | Variables           | %1 – “attribute”                                                                                                                          |
 315     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace"                                                                                       |
 316     |                     | %3 – "stndDefinedNamespace received not present in VES Collector routing configuration. Unable to route event to appropriate DMaaP topic" |
 317     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 318     | HTTP status code(s) | 400 Bad request                                                                                                                           |
 319     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 320
 321
 322