docs/sections/services/ves-http/stnd-defined-validation.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 StndDefined Events Collection Mechanism
   5 =======================================
   6
   7 Description
   8 -----------
   9
  10 This mechanism can be used to validate any JSON content incoming as JsonNode using OpenAPI standardized schemas.
  11 During validation externally located schemas are mapped to local schema files.
  12
  13 Validated JSON must have one field that will refer to an external schema, which will be mapped to local file and then
  14 validation of any chosen part of JSON is executed using local schema.
  15
  16 StndDefined validation is integrated with the event collecting functionality available under the endpoint
  17 */eventListener/v7*. Process of event collecting includes steps in the following order:
  18
  19 1. General event validation (1st stage validation)
  20 2. Event transformation
  21 3. **StndDefined event validation** (2nd stage validation)
  22 4. Event routing to DMaaP
  23
  24 Mapping file is cached on stndDefined validator creation, so it's not read every time validation is performed.
  25 Schemas' content couldn't be cached due to an external library restrictions (OpenAPI4j).
  26
  27 The value of the 'stndDefinedNamespace' field in any incoming stndDefined JSON event is used to match the topic from
  28 property *collector.dmaap.streamid*.
  29
  30 StndDefined properties
  31 ----------------------
  32
  33 There are 5 additional properties related to stndDefined validation in collector.properties file.
  34
  35 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
  36 | Name                                         | Description                                                                    | Example                                                                        |
  37 +==============================================+================================================================================+================================================================================+
  38 | collector.externalSchema.checkflag           | Flag is responsible for turning on/off stndDefined data validation.            | -1 or 1                                                                        |
  39 |                                              | By default this flag is set to 1, which means that the validation is enabled.  |                                                                                |
  40 |                                              | In case flag is set to -1, validation is disabled.                             |                                                                                |
  41 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
  42 | collector.externalSchema.mappingFileLocation | This should be a local filesystem path to file with mappings of public URLs    | etc/externalRepo/schema-map.json                                               |
  43 |                                              | to local URLs.                                                                 |                                                                                |
  44 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
  45 | collector.externalSchema.schemasLocation     | Schemas location is a directory under which stndDefined validator will search  | ./etc/externalRepo/ and first mapping from example mapping file below is taken,|
  46 |                                              | for local schemas.                                                             | validator will look for schema under the path:                                 |
  47 |                                              |                                                                                | ./etc/externalRepo/3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml  |
  48 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
  49 | event.externalSchema.schemaRefPath           | This is an internal path from validated JSON. It should define which field     | $.event.stndDefinedFields.schemaReference                                      |
  50 |                                              | will be taken as public schema reference, which is later mapped.               |                                                                                |
  51 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
  52 | event.externalSchema.stndDefinedDataPath     | This is internal path from validated JSON.                                     | $.event.stndDefinedFields.data                                                 |
  53 |                                              | It should define which field will be validated.                                |                                                                                |
  54 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
  55
  56 Format of the schema mapping file is a JSON file with list of mappings, as shown in the example below.
  57
  58 .. code-block:: json
  59
  60     [
  61       {
  62         "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/tree/SA88-Rel16/OpenAPI/faultMnS.yaml",
  63         "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml"
  64       },
  65       {
  66         "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/tree/SA88-Rel16/OpenAPI/heartbeatNtf.yaml",
  67         "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/heartbeatNtf.yaml"
  68       },
  69       {
  70         "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/tree/SA88-Rel16/OpenAPI/PerDataFileReportMnS.yaml",
  71         "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/PerDataFileReportMnS.yaml"
  72       },
  73       {
  74         "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/tree/SA88-Rel16/OpenAPI/provMnS.yaml",
  75         "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/provMnS.yaml"
  76       }
  77     ]
  78
  79 Collector.properties content may be overridden when deploying VES Collector via Cloudify. To keep VES settings consistent
  80 listed above properties has been updated in the VES Collector Cloudify blueprint (in blueprints/k8s-ves.yaml file under
  81 dcaegen2/platform/blueprints project) and in componentspec file (in dpo/spec/vescollector-componentspec.json file
  82 in VES project) which may be used for generation of VES Collector Cloudify blueprints in some scenarios.
  83
  84
  85 Requirements for stndDefined validation
  86 ---------------------------------------
  87
  88 To run stndDefined validation, both *collector.schema.checkflag* and *collector.externalSchema.checkflag* must be set to 1.
  89
  90 Despite the flag set, the validation will not start when:
  91
  92 - Domain of the incoming event is not 'stndDefined'.
  93 - General event validation (1st stage) failed.
  94 - Field of event referenced under the property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*):
  95     - Has an empty value.
  96     - Does not exist in the incoming event.
  97
  98 Validation scenarios
  99 --------------------
 100
 101 Positive scenario, which returns 202 Accepted HTTP code after successful stndDefined validation:
 102
 103 - *collector.schema.checkflag* and *collector.externalSchema.checkflag* is set to 1
 104 - Mapping file has valid format
 105 - Schema file mapped from referenced in the event is valid
 106 - The incoming event is valid against schema
 107
 108 Below are scenarios when, the stndDefined validation will end with failure and return 400 Bad Request HTTP code:
 109
 110 - One of stndDefined data fields has wrong type or value
 111 - StndDefined data has an empty body or is missing required field
 112 - Field of event referenced under the property event.externalSchema.schemaRefPath has publicURL which is not mapped in the schemas mappings
 113 - Field defining public schema in event (by default */event/stndDefinedFields/schemaReference*) after "#" has non existing reference in schema file
 114
 115 Schemas repository description
 116 ------------------------------
 117
 118 Schemas and mapping file location might be configured to any local directory through properties in collector.properties
 119 as described in 'StndDefined properties' section.
 120
 121 By default schemas repository is located under *etc/externalSchema* directory, as well as schemas mapping file called
 122 *schema-map.json*. There are files stored in the project repository which are schemas from 3GPP organisation. Every
 123 organisation which adds or mounts external schemas should store them in folder named by organisation
 124 name. Further folders structure may be whatever as long as schemas are correctly referenced in the mapping file.
 125
 126 Sample directory tree of *etc* directory:
 127
 128 .. code-block:: text
 129
 130     etc
 131     ├── ...
 132     └── externalRepo
 133         ├── schema-map.json
 134         └── 3gpp
 135             └── rep
 136                 └── sa5
 137                     └── data-models
 138                         └── blob
 139                             └── REL-16
 140                                 └── OpenAPI
 141                                     ├── faultMnS.yaml
 142                                     ├── heartbeatNtf.yaml
 143                                     ├── PerDataFileReportMnS.yaml
 144                                     └── provMnS.yaml
 145
 146 Routing of stndDefined domain events
 147 ------------------------------------
 148
 149 All events, expect those with 'stndDefined' domain, are routed to DMaaP topics basing on domain value. Events with
 150 'stndDefined' domain are sent to proper topic basing on field 'stndDefinedNamespace'.
 151
 152 This is the only difference from standard event routing, specific for 'stndDefined' domain. As in every other event
 153 routing value is being mapped for specific DMaaP stream. Stream ID to DMaaP channels mappings are located in
 154 *etc/collector.properties* file under property *collector.dmaap.streamid*. Channels descriptions are in
 155 *etc/DmaapConfig.json*, where destination DMaaP topics are selected.
 156
 157 With stndDefined domain managment 4 new mappings were added. Their routing has been described in the table below:
 158
 159     +---------------------------+--------------------------------+------------------------------------------------------+
 160     | Stream ID                 | Channel                        | DMaaP Stream                                         |
 161     +===========================+================================+======================================================+
 162     | 3GPP-FaultSupervision     | ves-3gpp-fault-supervision     | unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
 163     +---------------------------+--------------------------------+------------------------------------------------------+
 164     | 3GPP-Heartbeat            | ves-3gpp-heartbeat             | unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
 165     +---------------------------+--------------------------------+------------------------------------------------------+
 166     | 3GPP-Provisioning         | ves-3gpp-provisioning          | unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
 167     +---------------------------+--------------------------------+------------------------------------------------------+
 168     | 3GPP-PerformanceAssurance | ves-3gpp-performance-assurance | unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
 169     +---------------------------+--------------------------------+------------------------------------------------------+
 170
 171
 172 Error scenarios behaviour
 173 -------------------------
 174
 175 There are few error scenarios described in 'Validation scenarios' section. This section will describe user point of view
 176 of VES Collector behaviour when they happen. Messages returned as HTTP response contain data described below for each
 177 scenario.
 178
 179 1. StndDefined fields validation related errors
 180
 181 1.1. Schema file referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) not present in the schema repository.
 182
 183     +---------------------+------------------------------------------------------------------+
 184     | Property Name       | Property Description                                             |
 185     +=====================+==================================================================+
 186     | MessageId           | SVC2004                                                          |
 187     +---------------------+------------------------------------------------------------------+
 188     | Text                | "Invalid input value for %1 %2: %3"                              |
 189     +---------------------+------------------------------------------------------------------+
 190     | Variables           | %1 – “attribute”                                                 |
 191     |                     | %2 – "event.stndDefinedFields.schemaReference"                   |
 192     |                     | %3 – "Referred external schema not present in schema repository" |
 193     +---------------------+------------------------------------------------------------------+
 194     | HTTP status code(s) | 400 Bad request                                                  |
 195     +---------------------+------------------------------------------------------------------+
 196
 197 1.2. File referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) exists, but internal reference (part of URL after #) is incorrect.
 198
 199     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 200     | Property Name       | Property Description                                                                                                              |
 201     +=====================+===================================================================================================================================+
 202     | MessageId           | SVC2000                                                                                                                           |
 203     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 204     | Text                | The following service error occurred: %1. Error code is %2                                                                        |
 205     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 206     | Variables           | %1 - "event.stndDefinedFields.schemaReference value does not correspond to any external event schema file in externalSchema repo" |
 207     |                     | %2 - "400"                                                                                                                        |
 208     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 209     | HTTP status code(s) | 400 Bad request                                                                                                                   |
 210     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 211
 212 1.3. StndDefined validation executed, but event contents do not validate with referenced schema.
 213
 214     +---------------------+---------------------------------------------------------------------------------------------+
 215     | Property Name       | Property Description                                                                        |
 216     +=====================+=============================================================================================+
 217     | MessageId           | SVC2000                                                                                     |
 218     +---------------------+---------------------------------------------------------------------------------------------+
 219     | Text                | The following service error occurred: %1. Error code is %2                                  |
 220     +---------------------+---------------------------------------------------------------------------------------------+
 221     | Variables           | %1 - "event.stndDefinedFields.data invalid against event.stndDefinedFields.schemaReference" |
 222     |                     | %2 - "400"                                                                                  |
 223     +---------------------+---------------------------------------------------------------------------------------------+
 224     | HTTP status code(s) | 400 Bad request                                                                             |
 225     +---------------------+---------------------------------------------------------------------------------------------+
 226
 227 2. Problems with routing of stndDefined domain.
 228
 229 2.1. StndDefinedNamespace field not received in the incoming event.
 230
 231     +---------------------+-----------------------------------------------------+
 232     | Property Name       | Property Description                                |
 233     +=====================+=====================================================+
 234     | MessageId           | SVC2006                                             |
 235     +---------------------+-----------------------------------------------------+
 236     | Text                | Mandatory input %1 %2 is missing from request       |
 237     +---------------------+-----------------------------------------------------+
 238     | Variables           | %1 – “attribute”                                    |
 239     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
 240     +---------------------+-----------------------------------------------------+
 241     | HTTP status code(s) | 400 Bad Request                                     |
 242     +---------------------+-----------------------------------------------------+
 243
 244 2.2. StndDefinedNamespace field present, but value is empty.
 245
 246     +---------------------+-----------------------------------------------------+
 247     | Property Name       | Property Description                                |
 248     +=====================+=====================================================+
 249     | MessageId           | SVC2006                                             |
 250     +---------------------+-----------------------------------------------------+
 251     | Text                | Mandatory input %1 %2 is empty in request           |
 252     +---------------------+-----------------------------------------------------+
 253     | Variables           | %1 – “attribute”                                    |
 254     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
 255     +---------------------+-----------------------------------------------------+
 256     | HTTP status code(s) | 400 Bad Request                                     |
 257     +---------------------+-----------------------------------------------------+
 258
 259 2.3. StndDefinedNamespace field present, but value doesn't match any stream ID mapping.
 260
 261     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 262     | Property Name       | Property Description                                                                                                                      |
 263     +=====================+===========================================================================================================================================+
 264     | MessageId           | SVC2004                                                                                                                                   |
 265     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 266     | Text                | "Invalid input value for %1 %2: %3"                                                                                                       |
 267     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 268     | Variables           | %1 – “attribute”                                                                                                                          |
 269     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace"                                                                                       |
 270     |                     | %3 – "stndDefinedNamespace received not present in VES Collector routing configuration. Unable to route event to appropriate DMaaP topic" |
 271     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 272     | HTTP status code(s) | 400 Bad request                                                                                                                           |
 273     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 274
 275
 276