docs/sections/services/ves-http/stnd-defined-validation.rst

   1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   2 .. http://creativecommons.org/licenses/by/4.0
   3
   4 StndDefined Events Collection Mechanism
   5 =======================================
   6
   7 Description
   8 -----------
   9
  10 This mechanism can be used to validate any JSON content incoming as JsonNode using OpenAPI standardized schemas.
  11 During validation externally located schemas are mapped to local schema files.
  12
  13 Validated JSON must have one field that will refer to an external schema, which will be mapped to local file and then
  14 validation of any chosen part of JSON is executed using local schema.
  15
  16 StndDefined validation is integrated with the event collecting functionality available under the endpoint
  17 */eventListener/v7*. Process of event collecting includes steps in the following order:
  18
  19 1. General event validation (1st stage validation)
  20 2. Event transformation
  21 3. **StndDefined event validation** (2nd stage validation)
  22 4. Event routing to DMaaP
  23
  24 Mapping file is cached on stndDefined validator creation, so it's not read every time validation is performed.
  25 Schemas' content couldn't be cached due to an external library restrictions (OpenAPI4j).
  26
  27 The value of the 'stndDefinedNamespace' field in any incoming stndDefined JSON event is used to match the topic from
  28 property *collector.dmaap.streamid*.
  29
  30 StndDefined properties
  31 ----------------------
  32
  33 There are 5 additional properties related to stndDefined validation in collector.properties file.
  34
  35 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
  36 | Name                                         | Description                                                                    | Example                                                                        | Note                                                                          |
  37 +==============================================+================================================================================+================================================================================+===============================================================================+
  38 | collector.externalSchema.checkflag           | Flag is responsible for turning on/off stndDefined data validation.            | -1 or 1                                                                        |                                                                               |
  39 |                                              | By default this flag is set to 1, which means that the validation is enabled.  |                                                                                |                                                                               |
  40 |                                              | In case flag is set to -1, validation is disabled.                             |                                                                                |                                                                               |
  41 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
  42 | collector.externalSchema.mappingFileLocation | This should be a local filesystem path to file with mappings of public URLs    | etc/externalRepo/schema-map.json                                               |                                                                               |
  43 |                                              | to local URLs.                                                                 |                                                                                |                                                                               |
  44 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
  45 | collector.externalSchema.schemasLocation     | Schemas location is a directory under which stndDefined validator will search  | ./etc/externalRepo/ and first mapping from example mappingFile below is taken, |                                                                               |
  46 |                                              | for local schemas.                                                             | validator will look for schema under the path:                                 |                                                                               |
  47 |                                              |                                                                                | ./etc/externalRepo/3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml  |                                                                               |
  48 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
  49 | event.externalSchema.schemaRefPath           | This is an internal path from validated JSON. It should define which field     | /event/stndDefinedFields/schemaReference                                       | In SDK version 1.4.2 this path doesn’t use JSON path notation (with . signs). |
  50 |                                              | will be taken as public schema reference, which is later mapped.               |                                                                                | It might change in further versions                                           |
  51 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
  52 | event.externalSchema.stndDefinedDataPath     | This is internal path from validated JSON.                                     | /event/stndDefinedFields/data                                                  | In SDK version 1.4.2 this path doesn’t use JSON path notation (with . signs). |
  53 |                                              | It should define which field will be validated.                                |                                                                                | It might change in further versions                                           |
  54 +----------------------------------------------+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+-------------------------------------------------------------------------------+
  55
  56 Format of the schema mapping file is a JSON file with list of mappings, as shown in the example below.
  57
  58 .. code-block:: json
  59
  60     [
  61         {
  62             "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml",
  63             "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/faultMnS.yaml"
  64         },
  65         {
  66             "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/REL-16/OpenAPI/heartbeatNtf.yaml",
  67             "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/heartbeatNtf.yaml"
  68         },
  69         {
  70             "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/REL-16/OpenAPI/PerDataFileReportMnS.yaml",
  71             "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/PerDataFileReportMnS.yaml"
  72         },
  73         {
  74             "publicURL": "https://forge.3gpp.org/rep/sa5/data-models/blob/master/OpenAPI/provMnS.yaml",
  75             "localURL": "3gpp/rep/sa5/data-models/blob/REL-16/OpenAPI/provMnS.yaml"
  76         }
  77     ]
  78
  79
  80 Requirements for stndDefined validation
  81 ---------------------------------------
  82
  83 To run stndDefined validation, both *collector.schema.checkflag* and *collector.externalSchema.checkflag* must be set to 1.
  84
  85 Despite the flag set, the validation will not start when:
  86
  87 - Domain of the incoming event is not 'stndDefined'.
  88 - General event validation (1st stage) failed.
  89 - Field of event referenced under the property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*):
  90     - Has an empty value.
  91     - Does not exist in the incoming event.
  92
  93 Validation scenarios
  94 --------------------
  95
  96 Positive scenario, which returns 202 Accepted HTTP code after successful stndDefined validation:
  97
  98 - *collector.schema.checkflag* and *collector.externalSchema.checkflag* is set to 1
  99 - Mapping file has valid format
 100 - Schema file mapped from referenced in the event is valid
 101 - The incoming event is valid against schema
 102
 103 Below are scenarios when, the stndDefined validation will end with failure and return 400 Bad Request HTTP code:
 104
 105 - One of stndDefined data fields has wrong type or value
 106 - StndDefined data has an empty body or is missing required field
 107 - Field of event referenced under the property event.externalSchema.schemaRefPath has publicURL which is not mapped in the schemas mappings
 108 - Field defining public schema in event (by default */event/stndDefinedFields/schemaReference*) after "#" has non existing reference in schema file
 109
 110 Schemas repository description
 111 ------------------------------
 112
 113 Schemas and mapping file location might be configured to any local directory through properties in collector.properties
 114 as described in 'StndDefined properties' section.
 115
 116 By default schemas repository is located under *etc/externalSchema* directory, as well as schemas mapping file called
 117 *schema-map.json*. There are files stored in the project repository which are schemas from 3GPP organisation. Every
 118 organisation which adds or mounts external schemas should store them in folder named by organisation
 119 name. Further folders structure may be whatever as long as schemas are correctly referenced in the mapping file.
 120
 121 Sample directory tree of *etc* directory:
 122
 123 .. code-block:: text
 124
 125     etc
 126     ├── ...
 127     └── externalRepo
 128         ├── schema-map.json
 129         └── 3gpp
 130             └── rep
 131                 └── sa5
 132                     └── data-models
 133                         └── blob
 134                             └── REL-16
 135                                 └── OpenAPI
 136                                     ├── faultMnS.yaml
 137                                     ├── heartbeatNtf.yaml
 138                                     ├── PerDataFileReportMnS.yaml
 139                                     └── provMnS.yaml
 140
 141 Routing of stndDefined domain events
 142 ------------------------------------
 143
 144 All events, expect those with 'stndDefined' domain, are routed to DMaaP topics basing on domain value. Events with
 145 'stndDefined' domain are sent to proper topic basing on field 'stndDefinedNamespace'.
 146
 147 This is the only difference from standard event routing, specific for 'stndDefined' domain. As in every other event
 148 routing value is being mapped for specific DMaaP stream. Stream ID to DMaaP channels mappings are located in
 149 *etc/collector.properties* file under property *collector.dmaap.streamid*. Channels descriptions are in
 150 *etc/DmaapConfig.json*, where destination DMaaP topics are selected.
 151
 152 With stndDefined domain managment 4 new mappings were added. Their routing has been described in the table below:
 153
 154     +---------------------------+--------------------------------+------------------------------------------------------+
 155     | Stream ID                 | Channel                        | DMaaP Stream                                         |
 156     +===========================+================================+======================================================+
 157     | 3GPP-FaultSupervision     | ves-3gpp-fault-supervision     | unauthenticated.SEC_3GPP_FAULTSUPERVISION_OUTPUT     |
 158     +---------------------------+--------------------------------+------------------------------------------------------+
 159     | 3GPP-Heartbeat            | ves-3gpp-heartbeat             | unauthenticated.SEC_3GPP_HEARTBEAT_OUTPUT            |
 160     +---------------------------+--------------------------------+------------------------------------------------------+
 161     | 3GPP-Provisioning         | ves-3gpp-provisioning          | unauthenticated.SEC_3GPP_PROVISIONING_OUTPUT         |
 162     +---------------------------+--------------------------------+------------------------------------------------------+
 163     | 3GPP-PerformanceAssurance | ves-3gpp-performance-assurance | unauthenticated.SEC_3GPP_PERFORMANCEASSURANCE_OUTPUT |
 164     +---------------------------+--------------------------------+------------------------------------------------------+
 165
 166
 167 Error scenarios behaviour
 168 -------------------------
 169
 170 There are few error scenarios described in 'Validation scenarios' section. This section will describe user point of view
 171 of VES Collector behaviour when they happen. Messages returned as HTTP response contain data described below for each
 172 scenario.
 173
 174 1. StndDefined fields validation related errors
 175
 176 1.1. Schema file referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) not present in the schema repository.
 177
 178     +---------------------+------------------------------------------------------------------+
 179     | Property Name       | Property Description                                             |
 180     +=====================+==================================================================+
 181     | MessageId           | SVC2004                                                          |
 182     +---------------------+------------------------------------------------------------------+
 183     | Text                | "Invalid input value for %1 %2: %3"                              |
 184     +---------------------+------------------------------------------------------------------+
 185     | Variables           | %1 – “attribute”                                                 |
 186     |                     | %2 – "event.stndDefinedFields.schemaReference"                   |
 187     |                     | %3 – "Referred external schema not present in schema repository" |
 188     +---------------------+------------------------------------------------------------------+
 189     | HTTP status code(s) | 400 Bad request                                                  |
 190     +---------------------+------------------------------------------------------------------+
 191
 192 1.2. File referred under the path from property *event.externalSchema.schemaRefPath* (by default */event/stndDefinedFields/schemaReference*) exists, but internal reference (part of URL after #) is incorrect.
 193
 194     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 195     | Property Name       | Property Description                                                                                                              |
 196     +=====================+===================================================================================================================================+
 197     | MessageId           | SVC2000                                                                                                                           |
 198     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 199     | Text                | The following service error occurred: %1. Error code is %2                                                                        |
 200     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 201     | Variables           | %1 - "event.stndDefinedFields.schemaReference value does not correspond to any external event schema file in externalSchema repo" |
 202     |                     | %2 - "400"                                                                                                                        |
 203     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 204     | HTTP status code(s) | 400 Bad request                                                                                                                   |
 205     +---------------------+-----------------------------------------------------------------------------------------------------------------------------------+
 206
 207 1.3. StndDefined validation executed, but event contents do not validate with referenced schema.
 208
 209     +---------------------+---------------------------------------------------------------------------------------------+
 210     | Property Name       | Property Description                                                                        |
 211     +=====================+=============================================================================================+
 212     | MessageId           | SVC2000                                                                                     |
 213     +---------------------+---------------------------------------------------------------------------------------------+
 214     | Text                | The following service error occurred: %1. Error code is %2                                  |
 215     +---------------------+---------------------------------------------------------------------------------------------+
 216     | Variables           | %1 - "event.stndDefinedFields.data invalid against event.stndDefinedFields.schemaReference" |
 217     |                     | %2 - "400"                                                                                  |
 218     +---------------------+---------------------------------------------------------------------------------------------+
 219     | HTTP status code(s) | 400 Bad request                                                                             |
 220     +---------------------+---------------------------------------------------------------------------------------------+
 221
 222 2. Problems with routing of stndDefined domain.
 223
 224 2.1. StndDefinedNamespace field not received in the incoming event.
 225
 226     +---------------------+-----------------------------------------------------+
 227     | Property Name       | Property Description                                |
 228     +=====================+=====================================================+
 229     | MessageId           | SVC2006                                             |
 230     +---------------------+-----------------------------------------------------+
 231     | Text                | Mandatory input %1 %2 is missing from request       |
 232     +---------------------+-----------------------------------------------------+
 233     | Variables           | %1 – “attribute”                                    |
 234     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
 235     +---------------------+-----------------------------------------------------+
 236     | HTTP status code(s) | 400 Bad Request                                     |
 237     +---------------------+-----------------------------------------------------+
 238
 239 2.2. StndDefinedNamespace field present, but value is empty.
 240
 241     +---------------------+-----------------------------------------------------+
 242     | Property Name       | Property Description                                |
 243     +=====================+=====================================================+
 244     | MessageId           | SVC2006                                             |
 245     +---------------------+-----------------------------------------------------+
 246     | Text                | Mandatory input %1 %2 is empty in request           |
 247     +---------------------+-----------------------------------------------------+
 248     | Variables           | %1 – “attribute”                                    |
 249     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace" |
 250     +---------------------+-----------------------------------------------------+
 251     | HTTP status code(s) | 400 Bad Request                                     |
 252     +---------------------+-----------------------------------------------------+
 253
 254 2.3. StndDefinedNamespace field present, but value doesn't match any stream ID mapping.
 255
 256     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 257     | Property Name       | Property Description                                                                                                                      |
 258     +=====================+===========================================================================================================================================+
 259     | MessageId           | SVC2004                                                                                                                                   |
 260     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 261     | Text                | "Invalid input value for %1 %2: %3"                                                                                                       |
 262     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 263     | Variables           | %1 – “attribute”                                                                                                                          |
 264     |                     | %2 – "event.commonEventHeader.stndDefinedNamespace"                                                                                       |
 265     |                     | %3 – "stndDefinedNamespace received not present in VES Collector routing configuration. Unable to route event to appropriate DMaaP topic" |
 266     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 267     | HTTP status code(s) | 400 Bad request                                                                                                                           |
 268     +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+
 269
 270
 271