HV-VES (High Volume VES)
========================
-:Date: 2018-10-05
.. contents::
- :depth: 3
+ :depth: 4
+
..
Overview
.. _HV-VES Collector: ../../services/ves-hv/index.html
+.. _tcp_endpoint:
TCP Endpoint
============
HV-VES is exposed as NodePort service on Kubernetes cluster on port 30222/tcp.
-It uses plain TCP connections tunneled in SSL/TLS or can be run in insecure manner without data encryption on the socket.
-Without TLS client authentication/authorization is not possible.
-Connections are stream-based (as opposed to request-based) and long running.
+By default, as of the Frankfurt release, all TCP communications are secured using
+SSL/TLS. Plain, insecure TCP connections without socket data encryption can be enabled if needed.
+ (see ref:`ssl_tls_authorization`).
+
+Without TLS, client authentication/authorization is not possible.
+Connections are stream-based (as opposed to request-based) and long-running.
Communication is wrapped with thin Wire Transfer Protocol, which mainly provides delimitation.
.. literalinclude:: WTP.asn
:language: asn
-Payload is binary-encoded, currently using Google Protocol Buffers representation of the VES Common Header.
+Payload is binary-encoded, using Google Protocol Buffers (GPB) representation of the VES Event.
.. literalinclude:: VesEvent.proto
:language: protobuf
-The PROTO file, which contains the VES CommonHeader, comes with a binary-type Payload parameter, where domain-specific
-data shall be placed. Domain-specific data are encoded as well with GPB, and they do require a domain-specific
-PROTO file to decode the data.
+HV-VES makes routing decisions based on the content of the **domain** field or **stndDefinedNamespace** field in case of stndDefined events.
+
+The PROTO file, which contains the VES CommonEventHeader, comes with a binary-type Payload (eventFields) parameter, where domain-specific
+data should be placed. Domain-specific data are encoded as well with GPB. A domain-specific PROTO file is required to decode the data.
+
+API towards DMaaP
+=================
+
+HV-VES Collector forwards incoming messages to a particular DMaaP Kafka topic based on the domain (or stndDefinedNamespace) and configuration. Every Kafka record is comprised of a key and a value. In case of HV-VES:
+
+- **Kafka record key** is a GPB-encoded `CommonEventHeader`.
+- **Kafka record value** is a GPB-encoded `VesEvent` (`CommonEventHeader` and domain-specific `eventFields`).
+
+In both cases raw bytes might be extracted using ``org.apache.kafka.common.serialization.ByteArrayDeserializer``. The resulting bytes might be further passed to ``parseFrom`` methods included in classes generated from GPB definitions. WTP is not used here - it is only used in communication between PNF/VNF and the collector.
+
+In case of Helm deployment routing is defined in values.yaml file in HV-VES Helm Chart.
+
+
+.. _supported_domains:
+
+Supported domains
+=================
+
+Domains that are currently supported by HV-VES:
+
+- perf3gpp - basic domain to Kafka topic mapping
+- stndDefined - specific routing, when event has this domain, then stndDefinedNamespace field value is mapped to Kafka topic
+
+For domains descriptions, see :ref:`domains_supported_by_hvves`
-HV-VES makes routing decisions based mostly on the content of the **Domain** parameter in VES Common Header.
+.. _hv_ves_behaviors:
+HV-VES behaviors
+================
-Healthcheck
-===========
+Connections with HV-VES are stream-based (as opposed to request-based) and long-running. In case of interrupted or closed connection, the collector logs such event but does not try to reconnect to client.
+Communication is wrapped with thin Wire Transfer Protocol, which mainly provides delimitation. Wire Transfer Protocol Frame:
-Inside HV-VES docker container runs small http service for healthcheck - exact port for this service can be configured
-at deployment using `--health-check-api-port` command line option.
+- is dropped after decoding and validating and only GPB is used in further processing.
+- has to start with **MARKER_BYTE**, as defined in protocol specification (see :ref:`tcp_endpoint`). If **MARKER_BYTE** is invalid, HV-VES disconnects from client.
-This service exposes single endpoint **GET /health/ready** which returns **HTTP 200 OK** in case HV-VES is healthy
-and ready for connections. Otherwise it returns **HTTP 503 Service Unavailable** with short reason of unhealthiness.
+HV-VES decodes only CommonEventHeader from GPB message received. Collector does not decode or validate the rest of the GPB message and publishes it to Kafka topic intact.
+Kafka topic for publishing events with specific domain can be configured through Consul service as described in :ref:`run_time_configuration`.
+In case of Kafka service unavailability, the collector drops currently handled messages and disconnects the client.
+Messages handling:
+- HV-VES Collector skips messages with unknown/invalid GPB CommonEventHeader format.
+- HV-VES Collector skips messages with unsupported domain. Domain is unsupported if there is no route for it in configuration (see :ref:`run_time_configuration`).
+- HV-VES Collector skips messages with invalid Wire Frame format, unsupported WTP version or inconsistencies of data in the frame (other than invalid **MARKER_BYTE**).
+- HV-VES Collector interrupts connection when it encounters a message with too big GPB payload. Default maximum size and ways to change it are described in :ref:`deployment`.
+.. note:: xNF (VNF/PNF) can split messages bigger than 1 MiB and set `sequence` field in CommonEventHeader accordingly. It is advised to use smaller than 1 MiB messages for GPBs encoding/decoding efficiency.
+- Skipped messages (for any of the above reasons) might not leave any trace in HV-VES logs.
Code Review / dcaegen2.git / blobdiff