Introduction
""""""""""""
-DataFile Collector (DFC) is a part of DCAEGEN2. Some information about DFC and the reasons of its implementation can be found here: `5G bulk PM wiki page`_.
+DataFile Collector (DFC) is a part of DCAEGEN2. Some information about DFC and the reasons of its implementation can be
+found here: `5G bulk PM wiki page`_.
.. _5G bulk PM wiki page: https://wiki.onap.org/display/DW/5G+-+Bulk+PM
"""""""""""
DFC will interact with the DMaaP Message Router and with the Data Router via secured protocol, using json files.
So far, the implemented protocols are sftp and ftp(e)s.
+
+Retry mechanism
+"""""""""""""""
+DFC is designed to retry downloading and publishing of files in order to recover from temporary faults.
+Each time an event is received, DFC will try to download and and publish each previously unpublished file in the event.
+The event is received from the Message Router (MR), the files are fetched from a PNF and are published to Data Router
+(DR).
+Both fetching of a file and publishing is retried a number of times with an increasing delay between each attempt.
+After a number of attempts, the DFC will log an error message and give up. Failing of processing of one file does not
+affect the handling of others.
\ No newline at end of file
.. code-block:: json\r
\r
"keyCert": <path to DFC certificate>\r
- "keyPassword": <pssword for DFC certificate>\r
- "trustedCA": <path to xNF certificate>\r
- "trustedCAPassword": <password for xNF certificate>\r
+ "keyPassword": <password for DFC certificate>\r
+ "trustedCa": <path to xNF certificate>\r
+ "trustedCaPassword": <password for xNF certificate>\r
+\r
+securityConfiguration\r
+"""""""""""""""""""""\r
+\r
+.. code-block:: json\r
+\r
+ "trustStorePath": <path to trust store>\r
+ "trustStorePasswordPath": <path to trsust store password>\r
+ "keyStorePath": <path to key store>\r
+ "keyStorePasswordPath": <path to key store password>\r
+ "enableDmaapCertAuth": <boolean>. If false, all information above are ignored.\r
+\r
\r
\r
Sample JSON configuration\r
}\r
},\r
"ftp": {\r
- "ftpesConfiguration": {\r
- "keyCert": "config/ftpKey.jks",\r
- "keyPassword": "secret",\r
- "trustedCA": "config/cacerts",\r
- "trustedCAPassword": "secret"\r
+ "ftpesConfiguration": {\r
+ "keyCert": "config/dfc.jks",\r
+ "keyPassword": "secret",\r
+ "trustedCa": "config/ftp.jks",\r
+ "trustedCaPassword": "secret"\r
+ }\r
+ },\r
+ "security": {\r
+ "trustStorePath" : "change it",\r
+ "trustStorePasswordPath" : "change it",\r
+ "keyStorePath" : "change it",\r
+ "keyStorePasswordPath" : "change it",\r
+ "enableDmaapCertAuth" : "false"\r
}\r
- }\r
}\r
}\r
Docker Container\r
----------------\r
\r
-**datafile** is delivered as a docker container that can be downloaded from onap:\r
+**datafile** is delivered as a docker container. The latest onap automatically built version can be downloaded from nexus:\r
\r
- ``docker run -d -p 8100:8100 -p 8433:8433 nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile``\r
+ ``docker run -d -p 8100:8100 -p 8433:8433\r
+ nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:latest``\r
+\r
+\r
+Another option is to pull the container first, and then run it with the image ID:\r
+\r
+ ``docker pull nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:latest``\r
+ ``docker images | grep 'datafile'``\r
+ ``docker run -d -p 8100:8100 -p 8433:8433 <image ID>``
\ No newline at end of file
The following command will download the latest datafile container from\r
nexus and launch it in the container named "datafile":\r
\r
- ``docker run -d -p 8100:8100 -p 8433:8433 nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile``\r
+ ``docker run -d -p 8100:8100 -p 8433:8433\r
+ nexus3.onap.org:10001/onap/org.onap.dcaegen2.collectors.datafile.datafile-app-server:latest``\r
\r
**Where is the log file?**\r
\r
-The log file is located under /opt/log and called application.log.\r
+The log file is located under /var/log/ONAP/ and called application.log.
\ No newline at end of file
Release Notes
=============
-Version: 1.0.5
+Version 1.2.0
+-------------
+
+:Release Date: 2019-04-11 (Dublin, M4)
+
+Messages are now handled in parallel
+
+Retry mechanism implemented
+
+Adapting to ONAP logging standard
+
+
+Version: 1.1.1
--------------
:Release Date: 2019-01-30 (Casablanca Maintenance fixes)
**Bug Fixes**
DCAEGEN2-940 - Larger files of size 100Kb publish to DR
+
DCAEGEN2-941 - DFC error after running over 12 hours
+
DCAEGEN2-1001 - Multiple Fileready notification not handled
Interaction
"""""""""""
-PM Mapper will interact with the DMaaP Bus Controller for Subscribing to Data Router feeds and publishing to Message Router. PM Mapper also intracts with the Config Binding Service to get configuration information.
+PM Mapper interacts with the Config Binding Service to get configuration information.
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.\r
-.. http://creativecommons.org/licenses/by/4.0\r
-\r
-Configuration\r
-=============\r
-\r
-Filtering\r
-"""""""""\r
-PM mapper maps PM XML files to performance VES event by applying the mapper filtering information. Mapper filtering is configured during instantiation through cloudify manager.\r
-Mapper filtering is based on the PM dictionary fields.\r
-PM mapper expects the filter in the following JSON format:\r
-\r
-::\r
-\r
-\r
- "filters":[{\r
- "pmDefVsn": "1.3",\r
- "nfType": "gnb",\r
- "vendor": "Ericsson",\r
- "measTypes": [ "attTCHSeizures", "succTCHSeizures" ]\r
- }]\r
-\r
-\r
-\r
-==================== ============================ ================================\r
-Field Description Type\r
-==================== ============================ ================================\r
-pmDefVsn PM Dictionary version. String\r
-vendor Vendor of the xNF type. String\r
-nfType nfType is vendor String\r
- defined and should match the\r
- string used in file ready\r
- eventName.\r
-measTypes Measurement name used in PM Array of String\r
- file in 3GPP format where\r
- specified, else vendor\r
- defined.\r
-==================== ============================ ================================\r
-\r
-Feed Name\r
-"""""""""\r
-A default feed name "bulk_pm_feed" is configured in PM Mapper blueprint but the feed name can be changed to a user defined value through cloudify manager.
\ No newline at end of file
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Configuration
+=============
+
+Filtering
+"""""""""
+PM mapper maps PM XML files to performance VES event by applying the mapper filtering information. Mapper filtering is configured during instantiation through cloudify manager.
+Mapper filtering is based on the PM dictionary fields.
+PM mapper expects the filter in the following JSON format:
+
+::
+
+
+ "filters":[{
+ "pmDefVsn": "1.3",
+ "nfType": "gnb",
+ "vendor": "Ericsson",
+ "measTypes": [ "attTCHSeizures", "succTCHSeizures" ]
+ }]
+
+
+
+==================== ============================ ================================
+Field Description Type
+==================== ============================ ================================
+pmDefVsn PM Dictionary version. String
+vendor Vendor of the xNF type. String
+nfType nfType is vendor String
+ defined and should match the
+ string used in file ready
+ eventName.
+measTypes Measurement name used in PM Array of String
+ file in 3GPP format where
+ specified, else vendor
+ defined.
+==================== ============================ ================================
+
+Message Router Topic Name
+"""""""""
+PM Mapper publishes the perf3gpp VES PM Events to the following authenticated MR topic;
+
+::
+
+ topic.org.onap.dmaap.mr.PM_MAPPER
-.. This work is licensed under a Creative Commons Attribution 4.0 International License.\r
-.. http://creativecommons.org/licenses/by/4.0\r
-\r
-Delivery\r
-========\r
-\r
-Docker Container\r
-----------------\r
-\r
-**PM Mapper** is delivered as a docker image that can be downloaded from ONAP docker registry:\r
-\r
- ``docker run -d --name pmmapper -e CONFIG_BINDING_SERVICE_SERVICE_HOST=<IP Required> -e CONFIG_BINDING_SERVICE_SERVICE_PORT=<Port Required> -e HOSTNAME=<HOSTNAME> nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.pm-mapper:1.0-SNAPSHOT``\r
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+Delivery
+========
+
+Docker Container
+----------------
+
+**PM Mapper** is delivered as a docker image that can be downloaded from ONAP docker registry:
+
+::
+
+ ``docker run -d --name pmmapper -e CONFIG_BINDING_SERVICE_SERVICE_HOST=<IP Required> -e CONFIG_BINDING_SERVICE_SERVICE_PORT=<Port Required> -e HOSTNAME=<HOSTNAME> nexus3.onap.org:10001/onap/org.onap.dcaegen2.services.pm-mapper:1.0-SNAPSHOT``
**PM Mapper** logs this information when connected to Consul, but cannot find a valid JSON configuration.
-====
-
-**Unable to Subscriber to Data Router Feed**
-
-::
-
- 2019-02-11T16:44:37.012Z main DEBUG org.onap.dcaegen2.services.pmmapper.config.ConfigHandler Mapper configuration:\nMapperConfig(messageRouterTopicName=unauthenticated.PM_VES_OUTPUT, busControllerConfig=BusControllerConfig(dcaeLocation=dcaeLocation, deliveryURL=deliveryURL, feedId=2, lastMod=lastMod, username=username, password=password, dataRouterSubscribeEndpoint=null), streamsSubscribes=MapperConfig.StreamsSubscribes(dmaapSubscriber=MapperConfig.DmaapSubscriber(dmaapInfo=MapperConfig.DmaapInfo(location=location, username=username, password=password, deliveryUrl=delivery_url, subscriberId=subsriber_id))))
- 2019-02-11T16:44:37.013Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber Starting subscription to DataRouter ENTRY
- 2019-02-11T16:44:37.013Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber e04d9fb4-5bc3-405b-89ea-6b824de8f137 INVOKE [ SYNCHRONOUS ]
- 2019-02-11T16:44:37.061Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber Request to bus controller executed with Response Code: '404' and Response Event: 'Not Found'.
- 2019-02-11T16:44:39.062Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber d4982d98-574d-42e4-9922-640610ffa34d INVOKE [ SYNCHRONOUS ]
- 2019-02-11T16:44:39.071Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber Request to bus controller executed with Response Code: '404' and Response Event: 'Not Found'.
- 2019-02-11T16:44:43.079Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber 05c73228-d04c-47c2-b897-8e936de5c8fd INVOKE [ SYNCHRONOUS ]
- 2019-02-11T16:44:43.089Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber Request to bus controller executed with Response Code: '404' and Response Event: 'Not Found'.
- 2019-02-11T16:44:51.129Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber c898b5c7-a347-496a-af3b-aeac661d4f1b INVOKE [ SYNCHRONOUS ]
- 2019-02-11T16:44:51.138Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber Request to bus controller executed with Response Code: '404' and Response Event: 'Not Found'.
- 2019-02-11T16:45:07.220Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber b190fbc1-ab91-44da-b393-727810925374 INVOKE [ SYNCHRONOUS ]
- 2019-02-11T16:45:07.234Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber Request to bus controller executed with Response Code: '404' and Response Event: 'Not Found'.
- 2019-02-11T16:45:07.235Z main INFO org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber EXIT
- Exception in thread "main" org.onap.dcaegen2.services.pmmapper.exceptions.TooManyTriesException: Failed to subscribe within appropriate amount of attempts
- at org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber.subscribe(DataRouterSubscriber.java:152)
- at org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber.subscribe(DataRouterSubscriber.java:150)
- at org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber.subscribe(DataRouterSubscriber.java:150)
- at org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber.subscribe(DataRouterSubscriber.java:150)
- at org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber.subscribe(DataRouterSubscriber.java:150)
- at org.onap.dcaegen2.services.pmmapper.datarouter.DataRouterSubscriber.start(DataRouterSubscriber.java:97)
- at org.onap.dcaegen2.services.pmmapper.App.main(App.java:53)
-
-This log is printed when pm mapper couldn't subscribe to the data router.
-----------------
1. Certificate generation:
This step is done manually currently using Test AAF instance in POD25. Required namespace, DCAE identity (dcae@dcae.onap.org), roles and Subject Alternative Names for all components are preset. Using the procedure desribed by AAF (using ``agent.sh``), the certificates are generated. Using the Java keystore file (``.jks``) generated from AAF, create the .pem files and load them into tls-init-container under dcaegen2/deployment repository. The image has a script that runs when the image is deployed. The script copies the certificate artifacts into a Kubernetes volume. The container is used as an "init-container" included in the Kubernetes pod for a component that needs to use TLS.
+
+ Current SAN listing::
+
+ bbs-event-processor, bbs-event-processor.onap, bbs-event-processor.onap.svc.cluster.local, config-binding-service, config-binding-service.onap, config-binding-service.onap.svc.cluster.local, dcae-cloudify-manager, dcae-cloudify-manager.onap, dcae-cloudify-manager.onap.svc.cluster.local, dcae-datafile-collector, dcae-datafile-collector.onap, dcae-datafile-collector.onap.svc.cluster.local, dcae-hv-ves-collector, dcae-hv-ves-collector.onap, dcae-hv-ves-collector.onap.svc.cluster.local, dcae-pm-mapper, dcae-pm-mapper.onap, dcae-pm-mapper.onap.svc.cluster.local, dcae-prh, dcae-prh.onap, dcae-prh.onap.svc.cluster.local, dcae-tca-analytics, dcae-tca-analytics.onap, dcae-tca-analytics.onap.svc.cluster.local, dcae-ves-collector, dcae-ves-collector.onap, dcae-ves-collector.onap.svc.cluster.local, deployment-handler, deployment-handler.onap, deployment-handler.onap.svc.cluster.local, holmes-engine-mgmt, holmes-engine-mgmt.onap, holmes-engine-mgmt.onap.svc.cluster.local, holmes-rule-mgmt, holmes-rules-mgmt.onap, holmes-rules-mgmt.onap.svc.cluster.local, inventory, inventory.onap, inventory.onap.svc.cluster.local, policy-handler, policy-handler.onap, policy-handler.onap.svc.cluster.local
2. Plugin and Blueprint:
Update blueprint to include new (optional) node property (tls_info) to the type definitions for the Kubernetes component types. The property is a dictionary with two elements:
Code Review / dcaegen2.git / commitdiff