DL-Handler Doc

Issue-ID: DCAEGEN2-2028
Change-Id: Ifa26667a9ef1d6d8be711b1454d3e4ff6ea9a74b
Signed-off-by: Guobiao Mo <guobiaomo@chinamobile.com>

diff --git a/docs/sections/services/datalake-handler/DL-DES.PNG b/docs/sections/services/datalake-handler/DL-DES.PNG
new file mode 100644 (file)
index 0000000..aa3c494
Binary files /dev/null and b/docs/sections/services/datalake-handler/DL-DES.PNG differ

diff --git a/docs/sections/services/datalake-handler/arch.PNG b/docs/sections/services/datalake-handler/arch.PNG
new file mode 100644 (file)
index 0000000..eb809c6
Binary files /dev/null and b/docs/sections/services/datalake-handler/arch.PNG differ

diff --git a/docs/sections/services/datalake-handler/dbschema.PNG b/docs/sections/services/datalake-handler/dbschema.PNG
new file mode 100644 (file)
index 0000000..14c0d93
Binary files /dev/null and b/docs/sections/services/datalake-handler/dbschema.PNG differ

diff --git a/docs/sections/services/datalake-handler/feeder-arch.PNG b/docs/sections/services/datalake-handler/feeder-arch.PNG
new file mode 100644 (file)
index 0000000..cb995fa
Binary files /dev/null and b/docs/sections/services/datalake-handler/feeder-arch.PNG differ

diff --git a/docs/sections/services/datalake-handler/index.rst b/docs/sections/services/datalake-handler/index.rst
new file mode 100644 (file)
index 0000000..56ada5f
--- /dev/null
+++ b/docs/sections/services/datalake-handler/index.rst
@@ -0,0 +1,37 @@
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. http://creativecommons.org/licenses/by/4.0
+
+
+DataLake-Handler MS
+==============
+
+**DataLake-Handler MS** is a software component of ONAP that can systematically persist the events from DMaaP into supported Big Data storage systems. 
+It has a Admin UI, where a system administrator configures which Topics to be monitored, and to which data storage to store the data. 
+It is also used to manage the settings of the storage and associated data analytics tool. 
+The second part is the Feeder, which does the data transfer work and is horizontal scalable. 
+
+.. image:: DL-DES.PNG
+
+DataLake-Handler MS overview and functions
+------------------------------------------
+
+.. toctree::
+    :maxdepth: 1
+
+    ./overview.rst
+
+DataLake-Handler MS Installation Steps and Configurations
+---------------------------------------------------------
+
+.. toctree::
+    :maxdepth: 1
+
+    ./installation.rst
+
+DataLake-Handler MS Admin UI User Guide
+---------------------------------------
+
+.. toctree::
+    :maxdepth: 1
+
+    ./userguide.rst 

diff --git a/docs/sections/services/datalake-handler/installation.rst b/docs/sections/services/datalake-handler/installation.rst
new file mode 100644 (file)
index 0000000..2235198
--- /dev/null
+++ b/docs/sections/services/datalake-handler/installation.rst
@@ -0,0 +1,4 @@
+Installation
+============
+
+DataLake handler microservice can be deployed using ... (TODO) 
\ No newline at end of file

diff --git a/docs/sections/services/datalake-handler/overview.rst b/docs/sections/services/datalake-handler/overview.rst
new file mode 100644 (file)
index 0000000..51dab10
--- /dev/null
+++ b/docs/sections/services/datalake-handler/overview.rst
@@ -0,0 +1,76 @@
+.. This work is licensed under a Creative Commons Attribution 4.0\r
+   International License. http://creativecommons.org/licenses/by/4.0\r
+   \r
+.. _docs_Datalake_Handler_MS:\r
+\r
+Architecture\r
+------------\r
+\r
+\r
+Background\r
+~~~~~~~~~~\r
+There are large amount of data flowing among ONAP components, mostly via DMaaP and Web Services. \r
+For example, all events/feed collected by DCAE collectors go through DMaaP. \r
+DMaaP is backed by Kafka, which is a system for Publish-Subscribe, \r
+where data is not meant to be permanent and gets deleted after certain retention period. \r
+Kafka is not a database, means that data there is not for query.\r
+Though some components may store processed result into their local databases, most of the raw data will eventually lost. \r
+We should provide a systematic way to store these raw data, and even the processed result,\r
+which will serve as the source for data analytics and machine learning, providing insight to the network operation.\r
+\r
+\r
+Relations with Other ONAP Components\r
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r
+The architecture below depicts the DataLake MS as a part of ONAP. Only the relevant interactions and components are shown.\r
+\r
+.. image:: ./arch.PNG\r
+\r
+Note that not all data storage systems in the picture are supported. In R6, the following storage are supported:\r
+  - MongoDB\r
+  - Couchbase\r
+  - Elasticsearch and Kibana\r
+  - HDFS\r
+Depending on demands, new systems may be added to the supported list. In the following we use the term database for the storage, \r
+even though HDFS is a file system (but with simple settings, it can be treats as a database, e.g. Hive.)\r
+\r
+Note that once the data is stored in databases, other ONAP components and systems will directly query data from the databases, \r
+without interacting with DataLake Handler.\r
+\r
+Description\r
+~~~~~~~~~~~\r
+DataLate Handler's main function is to monitor and persist data flow through DMaaP. The databases are outside of ONAP scope, \r
+since the data is expected to be huge, and a database may be a complicated cluster consisting of thousand of nodes.\r
+\r
+Admin UI\r
+~~~~~~~~\r
+A system administrator uses DataLake Admin UI to:\r
+  - Configure external database connections, such as host, port, login.\r
+  - Configure which Topics to monitor, which databases to store the data for each Topic.\r
+  - Pre-configured 3rd Party Tools dashboards and templates.\r
+\r
+This UI tool is used to manage all the Dayalake settings stored in MariaDB. Here is the database schema:\r
+\r
+.. image:: ./dbschema.PNG\r
+\r
+Feeder\r
+~~~~~~\r
+Architecture\r
+.. image:: ./feeder-arch.PNG\r
+\r
+Features\r
+\r
+   - Read data directly from Kafka for performance.\r
+   - Support for pluggable databases. To add a new database, we only need to implement its corrosponding service.\r
+   - Support REST API for inter-component communications. Besides managing DatAlake settings in MariaDB, \r
+   Admin UI also use this API to start/stop Feeder, query Feeder status and statistics.\r
+   - Use MariaDB to store settings.\r
+   - Support data processing features. Before persisting data, data can be massaged in Feeder. \r
+   Currently two features are implemented: Correlate Cleared Message (in org.onap.datalake.feeder.service.db.ElasticsearchService) \r
+   and Flatten JSON Array (org.onap.datalake.feeder.service.StoreService).    \r
+   - Connection to Kafka and DBs are secured\r
+\r
+\r
+Links\r
+~~~~~\r
+   - DataLake Development Environment Setup https://wiki.onap.org/display/DW/DataLake+Development+Environment+Setup\r
+   - Source Code https://gerrit.onap.org/r/gitweb?p=dcaegen2/services.git;a=tree;f=components/datalake-handler;hb=HEAD\r

diff --git a/docs/sections/services/datalake-handler/userguide.rst b/docs/sections/services/datalake-handler/userguide.rst
new file mode 100644 (file)
index 0000000..4a0957f
--- /dev/null
+++ b/docs/sections/services/datalake-handler/userguide.rst
@@ -0,0 +1,4 @@
+Admin UI User Guide
+---------------------
+ 
+To be filled.

diff --git a/docs/sections/services/son-handler/son_handler_overview.rst b/docs/sections/services/son-handler/son_handler_overview.rst
index 23abbaa..41dbd84 100644 (file)
--- a/docs/sections/services/son-handler/son_handler_overview.rst
+++ b/docs/sections/services/son-handler/son_handler_overview.rst
@@ -24,7 +24,7 @@ The core logic is implemented as 3 threads - main thread, child thread(s) for ha
 (b) Autonomously taking actions for ANR updates\r
 (c) Preparing the message contents required by SDN-R to re-configure the RAN nodes with PCI/ANR updates\r
 \r
-The logic may not be 100% fool-proof (i.e., cover all possible scenarios and boundary-counditions as in real field deployments), as well as the most efficient one. An attempt has been made to balance the usefulness for a PoC versus the complexity of handling all possible scenarios. It is intended to provide a good base for the community/users to enhance it further as required.\r
+The logic may not be 100% fool-proof (i.e., cover all possible scenarios and boundary-conditions as in real field deployments), as well as the most efficient one. An attempt has been made to balance the usefulness for a PoC versus the complexity of handling all possible scenarios. It is intended to provide a good base for the community/users to enhance it further as required.\r
 \r
 The details of the state machines of all the threads in the core logic are available in https://wiki.onap.org/pages/viewpage.action?pageId=56131985.\r
 \r