--- /dev/null
+.. This work is licensed under a Creative Commons Attribution 4.0 International License.\r
+.. http://creativecommons.org/licenses/by/4.0\r
+\r
+.. _cdap-specification:\r
+\r
+Component specification (CDAP)\r
+==============================\r
+\r
+The CDAP component specification contains the following groups of\r
+information. Many of these are common to both CDAP and Docker components\r
+and are therefore described in the common specification.\r
+\r
+- :any:`Metadata <metadata>`\r
+- :any:`Interfaces <interfaces>` including the\r
+ associated :any:`Data Formats <data-formats>`\r
+- :any:`Parameters <parameters>` - for specifying parameters in your\r
+ AppConfig, AppPreferences, and ProgramPreferences to the Designer and\r
+ Policy. This of course is CDAP-specific and is described below.\r
+- :any:`Auxiliary Details <auxiliary-details>`\r
+- :any:`List of artifacts <artifacts>`\r
+\r
+Current Limitations and TODOs\r
+-----------------------------\r
+\r
+- The integration of DMD is likely to significantly change the\r
+ :any:`Interfaces <interfaces>` section in this specification, see\r
+ :any:`DMaaP abstraction <dmaap-abstraction>`.\r
+\r
+.. _parameters:\r
+\r
+Parameters\r
+----------\r
+\r
+There is a ``parameters`` section in your component specification. This\r
+section contains three optional keys: `app_config <#appconfig>`__,\r
+`app_preferences <#apppreferences>`__, and\r
+`propram_preferences <#programpreferences>`__:\r
+\r
+::\r
+\r
+ "parameters" : {\r
+ "app_config" : [ ...], \r
+ "app_preferences" : [ ...], \r
+ "program_preferences" : [...]\r
+ // any additional keys are ignored\r
+ }\r
+\r
+- Each section details the parameters that are a part of each of these\r
+ CDAP constructs (see below).\r
+- All such parameters will be exposed to the designer and to policy for\r
+ override.\r
+- These parameters should have default values specified by the\r
+ component developer where necessary, i.e., parameters that *must*\r
+ come from the designer/policy should not have defaults.\r
+- All of these keys are optional because not every CDAP application\r
+ uses preferences and not every application uses the AppConfig.\r
+ However, you should specify at least one, or else your application\r
+ will have no parameters exposed to policy or to the DCAE designer,\r
+ which means it would be non-configurable.\r
+- Despite the AppConfig being optional to *specify* in the case that\r
+ you have no parameters in your AppConfig, it is *required for\r
+ processing* in your application. That is because the DCAE platform\r
+ will place important information into your AppConfig as discussed\r
+ below.\r
+\r
+Parameter\r
+~~~~~~~~~\r
+\r
+The following CDAP specific definitions use ``param1`` to refer to the\r
+common parameter layout in\r
+:any:`Parameter <parameters>`\r
+\r
+AppConfig\r
+~~~~~~~~~\r
+\r
+The ``app_config`` key refers to the `CDAP AppConfig <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/configuration.html>`_.\r
+It is expected to be a JSON:\r
+\r
+::\r
+\r
+ "app_config" : [ // list of JSON\r
+ param1, // common parameter layout\r
+ ...\r
+ ]\r
+\r
+Unfortunately, at the time of writing, the AppConfig is a Java map of\r
+``string:string``, which means you cannot have more complex structures\r
+(than string) as any value in your AppConfig. However, there is a way to\r
+bypass this constraint: you can pass a JSON by encoding the JSON as a\r
+string. E.g., the ``json.dumps()`` and it’s converse ``loads`` methods\r
+in Python:\r
+\r
+::\r
+\r
+ >>> import json\r
+ >>> json.dumps({"foo" : "bar"}) # This is a real JSON\r
+ '{"foo": "bar"}' # It is now a string: pass this in as your parameter value\r
+ >>> json.loads('{"foo": "bar"}') # Do the equivelent of this in your application\r
+ {u'foo': u'bar'} # ...and you'll get back a real JSON \r
+ >>>\r
+\r
+The final AppConfig (after the designer and policy override parameter\r
+values) is passed into CDAP’s AppConfig API when starting the\r
+application.\r
+\r
+\r
+AppPreferences\r
+~~~~~~~~~~~~~~\r
+\r
+In addition to the CDAP AppConfig, the platform supports `Application Preferences <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/preferences.html#set-preferences>`_.\r
+The format of the ``app_preferences`` value is the same as the above:\r
+\r
+::\r
+\r
+ "app_preferences" : [ // list of JSON\r
+ param1, // common parameter layout\r
+ ...\r
+ ]\r
+\r
+The final Application Preferences JSON (after the designer and policy\r
+override parameter values) is passed into CDAP’s Preferences API when\r
+starting your application.\r
+\r
+\r
+ProgramPreferences\r
+~~~~~~~~~~~~~~~~~~\r
+\r
+Preferences can also be specified `per program <http://docs.cask.co/cdap/current/en/reference-manual/http-restful-api/lifecycle.html#program-lifecycle>`_\r
+in CDAP. This key’s value is a list of JSON with the following format:\r
+\r
+::\r
+\r
+ "program_preferences" : [ // note: this is a list of JSON \r
+ {\r
+ "program_id" : "program name 1", // the name of this CDAP program\r
+ "program_type" : "e.g., flows", // "must be one of flows, mapreduce, schedules, spark, workflows, workers, or services",\r
+ "program_pref" : [ // list of JSON\r
+ param1, // common parameter layout\r
+ ...\r
+ ]\r
+ },\r
+ // repeat for each program that receives a program_preferences JSON \r
+ ]\r
+\r
+Each ``program_pref`` JSON is passed into the CDAP API as the preference\r
+for ``program_id``.\r
+\r
+NOTE: for CDAP, this section is very likely to change when DMD is\r
+available. The *future* vision, as per :any:`DMaaP intentionally abstracted <dmaap-abstraction>` is\r
+that you would publish your data as a series of files on HDFS, and DMD\r
+will pick them up and send them to the appropriate DMaaP feeds or\r
+directly when needed.\r
+\r
+.. _auxiliary-details:\r
+\r
+Auxiliary Details\r
+-----------------\r
+\r
+*auxiliary* contains details about CDAP specific parameters.\r
+\r
++----------------------+----------------------+----------------------+\r
+| Property Name | Type | Description |\r
++======================+======================+======================+\r
+| streamname | string | *Required*. |\r
++----------------------+----------------------+----------------------+\r
+| artifact_name | string | |\r
++----------------------+----------------------+----------------------+\r
+| artifact_version | string | the version of your |\r
+| | | CDAP JAR artifact |\r
++----------------------+----------------------+----------------------+\r
+| namespace | string | the CDAP namespace |\r
+| | | to deploy into, |\r
+| | | default is ‘default’ |\r
++----------------------+----------------------+----------------------+\r
+| programs | array | contains each CDAP |\r
+| | | entity represented |\r
+| | | in the artifact |\r
++----------------------+----------------------+----------------------+\r
+| program_type | string | CDAP entity (eg |\r
+| | | “flows”) |\r
++----------------------+----------------------+----------------------+\r
+| program_id | string | name of CDAP entity |\r
+| | | (eg “WhoFlow”) |\r
++----------------------+----------------------+----------------------+\r
+\r
+Example:\r
+\r
+.. code:: json\r
+\r
+ "auxiliary": {\r
+ "streamname" : "who",\r
+ "artifact_name" : "HelloWorld",\r
+ "artifact_version" : "3.4.3",\r
+ "namespace" : "hw",\r
+ "programs" : [\r
+ {"program_type" : "flows", "program_id" : "WhoFlow"}, \r
+ {"program_type" : "services", "program_id" : "Greeting"},\r
+ ...\r
+ ],\r
+ }\r
+\r
+The ``programs`` key is identical to the ``program_preferences`` key\r
+discussed `above <#programpreferences>`__ except:\r
+\r
+- each JSON in the list does not contain ``program_pref``\r
+- this is required! You must include all of your programs in this, as\r
+ it is used to start each program as well as for DCAE to perform\r
+ periodic healthchecks on your application. Don’t forget about your\r
+ services; they are programs too.\r
Code Review / dcaegen2.git / blobdiff