docs/index.rst

   1
   2
   3
   4
   5 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
   6 .. http://creativecommons.org/licenses/by/4.0
   7 .. Copyright © 2017 AT&T Intellectual Property. All rights reserved.
   8
   9 Logging Enhancements Project
  10 ==================================================
  11 .. contents::
  12    :depth: 3
  13 ..
  14
  15 ONAP consists of many components and containers, and consequently writes
  16 to many logfiles. The volume of logger output may be enormous,
  17 especially when debugging. Large, disparate logfiles are difficult to
  18 monitor and analyze, and tracing requests across many files, file
  19 systems and containers is untenable without tooling.
  20
  21 The problem of decentralized logger output is addressed by analytics
  22 pipelines such as \ `*Elastic
  23 Stack* <https://www.elastic.co/products>`__ (ELK). Elastic Stack
  24 consumes logs, indexes their contents
  25 in \ `*Elasticsearch* <https://www.elastic.co/products/elasticsearch>`__,
  26 and makes them accessible, queryable and navigable via a sophisticated
  27 UI, \ `*Kibana
  28 Discover* <https://www.elastic.co/guide/en/kibana/current/discover.html>`__.
  29 This elevates the importance of standardization and machine-readability.
  30 Logfiles can remain browsable, but output can be simplified.
  31
  32 Logger configurations in ONAP are diverse and idiosyncratic. Addressing
  33 these issues will prevent costs from being externalized to consumers
  34 such as analytics. It also affords the opportunity to remedy any issues
  35 with the handling and propagation of contextual information such as
  36 transaction identifiers (presently passed as \ **X-ECOMP-RequestID **-
  37 to be\ ** X-ONAP-RequestID**). This propagation is critical to tracing
  38 requests as they traverse ONAP and related systems, and is the basis for
  39 many analytics functions.
  40
  41 Rationalized logger configuration and output also paves the way for
  42 other high-performance logger transports, including publishing directly
  43 to analytics via SYSLOG
  44 (`*RFC3164* <https://www.ietf.org/rfc/rfc3164.txt>`__, \ `*RFC5425* <https://www.ietf.org/rfc/rfc5425.txt>`__, \ `*RFC5426* <https://www.ietf.org/rfc/rfc5426.txt>`__)
  45 and streams, and mechanisms for durability.
  46
  47 Each change is believed to be individually beneficial:
  48
  49 1. The intention is to consolidate the required setup within this
  50    project however some changes and bug fixes might have to be applied
  51    in the relative component project, requiring each components'
  52    co-operation on the contribution.
  53
  54 2. There is an economy of scale if everything can happen under a single
  55    remit.
  56
  57 3. Standardization benefits all, including those who want to deviate
  58    from defaults.
  59
  60 ONAP Application Logging Guidelines v1.1
  61 ========================================
  62
  63 Introduction
  64 ------------
  65
  66 The purpose of ONAP logging is to capture information needed to operate,
  67 troubleshoot and report on the performance of the ONAP platform and its
  68 constituent components. Log records may be viewed and consumed directly
  69 by users and systems, indexed and loaded into a datastore, and used to
  70 compute metrics and generate reports.
  71
  72 The processing of a single client request will often involve multiple
  73 ONAP components and/or subcomponents (interchangeably referred to as
  74 application in this document). The ability to track flows across
  75 components is critical to understanding ONAPs behavior and performance.
  76 ONAP logging uses a universally unique RequestID value in log records to
  77 track the processing of every client request through all the ONAP
  78 components involved in its processing.
  79
  80 A reference configuration of \ `*Elastic
  81 Stack * <https://www.elastic.co/products>`__\ can be deployed
  82 using \ `*ONAP Operations
  83 Manager* <https://wiki.onap.org/display/DW/ONAP+Operations+Manager+Project>`__.
  84
  85 This document gives conventions you can follow to generate conformant,
  86 indexable logging output from your component.
  87
  88 How to Log
  89 ----------
  90
  91 ONAP prescribes conventions. The use of certain APIs and providers is
  92 recommended, but they are not mandatory. Most components log
  93 via \ `*EELF* <https://github.com/att/EELF>`__ or `*SLF4J* <https://www.slf4j.org/>`__ to
  94 a provider
  95 like \ `*Logback* <https://logback.qos.ch/>`__ or `*Log4j* <https://logging.apache.org/log4j/2.x/>`__.
  96
  97 ****
  98 EELF
  99 ****
 100
 101 EELF is the\ **Event and Error Logging Framework**, described
 102 at \ `*https://github.com/att/EELF* <https://github.com/att/EELF>`__.
 103
 104 EELF abstracts your choice of logging provider, and decorates the
 105 familiar Logger contracts with features like:
 106
 107 -  Localization.
 108
 109 -  Error codes.
 110
 111 -  Generated wiki documentation.
 112
 113 -  Separate audit, metric, security and debug logs.
 114
 115 EELF is a facade, so logging output is configured in two ways:
 116
 117 1. By selection of a logging provider such as Logback or Log4j,
 118    typically via the classpath.
 119
 120 2. By way of a provider configuration document,
 121    typically \ **logback.xml** or **log4j.xml**.
 122    See \ `*Providers* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-Providers>`__.
 123
 124 *****
 125 SLF4J
 126 *****
 127
 128 `*SLF4J* <https://www.slf4j.org/>`__ is a logging facade, and a humble
 129 masterpiece. It combines what's common to all major, modern Java logging
 130 providers into a single interface. This decouples the caller from the
 131 provider, and encourages the use of what's universal, familiar and
 132 proven.
 133
 134 EELF also logs via SLF4J's abstractions.
 135
 136 *********
 137 Providers
 138 *********
 139
 140 Logging providers are normally enabled by their presence in the
 141 classpath. This means the decision may have been made for you, in some
 142 cases implicitly by dependencies. If you have a strong preference then
 143 you can change providers, but since the implementation is typically
 144 abstracted behind EELF or SLF4J, it may not be worth the effort.
 145
 146 *******
 147 Logback
 148 *******
 149
 150 Logback is the most commonly used provider. It is generally configured
 151 by an XML document named \ **logback.xml**.
 152 See \ `*Configuration* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-Configuration>`__.
 153
 154 *************
 155 Log4j 2.X
 156 *************
 157
 158 Log4j 2.X is somewhat less common than Logback, but equivalent. It is
 159 generally configured by an XML document
 160 named \ **log4j.xml**. See \ `*Configuration* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-Configuration>`__.
 161
 162 *********
 163 Log4j 1.X
 164 *********
 165
 166 Avoid, since 1.X is EOL, and since it does not support escaping, so its
 167 output may not be
 168 machine-readable. See \ `*https://logging.apache.org/log4j/1.2/* <https://logging.apache.org/log4j/1.2/>`__.
 169
 170 This affects existing OpenDaylight-based components like SDNC and APPC,
 171 since ODL releases prior
 172 to \ `*Carbon* <https://www.opendaylight.org/what-we-do/current-release>`__ bundle
 173 Log4j 1.X, and make it difficult to replace. The \ `*Common Controller
 174 SDK
 175 Project* <https://wiki.onap.org/display/DW/Common+Controller+SDK+Project>`__ project
 176 targets ODL Carbon, so the problem should resolve in time.
 177
 178 What to Log
 179 ===========
 180
 181 The purpose of logging is to capture diagnostic information.
 182
 183 An important aspect of this is analytics, which requires tracing of
 184 requests between components. In a large, distributed system such as ONAP
 185 this is critical to understanding behavior and performance.
 186
 187 Messages, Levels, Components and Categories
 188 --------------------------------------------
 189
 190 It isn't the aim of this document to reiterate the basics, so advice
 191 here is general:
 192
 193 -  Use a logger. Consider using EELF.
 194
 195 -  Write log messages in English.
 196
 197 -  Write meaningful messages. Consider what will be useful to consumers
 198    of logger output.
 199
 200 -  Use errorcodes to characterise exceptions.
 201
 202 -  Log at the appropriate level. Be aware of the volume of logs that
 203    will be produced.
 204
 205 -  Log in a machine-readable format. See Conventions.
 206
 207 -  Log for analytics as well as troubleshooting.
 208
 209 Others have written extensively on this:
 210
 211 -  `*http://www.masterzen.fr/2013/01/13/the-10-commandments-of-logging/* <http://www.masterzen.fr/2013/01/13/the-10-commandments-of-logging/>`__
 212
 213 -  `*https://www.loggly.com/blog/how-to-write-effective-logs-for-remote-logging/* <https://www.loggly.com/blog/how-to-write-effective-logs-for-remote-logging/>`__
 214
 215 -  And so on.
 216
 217 Context
 218 -------
 219
 220 **TODO: more on the importance of transaction ID propagation.**
 221
 222 MDCs
 223 ====
 224
 225
 226 A Mapped Diagnostic Context (MDC) allows an arbitrary string-valued
 227 attribute to be attached to a Java thread. The MDC's value is then
 228 emitted with each log message. The set of MDCs associated with a log
 229 message is serialized as unordered name-value pairs (see `*Text
 230 Output* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-TextOutput>`__).
 231
 232 A good discussion of MDCs can be found
 233 at \ `*https://logback.qos.ch/manual/mdc.html* <https://logback.qos.ch/manual/mdc.html>`__.
 234
 235 **MDCs:**
 236
 237 -  Must be set as early in invocation as possible.
 238
 239 -  Must be unset on exit.
 240
 241 Logging
 242 -------
 243
 244 **Via SLF4J:**
 245
 246 .. code-block:: java
 247
 248  import java.util.UUID;
 249  import org.slf4j.Logger;
 250  import org.slf4j.LoggerFactory;
 251  import org.slf4j.MDC;
 252  // ...
 253  final Logger logger = LoggerFactory.getLogger(this.getClass());
 254  MDC.put("SomeUUID", UUID.randomUUID().toString());
 255  try {
 256      logger.info("This message will have a UUID-valued 'SomeUUID' MDC attached.");
 257  // ...
 258  }
 259  finally {
 260      MDC.clear();
 261  }
 262
 263 EELF doesn't directly support MDCs, but SLF4J will receive any MDC that
 264 is set (where **com.att.eelf.configuration.SLF4jWrapper** is the
 265 configured EELF provider):
 266
 267 .. code-block:: java
 268
 269  import java.util.UUID;
 270  import org.slf4j.Logger;
 271  import org.slf4j.LoggerFactory;
 272  import org.slf4j.MDC;
 273  import com.att.eelf.configuration.EELFLogger;
 274  import com.att.eelf.configuration.EELFManager;
 275  // ...
 276  final EELFLogger logger = EELFManager.getInstance().getLogger(this.getClass());
 277  MDC.put("SomeUUID", UUID.randomUUID().toString());
 278  try {
 279      logger.info("This message will have a UUID-valued 'SomeUUID' MDC attached.");
 280      // ...
 281  }
 282  finally {
 283      MDC.clear();
 284  }
 285
 286 Serializing
 287 -----------
 288
 289 Output of MDCs must ensure that:
 290
 291 -  All reported MDCs are logged with both name AND value. Logging output
 292    should not treat any MDCs as special.
 293
 294 -  All MDC names and values are escaped.
 295
 296 Escaping in Logback configuration can be achieved with:
 297
 298 .. code-block:: bash
 299
 300  %replace(%replace(%mdc){'\\t','\\\\\\\\t'}){'\\n','\\\\\\\\n'}
 301
 302 MDC - RequestID
 303 ---------------
 304
 305 This is often referred to by other names, including "Transaction ID",
 306 and one of several (pre-standardization) REST header names
 307 including \ **X-ECOMP-RequestID** and **X-ONAP-RequestID**.
 308
 309 ONAP logging uses a universally unique "**RequestID"** value in log
 310 records to track the processing of each client request across all the
 311 ONAP components involved in its processing.
 312
 313 This value:
 314
 315 -  Is logged as a \ **RequestID** MDC.
 316
 317 -  Is propagated between components in REST calls as
 318    an \ **X-TransactionID** HTTP header.
 319
 320 Receiving the \ **X-TransactionID** will vary by component according to
 321 APIs and frameworks. In general:
 322
 323 .. code-block:: java
 324
 325   import javax.ws.rs.core.HttpHeaders;
 326   // ...
 327   final HttpHeaders headers = ...;
 328   // ...
 329   String txId = headers.getRequestHeaders().getFirst("X-TransactionID");
 330   if (StringUtils.isBlank(txId)) {
 331       txId = UUID.randomUUID().toString();
 332   }
 333   MDC.put("RequestID", txID);
 334
 335 Setting the \ **X-TransactionID** likewise will vary. For example:
 336
 337 .. code-block:: java
 338
 339  final String txID = MDC.get("RequestID");
 340  HttpURLConnection cx = ...;
 341  // ...
 342  cx.setRequestProperty("X-TransactionID", txID);
 343
 344 MDC - InvocationID
 345 ------------------
 346
 347 **InvocationID** is similar to \ **RequestID**, but
 348 where \ **RequestID** correlates records relating a single, top-level
 349 invocation of ONAP as it traverses many
 350 systems, \ **InvocationID** correlates log entries relating to a single
 351 invocation of a single component. Typically this means via REST, but in
 352 certain cases an \ **InvocationID** may be allocated without a new
 353 invocation, e.g. when a request is retried.
 354
 355 **RequestID** and** InvocationID** allow an execution graph to be
 356 derived. This requires that:
 357
 358 -  The relationship between \ **RequestID** and **InvocationID** is
 359    reported.
 360
 361 -  The relationship between caller and recipient is reported for each
 362    invocation.
 363
 364 The proposed approach is that:
 365
 366 -  Callers:
 367
 368    -  Issue a new, unique \ **InvocationID** UUID for each downstream
 369       call they make.
 370
 371    -  Log the new \ **InvocationID**, indicating the intent to invoke:
 372
 373       -  With Markers \ **INVOKE**, and \ **SYNCHRONOUS** if the
 374          invocation is synchronous.
 375
 376       -  With their own \ **InvocationID** still set as an MDC.
 377
 378    -  Pass the \ **InvocationID** as an \ **X-InvocationID** REST
 379       header.
 380
 381 -  Invoked components:
 382
 383    -  Retrieve the \ **InvocationID** from REST headers upon invocation,
 384       or generate a UUID default.
 385
 386    -  Set the \ **InvocationID** MDC.
 387
 388    -  Write a log entry with the Marker \ **ENTRY**. (In EELF this will
 389       be to the AUDIT log).
 390
 391    -  Act as per Callers in all downstream requests.
 392
 393    -  Write a log entry with the Marker \ **EXIT** upon return. (In EELF
 394       this will be to the METRIC log).
 395
 396    -  Unset all MDCs on exit.
 397
 398 That seems onerous, but:
 399
 400 -  It's only a few calls.
 401
 402 -  It can be largely abstracted in the case of EELF logging.
 403
 404 **TODO: code.**
 405
 406 MDCs - the Rest
 407 ---------------
 408
 409 Other MDCs are logged in a wide range of contexts.
 410
 411 Certain MDCs and their semantics may be specific to EELF log types.
 412
 413 **TODO: cross-reference EELF output to v1 doc.**
 414
 415 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 416 | **ID**   | **MDC**               | **Description**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | **Required**   | **EELF Audit**   | **EELF Metric**   | **EELF Error**   | **EELF Debug**   |
 417 +==========+=======================+================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+================+==================+===================+==================+==================+
 418 | 1        | BeginTimestamp        | Date-time that processing activities being logged begins. The value should be represented in UTC and formatted per ISO 8601, such as œ2015-06-03T13:21:58+00:00\9d. The time should be shown with the maximum resolution available to the logging component (e.g., milliseconds, microseconds) by including the appropriate number of decimal digits. For example, when millisecond precision is available, the date-time value would be presented as, as œ2015-06-03T13:21:58.340+00:00\9d.                                                                                       | Y              |                  |                   |                  |                  |
 419 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 420 | 2        | EndTimestamp          | Date-time that processing for the request or event being logged ends. Formatting rules are the same as for the BeginTimestamp field above.                                                                                                                                                                                                                                                                                                                                                                                                                                     | Y              |                  |                   |                  |                  |
 421 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 422 |          |                       | In the case of a request that merely logs an event and has not subsequent processing, the EndTimestamp value may equal the BeginTimestamp value.                                                                                                                                                                                                                                                                                                                                                                                                                               |                |                  |                   |                  |                  |
 423 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 424 | 3        | ElapsedTime           | This field contains the elapsed time to complete processing of an API call or transaction request (e.g., processing of a message that was received). This value should be the difference between. EndTimestamp and BeginTimestamp fields and must be expressed in milliseconds.                                                                                                                                                                                                                                                                                                | Y              |                  |                   |                  |                  |
 425 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 426 | 4        | ServiceInstanceID     | This field is optional and should only be included if the information is readily available to the logging component.                                                                                                                                                                                                                                                                                                                                                                                                                                                           |                |                  |                   |                  |                  |
 427 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 428 |          |                       | | Transaction requests that create or operate on a particular instance of a service/resource can                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                |                  |                   |                  |                  |
 429 |          |                       | | identify/reference it via a unique œserviceInstanceID\9d value. This value can be used as a primary key for                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                |                  |                   |                  |                  |
 430 |          |                       | | obtaining or updating additional detailed data about that specific service instance from the inventory                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                |                  |                   |                  |                  |
 431 |          |                       | | (e.g., AAI). In other words:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |                |                  |                   |                  |                  |
 432 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 433 |          |                       | -  In the case of processing/logging a transaction request for creating a new service instance, the serviceInstanceID value is determined by either a) the MSO client and passed to MSO or b) by MSO itself upon receipt of a such a request.                                                                                                                                                                                                                                                                                                                                  |                |                  |                   |                  |                  |
 434 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 435 |          |                       | -  In other cases, the serviceInstanceID value can be used to reference a specific instance of a service as would happen in a œMACD\9d-type request.                                                                                                                                                                                                                                                                                                                                                                                                                             |                |                  |                   |                  |                  |
 436 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 437 |          |                       | -  ServiceInstanceID is associated with a requestID in log records to facilitate tracing its processing over multiple requests and for a specific service instance. Its value may be left œempty\9d in subsequent record to the 1 st record where a requestID value is associated with the serviceInstanceID value.                                                                                                                                                                                                                                                              |                |                  |                   |                  |                  |
 438 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 439 |          |                       | NOTE: AAI wont have a serviceInstanceUUID for every service instance. For example, no serviceInstanceUUID is available when the request is coming from an application that may import inventory data.                                                                                                                                                                                                                                                                                                                                                                          |                |                  |                   |                  |                  |
 440 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 441 | 5        | VirtualServerName     | Physical/virtual server name. Optional: empty if determined that its value can be added by the agent that collects the log files collecting.                                                                                                                                                                                                                                                                                                                                                                                                                                   |                |                  |                   |                  |                  |
 442 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 443 | 6        | ServiceName           | For Audit log records that capture API requests, this field contains the name of the API invoked at the component creating the record (e.g., Layer3ServiceActivateRequest).                                                                                                                                                                                                                                                                                                                                                                                                    | Y              |                  |                   |                  |                  |
 444 |          |                       |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 445 |          |                       | For Audit log records that capture processing as a result of receipt of a message, this field should contain the name of the module that processes the message.                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 446 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 447 | 7        | PartnerName           | This field contains the name of the client application user agent or user invoking the API if known.                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Y              |                  |                   |                  |                  |
 448 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 449 | 8        | StatusCode            | This field indicates the high level status of the request. It must have the value COMPLETE when the request is successful and ERROR when there is a failure.                                                                                                                                                                                                                                                                                                                                                                                                                   | Y              |                  |                   |                  |                  |
 450 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 451 | 9        | ResponseCode          | This field contains application-specific error codes. For consistency, common error categorizations should be used.                                                                                                                                                                                                                                                                                                                                                                                                                                                            |                |                  |                   |                  |                  |
 452 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 453 | 10       | ResponseDescription   | This field contains a human readable description of the \ **ResponseCode**.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |                |                  |                   |                  | 11               |
 454 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 455 | 11       | InstanceUUID          | If known, this field contains a universally unique identifier used to differentiate between multiple instances of the same (named) log writing service/application. Its value is set at instance creation time (and read by it, e.g., at start/initialization time from the environment). This value should be picked up by the component instance from its configuration file and subsequently used to enable differentiation of log records created by multiple, locally load balanced ONAP component or subcomponent instances that are otherwise identically configured.   |                |                  |                   |                  |                  |
 456 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 457 | 12       | Severity              | Optional: 0, 1, 2, 3 see \ `*Nagios* <https://en.wikipedia.org/wiki/Nagios>`__ monitoring/alerting for specifics/details.                                                                                                                                                                                                                                                                                                                                                                                                                                                      |                |                  |                   |                  |                  |
 458 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 459 | 13       | TargetEntity          | It contains the name of the ONAP component or sub-component, or external entity, at which the operation activities captured in this metrics log record is invoked.                                                                                                                                                                                                                                                                                                                                                                                                             | Y              |                  |                   |                  |                  |
 460 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 461 | 14       | TargetServiceName     | It contains the name of the API or operation activities invoked at the TargetEntity.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | Y              |                  |                   |                  |                  |
 462 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 463 | 15       | Server                | This field contains the Virtual Machine (VM) Fully Qualified Domain Name (FQDN) if the server is virtualized. Otherwise, it contains the host name of the logging component.                                                                                                                                                                                                                                                                                                                                                                                                   | Y              |                  |                   |                  |                  |
 464 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 465 | 16       | ServerIPAddress       | This field contains the logging component host servers IP address if known (e.g. Jetty containers listening IP address). Otherwise it is empty.                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 466 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 467 | 17       | ServerFQDN            | Unclear, but possibly duplicating one or both of \ **Server** and **ServerIPAddress**.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |                |                  |                   |                  |                  |
 468 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 469 | 18       | ClientIPAddress       | This field contains the requesting remote client applications IP address if known. Otherwise this field can be empty.                                                                                                                                                                                                                                                                                                                                                                                                                                                          |                |                  |                   |                  |                  |
 470 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 471 | 19       | ProcessKey            | This field can be used to capture the flow of a transaction through the system by indicating the components and operations involved in processing. If present, it can be denoted by a comma separated list of components and applications.                                                                                                                                                                                                                                                                                                                                     |                |                  |                   |                  |                  |
 472 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 473 | 20       | RemoteHost            | Unknown.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                |                  |                   |                  |                  |
 474 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 475 | 21       | AlertSeverity         | Unknown.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |                |                  |                   |                  |                  |
 476 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 477 | 22       | TargetVirtualEntity   | Unknown                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |                |                  |                   |                  |                  |
 478 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 479 | 23       | ClassName             | Defunct. Doesn't require an MDC.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                |                  |                   |                  |                  |
 480 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 481 | 24       | ThreadID              | Defunct. Doesn't require an MDC.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |                |                  |                   |                  |                  |
 482 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 483 | 25       | CustomField1          | (Defunct now that MDCs are serialized as NVPs.)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 484 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 485 | 26       | CustomField2          | (Defunct now that MDCs are serialized as NVPs.)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 486 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 487 | 27       | CustomField3          | (Defunct now that MDCs are serialized as NVPs.)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 488 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 489 | 28       | CustomField4          | (Defunct now that MDCs are serialized as NVPs.)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |                |                  |                   |                  |                  |
 490 +----------+-----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+------------------+-------------------+------------------+------------------+
 491
 492 Examples
 493 --------
 494
 495 ******
 496 SDC-BE
 497 ******
 498
 499 20170907: audit.log
 500
 501 .. code-block:: bash
 502
 503  root@ip-172-31-93-160:/dockerdata-nfs/onap/sdc/logs/SDC/SDC-BE# tail -f audit.log
 504  2017-09-07T18:04:03.679Z\|\|\|\|\|qtp1013423070-72297\|\|ASDC\|SDC-BE\|\|\|\|\|\|\|N/A\|INFO\|\|\|\|10.42.88.30\|\|o.o.s.v.r.s.VendorLicenseModelsImpl\|\|ActivityType=<audit>, Desc=< --Audit-- Create VLM. VLM Name: lm4>
 505
 506 **TODO: this is the earlier output format. Let's find an example which matches the latest line format.**
 507
 508 Markers
 509 =======
 510
 511 Markers differ from MDCs in two important ways:
 512
 513 1. They have a name, but no value. They are a tag.
 514
 515 2. Their scope is limited to logger calls which specifically reference
 516    them; they are
 517    not \ `*ThreadLocal* <https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html>`__.
 518
 519 Logging
 520 -------
 521
 522 **********
 523 Via SLF4J:
 524 **********
 525
 526 .. code-block:: java
 527
 528  import org.slf4j.Logger;
 529  import org.slf4j.LoggerFactory;
 530  import org.slf4j.Marker;
 531  import org.slf4j.MarkerFactory;
 532  // ...
 533  final Logger logger = LoggerFactory.getLogger(this.getClass());
 534  final Marker marker = MarkerFactory.getMarker("MY\_MARKER");
 535  logger.warn(marker, "This warning has a 'MY\_MARKER' annotation.");
 536
 537 EELF does not allow Markers to be set directly. See notes on
 538 the \ **InvocationID** MDC.
 539
 540 Serializing
 541 -----------
 542
 543 Marker names also need to be escaped, though they're much less likely to
 544 contain problematic characters than MDC values.
 545
 546 Escaping in Logback configuration can be achieved with:
 547
 548 .. code-block:: bash
 549
 550  %replace(%replace(%marker){'\\t','\\\\\\\\t'}){'\\n','\\\\\\\\n'}
 551
 552 Marker - ENTRY
 553 --------------
 554
 555 This should be reported as early in invocation as possible, immediately
 556 after setting the \ **RequestID** and **InvocationID** MDCs.
 557
 558 It can be automatically set by EELF, and written to the AUDIT log.
 559
 560 It must be manually set otherwise.
 561
 562 *****
 563 EELF
 564 *****
 565
 566 *****
 567 SLF4J
 568 *****
 569
 570 **SLF4J**
 571 .. code-block:: java
 572
 573  public static final Marker ENTRY = MarkerFactory.getMarker("ENTRY");
 574  // ...
 575  final Logger logger = LoggerFactory.getLogger(this.getClass());
 576  logger.debug(ENTRY, "Entering.");
 577
 578 Marker - EXIT
 579 -------------
 580
 581 This should be reported as late in invocation as possible, immediately
 582 before unsetting the \ **RequestID** and **InvocationID** MDCs.
 583
 584 It can be automatically reported by EELF, and written to the METRIC
 585 log.
 586
 587 It must be manually set otherwise.
 588
 589 *****
 590 EELF
 591 *****
 592
 593 ******
 594 SLF4J
 595 ******
 596
 597 **SLF4J**
 598
 599 .. code-block:: java
 600
 601  public static final Marker EXIT = MarkerFactory.getMarker("EXIT");
 602  // ...
 603  final Logger logger = LoggerFactory.getLogger(this.getClass());
 604  logger.debug(EXIT, "Exiting.");
 605
 606 Marker - INVOKE
 607 ---------------
 608
 609 This should be reported by the caller of another ONAP component via
 610 REST, including a newly allocated \ **InvocationID**, which will be
 611 passed to the caller.
 612
 613 **SLF4J**
 614
 615 .. code-block:: java
 616
 617  public static final Marker INVOKE = MarkerFactory.getMarker("INVOKE");
 618  // ...
 619  // Generate and report invocation ID.
 620  final String invocationID = UUID.randomUUID().toString();
 621  MDC.put(MDC_INVOCATION_ID, invocationID);
 622  try {
 623      logger.debug(INVOKE_SYNCHRONOUS, "Invoking synchronously ... ");
 624  }
 625  finally {
 626      MDC.remove(MDC_INVOCATION_ID);
 627  }
 628  // Pass invocationID as HTTP X-InvocationID header.
 629  callDownstreamSystem(invocationID, ... );
 630
 631 **TODO: EELF, without changing published APIs.**
 632
 633 Marker - SYNCHRONOUS
 634 ------------------------
 635
 636 This should accompany \ **INVOKE** when the invocation is synchronous.
 637
 638 **SLF4J**
 639
 640 .. code-block:: java
 641
 642  public static final Marker INVOKE_SYNCHRONOUS;
 643  static {
 644      INVOKE_SYNCHRONOUS = MarkerFactory.getMarker("INVOKE");
 645      INVOKE_SYNCHRONOUS.add(MarkerFactory.getMarker("SYNCHRONOUS"));
 646  }
 647  // ...
 648  // Generate and report invocation ID.
 649  final String invocationID = UUID.randomUUID().toString();
 650  MDC.put(MDC_INVOCATION_ID, invocationID);
 651  try {
 652      logger.debug(INVOKE_SYNCHRONOUS, "Invoking synchronously ... ");
 653  }
 654  finally {
 655      MDC.remove(MDC_INVOCATION_ID);
 656  }
 657  // Pass invocationID as HTTP X-InvocationID header.
 658  callDownstreamSystem(invocationID, ... );
 659
 660 **TODO: EELF, without changing published APIs.**
 661
 662 Errorcodes
 663 ==========
 664
 665 Errorcodes are reported as MDCs.
 666
 667 Exceptions should be accompanied by an errrorcode. Typically this is
 668 achieved by incorporating errorcodes into your exception hierarchy and
 669 error handling. ONAP components generally do not share this kind of
 670 code, though EELF defines a marker interface (meaning it has no
 671 methods) \ **EELFResolvableErrorEnum**.
 672
 673 A common convention is for errorcodes to have two components:
 674
 675 1. A \ **prefix**, which identifies the origin of the error.
 676
 677 2. A \ **suffix**, which identifies the kind of error.
 678
 679 Suffixes may be numeric or text. They may also be common to more than
 680 one component.
 681
 682 For example:
 683
 684 ::
 685
 686  COMPONENT\_X.STORAGE\_ERROR
 687
 688 Output Format
 689 -------------
 690
 691 Several considerations:
 692
 693 1. Logs should be human-readable (within reason).
 694
 695 2. Shipper and indexing performance and durability depends on logs that
 696    can be parsed quickly and reliably.
 697
 698 3. Consistency means fewer shipping and indexing rules are required.
 699
 700 ***********
 701 Text Output
 702 ***********
 703
 704 ONAP needs to strike a balance between human-readable and
 705 machine-readable logs. This means:
 706
 707 -  The use of tab (**\\t**) as a delimiter.
 708
 709 -  Escaping all messages, exceptions, MDC values, Markers, etc. to
 710    replace tabs in their content.
 711
 712 -  Escaping all newlines with \ **\\n** so that each entry is on one
 713    line.
 714
 715 In logback, this looks like:
 716
 717 ::
 718
 719  <property name="defaultPattern" value="%nopexception%logger
 720  %date{yyyy-MM-dd'T'HH:mm:ss.SSSXXX,UTC}
 721  %level
 722  %replace(%replace(%message){'\\t','\\\\\\\\t'}){'\\n','\\\\\\\\n'}
 723  %replace(%replace(%mdc){'\\t','\\\\\\\\t'}){'\\n','\\\\\\\\n'}
 724  %replace(%replace(%rootException){'\\t','\\\\\\\\t'}){'\\n','\\\\\\\\n'}
 725  %replace(%replace(%marker){'\\t','\\\\\\\\t'}){'\\n','\\\\\\\\n'}
 726  %thread
 727  %n"/>
 728
 729 The output of which, with MDCs, a Marker and a nested exception, with newlines added for readability looks like:
 730
 731 TODO: remove tab below
 732
 733 ::
 734
 735  org.onap.example.component1.subcomponent1.LogbackTest
 736  2017-08-06T16:09:03.594Z
 737  ERROR
 738  Here's an error, that's usually bad
 739  key1=value1, key2=value2 with space, key5=value5"with"quotes, key3=value3 with newlines, key4=value4 with tabs
 740  java.lang.RuntimeException: Here's Johnny
 741  at org.onap.example.component1.subcomponent1.LogbackTest.main(LogbackTest.java:24)
 742  Wrapped by: java.lang.RuntimeException: Little pigs, little pigs, let me come in
 743  at org.onap.example.component1.subcomponent1.LogbackTest.main(LogbackTest.java:27)
 744  AMarker1
 745  main
 746
 747 Default Logstash indexing rules understand output in this format.
 748
 749 **********
 750 XML Output
 751 **********
 752
 753 For Log4j 1.X output, since escaping is not supported, the best
 754 alternative is to emit logs in XML format.
 755
 756 There may be other instances where XML (or JSON) output may be
 757 desirable. Default indexing rules support.
 758
 759 Default Logstash indexing rules understand the XML output of \ `*Log4J's
 760 XMLLayout* <https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/xml/XMLLayout.html>`__.
 761
 762 Output Location
 763 ===============
 764
 765 Standardization of output locations makes logs easier to locate and ship
 766 for indexing.
 767
 768 Logfiles should default to beneath \ **/var/log**, and
 769 beneath \ **/var/log/ONAP** in the case of core ONAP components:
 770
 771 .. code-block:: bash
 772
 773  /var/log/ONAP/<component>[/<subcomponent>]/\*.log
 774
 775 Configuration
 776 =============
 777
 778 Logging providers should be configured by file. Files should be at a
 779 predictable, static location, so that they can be written by deployment
 780 automation. Ideally this should be under \ **/etc/ONAP**, but compliance
 781 is low.
 782
 783 Locations
 784 =========
 785
 786 All logger provider configuration document locations namespaced by
 787 component and (if applicable) subcomponent by default:
 788
 789 .. code-block:: bash
 790
 791  /etc/onap/<component>[/<subcomponent>]/<provider>.xml
 792
 793 Where \ **<provider>.xml**, will typically be one of:
 794
 795 1. logback.xml
 796
 797 2. log4j.xml
 798
 799 3. log4j.properties
 800
 801 Reconfiguration
 802 ===============
 803
 804 Logger providers should reconfigure themselves automatically when their
 805 configuration file is rewritten. All major providers should support
 806 this.
 807
 808 The default interval is 10s.
 809
 810 Overrides
 811 =========
 812
 813 The location of the configuration file MAY be overrideable, for example
 814 by an environment variable, but this is left for individual components
 815 to decide.
 816
 817 Archetypes
 818 ============
 819
 820 Configuration archetypes can be found in the ONAP codebase. Choose
 821 according to your provider, and whether you're logging via EELF. Efforts
 822 to standardize them are underway, so the ones you should be looking for
 823 are where pipe (\|) is used as a separator. (Previously it was "\|").
 824
 825 Retention
 826 =========
 827
 828 Logfiles are often large. Logging providers allow retention policies to
 829 be configured.
 830
 831 Retention has to balance:
 832
 833 -  The need to index logs before they're removed.
 834
 835 -  The need to retain logs for other (including regulatory) purposes.
 836
 837 Defaults are subject to change. Currently they are:
 838
 839 1. Files <= 50MB before rollover.
 840
 841 2. Files retain for 30 days.
 842
 843 3. Total files capped at 10GB.
 844
 845 In Logback configuration XML:
 846
 847 .. code-block:: xml
 848
 849  <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
 850      <file>${outputDirectory}/${outputFilename}.log</file>
 851      <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
 852          <fileNamePattern>${outputDirectory}/${outputFilename}.%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>
 853          <maxFileSize>50MB</maxFileSize>
 854          <maxHistory>30</maxHistory>
 855          <totalSizeCap>10GB</totalSizeCap>
 856      </rollingPolicy>
 857      <encoder>
 858          <!-- ... -->
 859      </encoder>
 860  </appender>
 861
 862 Types of EELF Logs
 863 ==================
 864
 865 EELF guidelines stipulate that an application should output log records
 866 to four separate files:
 867
 868 1. audit
 869
 870 2. metric
 871
 872 3. error
 873
 874 4. debug
 875
 876 This applies only to EELF logging. Components which log directly to a
 877 provider may choose to emit the same set of logs, but most do not.
 878
 879 Audit Log
 880 ---------
 881
 882 An audit log is required for EELF-enabled components, and provides a
 883 summary view of the processing of a (e.g., transaction) request within
 884 an application. It captures activity requests that are received by an
 885 ONAP component, and includes such information as the time the activity
 886 is initiated, then it finishes, and the API that is invoked at the
 887 component.
 888
 889 Audit log records are intended to capture the high level view of
 890 activity within an ONAP component. Specifically, an API request handled
 891 by an ONAP component is reflected in a single Audit log record that
 892 captures the time the request was received, the time that processing was
 893 completed, as well as other information about the API request (e.g., API
 894 name, on whose behalf it was invoked, etc).
 895
 896 Metric Log
 897 ----------
 898
 899 A metric log is required for EELF-enabled components, and provides a
 900 more detailed view into the processing of a transaction within an
 901 application. It captures the beginning and ending of activities needed
 902 to complete it. These can include calls to or interactions with other
 903 ONAP or non-ONAP entities.
 904
 905 Suboperations invoked as part of the processing of the API request are
 906 logged in the Metrics log. For example, when a call is made to another
 907 ONAP component or external (i.e., non-ONAP) entity, a Metrics log record
 908 captures that call. In such a case, the Metrics log record indicates
 909 (among other things) the time the call is made, when it returns, the
 910 entity that is called, and the API invoked on that entity. The Metrics
 911 log record contain the same RequestID as the Audit log record so the two
 912 can be correlated.
 913
 914 Note that a single request may result in multiple Audit log records at
 915 an ONAP component and may result in multiple Metrics log records
 916 generated by the component when multiple suboperations are required to
 917 satisfy the API request captured in the Audit log record.
 918
 919 Error Log
 920 ---------
 921
 922 An error log is required for EELF-enabled components, and is intended to
 923 capture info, warn, error and fatal conditions sensed (œexception
 924 handled\9d) by the software components.
 925
 926 Debug Log
 927 ---------
 928
 929 A debug log is optional for EELF-enabled components, and is intended to
 930 capture whatever data may be needed to debug and correct abnormal
 931 conditions of the application.
 932
 933 Engine.out
 934 ==========
 935
 936 Console logging may also be present, and is intended to capture
 937 system/infrastructure records. That is stdout and stderr assigned to a
 938 single œengine.out file in a directory configurable (e.g. as an
 939 environment/shell variable) by operations personnel.
 940
 941 New ONAP Component Checklist
 942 ============================
 943
 944 By following a few simple rules:
 945
 946 -  Your component's output will be indexed automatically.
 947
 948 -  Analytics will be able to trace invocation through your component.
 949
 950 Obligations fall into two categories:
 951
 952 1. Conventions regarding configuration, line format and output.
 953
 954 2. Ensuring the propagation of contextual information.
 955
 956 You must:
 957
 958 1. Choose a Logging provider and/or EELF. Decisions, decisions.
 959
 960 2. Create a configuration file based on an existing
 961    archetype. See \ `*Configuration* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-Configuration>`__.
 962
 963 3. Read your configuration file when your components initialize logging.
 964
 965 4. Write logs to a standard location so that they can be shipped by
 966    Filebeat for indexing. See \ `*Output
 967    Location* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-OutputLocation>`__.
 968
 969 5. Report transaction state:
 970
 971    a. Retrieve, default and propagate \ **RequestID**. See \ `*MDC -
 972       RequestID* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-MDC-RequestID>`__.
 973
 974    b. At each invocation of one ONAP component by another:
 975
 976       i.   Initialize and propagate \ **InvocationID**. See \ `*MDC -
 977            Invocation
 978            ID* <https://wiki.onap.org/display/DW/ONAP+Application+Logging+Guidelines+v1.1#ONAPApplicationLoggingGuidelinesv1.1-MDC-InvocationID>`__.
 979
 980       ii.  Report \ **INVOKE** and **SYNCHRONOUS** markers in caller.
 981
 982       iii. Report \ **ENTRY** and **EXIT** markers in recipient.
 983
 984 6. Write useful logs!
 985
 986  They are unordered.