5 A filter examine an event and decides if it matches or doesn't. Filters
6 are mainly used in rules to decide if the processing entries should be
7 executed on the given event. They're also used for settings, and systems
8 like the Graph Correlator re-use Highland Park's filter mechanism to
9 specify which alarms fit in a correlation. Some publishers may produce
10 topics with a lot of volume and a subscriber may want only a portion of
11 those messages. The subscriber can certainly filter messages after
12 receiving them, but it may be substantially more efficient to ask the
13 API server to filter the messages before sending them to the
14 subscriber.The standard library includes a number of simple filters. The
15 Equals filter, for example, compares a field's value with some other
16 value and returns true if the values match. The standard library also
17 includes filter classes called And, Or, and Not, so you can compose more
18 complex filters. For example, written in the standard JSON config
21 +-----------------------------------------+
26 | { "class":"Equals", "foo":"abc" }, |
27 | { "class":"Assigned", "field":"bar" } |
30 +-----------------------------------------+
32 - This filter matches events in which the field "foo" has value "abc"
33 and the field "bar" is assigned to some value (as opposed to not being
34 present on the event). Filters are used by the consumers to filter out
35 data and consume only specific set of data which matches the
36 conditions mentioned in filter. Filters can be passed as a query
37 parameter by the consumer in consume request as mentioned below:
38 **http://localhost:8080/DMaaP/dmaaprest/events/DMaaP/consumergroup/mHOeNFY4XiWx4CBa?filter=\\{"class":"Equals",
39 "field":"email", "value":"test@abc.com" }**
40 - Filters can be applied only on data in JSON format i.e. if applied,
41 filters will automatically ignore any non-json data. While consuming,
42 request CONTENT\_TYPE is not relevant to filter.
44 All the supported filter can be found below.
49 - DMaaP Message Router supports all the filters which were supported by
50 DMaaP Message Router and are mentioned below:-
54 - Create a set of filters. This filter matches when all of them matches.
56 +-------------+--------------------+------------+---------------------+
57 | **Field** | **Description** | **Type** | **Notes** |
58 +=============+====================+============+=====================+
59 | filters | Combined Filters | LIST | A list of filters |
60 +-------------+--------------------+------------+---------------------+
62 +-------------+--------------------+------------+---------------------+
65 - Choose a field from the event to check for assignment. This filter
66 matches when the field is assigned.
68 +-------------------+--------------------------------------------------------+------------+-----------------+
69 | **Field** | **Description** | **Type** | **Notes** |
70 +===================+========================================================+============+=================+
71 | field | The field to check for on the event. | STRING | A field name |
72 +-------------------+--------------------------------------------------------+------------+-----------------+
73 | emptyIsAssigned | If true, an empty value is considered an assignment. | BOOLEAN | True or False |
74 +-------------------+--------------------------------------------------------+------------+-----------------+
77 - Check if a search string contains another string.
79 +-------------+---------------------------------------------------+------------+--------------+
80 | **Field** | **Description** | **Type** | **Notes** |
81 +=============+===================================================+============+==============+
82 | String | The value to search. Supports ${} notation. | STRING | Any string |
83 +-------------+---------------------------------------------------+------------+--------------+
84 | Value | The value to search for. Supports ${} notation. | STRING | Any string |
85 +-------------+---------------------------------------------------+------------+--------------+
87 +-------------+---------------------------------------------------+------------+--------------+
90 - Check if a search string ends with another string.
92 +-------------+---------------------------------------------------+------------+--------------+
93 | **Field** | **Description** | **Type** | **Notes** |
94 +=============+===================================================+============+==============+
95 | string | The value to search. Supports ${} notation. | STRING | Any string |
96 +-------------+---------------------------------------------------+------------+--------------+
97 | value | The value to search for. Supports ${} notation. | STRING | Any string |
98 +-------------+---------------------------------------------------+------------+--------------+
102 - Choose a field from the event and a value to check for equality.
104 +-------------+----------------------------------------------+------------+--------------+
105 | **Field** | **Description** | **Type** | **Notes** |
106 +=============+==============================================+============+==============+
107 | field | The field to check. Supports ${} notation. | STRING | Any string |
108 +-------------+----------------------------------------------+------------+--------------+
109 | value | The value to match. Supports ${} notation. | STRING | Any string |
110 +-------------+----------------------------------------------+------------+--------------+
112 - **FlatironObjectExists**
114 - Matches when the given object exists in the given Flatiron instance.
116 +-------------+---------------------------------------------+------------+--------------+
117 | **Field** | **Description** | **Type** | **Notes** |
118 +=============+=============================================+============+==============+
119 | oid | The OID of the object to look for. | STRING | Any string |
120 +-------------+---------------------------------------------+------------+--------------+
121 | flatiron | The name of the Flatiron client instance. | STRING | Any string |
122 +-------------+---------------------------------------------+------------+--------------+
125 - Choose a field to test. This filter matches if the expression is
128 +-------------+---------------------------------------------+------------+--------------+
129 | **Field** | **Description** | **Type** | **Notes** |
130 +=============+=============================================+============+==============+
131 | field | The field to test. Supports ${} notation. | STRING | Any string |
132 +-------------+---------------------------------------------+------------+--------------+
135 - Choose a field to test. This filter matches if the expression is
138 +-------------+---------------------------------------------+------------+--------------+
139 | **Field** | **Description** | **Type** | **Notes** |
140 +=============+=============================================+============+==============+
141 | field | The field to test. Supports ${} notation. | STRING | Any string |
142 +-------------+---------------------------------------------+------------+--------------+
145 - Choose a field from the event and a value for logical math conditions.
147 +-------------+-------------------------------------------------+------------+-----------------------------------+
148 | **Field** | **Description** | **Type** | **Notes** |
149 +=============+=================================================+============+===================================+
150 | Field | The field to check. Supports ${} notation. | STRING | Any string |
151 +-------------+-------------------------------------------------+------------+-----------------------------------+
152 | Value | The value to consider. Supports ${} notation. | STRING | Any string |
153 +-------------+-------------------------------------------------+------------+-----------------------------------+
154 | operator | The operation. | STRING | One of { "<=", ">=", ">", "<" } |
155 +-------------+-------------------------------------------------+------------+-----------------------------------+
157 +-------------+-------------------------------------------------+------------+-----------------------------------+
160 - Don't match any alarms.
162 - Negate the configured filter.
164 +-------------+-------------------------+------------+-------------+
165 | **Field** | **Description** | **Type** | **Notes** |
166 +=============+=========================+============+=============+
167 | filter | The filter to negate. | FILTER | A filter |
168 +-------------+-------------------------+------------+-------------+
171 - Choose a field from the event and a value to check for inequality.
173 +-------------+----------------------------------------------+------------+--------------+
174 | **Field** | **Description** | **Type** | **Notes** |
175 +=============+==============================================+============+==============+
176 | field | The field to check. Supports ${} notation. | STRING | Any string |
177 +-------------+----------------------------------------------+------------+--------------+
178 | value | The value to match. Supports ${} notation. | STRING | Any string |
179 +-------------+----------------------------------------------+------------+--------------+
182 - Match when the specified field does not have a value from the given list.
184 +-------------+---------------------------------------------+------------+---------------------+
185 | **Field** | **Description** | **Type** | **Notes** |
186 +=============+=============================================+============+=====================+
187 | field | The field to test. Supports ${} notation. | STRING | Any string |
188 +-------------+---------------------------------------------+------------+---------------------+
189 | values | The matching values. | LIST | A list of strings |
190 +-------------+---------------------------------------------+------------+---------------------+
193 - Match when the specified field has a value from the given list.
195 +-------------+---------------------------------------------+------------+---------------------+
196 | **Field** | **Description** | **Type** | **Notes** |
197 +=============+=============================================+============+=====================+
198 | field | The field to test. Supports ${} notation. | STRING | Any string |
199 +-------------+---------------------------------------------+------------+---------------------+
200 | values | The matching values. | LIST | A list of strings |
201 +-------------+---------------------------------------------+------------+---------------------+
204 - Create a set of filters. This filter matches when any one of them
207 +-------------+--------------------+------------+---------------------+
208 | **Field** | **Description** | **Type** | **Notes** |
209 +=============+====================+============+=====================+
210 | filters | Combined Filters | LIST | A list of filters |
211 +-------------+--------------------+------------+---------------------+
214 - Choose a field from the event to match against the regular expression
217 +-------------+---------------------------------------------------------+------------+--------------+
218 | **Field** | **Description** | **Type** | **Notes** |
219 +=============+=========================================================+============+==============+
220 | field | The text to check for a match. Supports ${} notation. | STRING | Any string |
221 +-------------+---------------------------------------------------------+------------+--------------+
222 | value | The regular expression (pattern) to match. | STRING | Any string |
223 +-------------+---------------------------------------------------------+------------+--------------+
226 - Check if a search string starts with another string.
228 +-------------+---------------------------------------------------+------------+--------------+
229 | **Field** | **Description** | **Type** | **Notes** |
230 +=============+===================================================+============+==============+
231 | string | The value to search. Supports ${} notation. | STRING | Any string |
232 +-------------+---------------------------------------------------+------------+--------------+
233 | Value | The value to search for. Supports ${} notation. | STRING | Any string |
234 +-------------+---------------------------------------------------+------------+--------------+
237 - Choose a field from the event to check for assignment. This filter
238 matches when the field is not assigned.
240 +-------------------+--------------------------------------------------------+------------+-----------------+
241 | **Field** | **Description** | **Type** | **Notes** |
242 +===================+========================================================+============+=================+
243 | field | The field to check for on the event. | STRING | A field name |
244 +-------------------+--------------------------------------------------------+------------+-----------------+
245 | emptyIsAssigned | If true, an empty value is considered an assignment. | BOOLEAN | True or False |
246 +-------------------+--------------------------------------------------------+------------+-----------------+
248 - **WithinSecondsFrom**
249 - This filter matches when the specified epoch time value is within the
250 given number of seconds from the baseline time value. Both time values
251 are assumed to be in seconds. If a value is in milliseconds, set
252 baselineTimeInMillis and/or eventTimeInMillis to true.
254 +------------------------+------------------------------------------------------------+------------+-----------------+
255 | **Field** | **Description** | **Type** | **Notes** |
256 +========================+============================================================+============+=================+
257 | field | The time value to test. Supports ${} | STRING | A field name |
258 +------------------------+------------------------------------------------------------+------------+-----------------+
259 | eventTimeInMillis | Whether to convert the event value from milliseconds. | BOOLEAN | True or False |
260 +------------------------+------------------------------------------------------------+------------+-----------------+
261 | seconds | The number of seconds. | NUMBER | A number |
262 +------------------------+------------------------------------------------------------+------------+-----------------+
263 | baselineTimeInMillis | Whether to convert the baseline value from milliseconds. | BOOLEAN | True or False |
264 +------------------------+------------------------------------------------------------+------------+-----------------+
265 | baseline | The baseline time value. Supports ${}. | STRING | Any string |
266 +------------------------+------------------------------------------------------------+------------+-----------------+
268 - **WithinTimeFromNow**
269 - This filter matches when the named field has an epoch time value
270 within the given number of seconds from the current time. The event's
271 time value is assumed to be in seconds. If it's in milliseconds, set
272 eventTimeInMillis to true.
274 +---------------------+---------------------------------------------------------+------------+-----------------+
275 | **Field** | **Description** | **Type** | **Notes** |
276 +=====================+=========================================================+============+=================+
277 | field | The field to check on the event. | STRING | A field name |
278 +---------------------+---------------------------------------------------------+------------+-----------------+
279 | eventTimeInMillis | Whether to convert the event value from milliseconds. | BOOLEAN | True or False |
280 +---------------------+---------------------------------------------------------+------------+-----------------+
281 | seconds | The number of seconds. | NUMBER | A number |
282 +---------------------+---------------------------------------------------------+------------+-----------------+
286 - Limit is the integer value and DMaaP Message Router will consumes
287 only that set of message which are specified in limit.
291 Suppose if we set limit=2, then only 2 sets of data will be consumed.
292 *Get \ **<<topicName>>/group/2?limit=4***
294 **No of data available** = 4
296 i.e. limit>no of data
297 In this scenario all 4 sets of data will be consumed.
299 - If limit is not passed with the url then by default limit is set to
304 i.e. 4096 sets of data will be consumed.
305 **Timeout and Long Poll:**
307 - Timeout is the integer value which will be treated by DMaaP Message
308 Router as time in millisecond.
314 +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
315 | `***http://localhost/DMaaP/dmaaprest/events/<<topicName>>/group/2?timeout=20000*** <http://localhost/DMaaP/dmaaprest/events/%3c%3ctopicName%3e%3e/group/2?timeout=20000>`__ |
316 +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
318 - If there is no data available to be consumed, then DMaaP Message
319 Router will poll for the particular period of time specified in
320 timeout this mechanism is known as Long Poll.
322 - If timeout is not passed with url then DMaaP Message Router will set
323 the value of timeout =10000
325 - i.e. if no set of data are available then DMaaP Message Router will
330 - Meta is a Boolean value.
332 - DMaaP Message Router reads the value of meta from
333 MRConfiguration.properties file at the time of startup.
335 - If the value of meta is not null and if value of meta is one of these
336 values true, yes, on, 1, y, checked then DMaaP Message Router will
337 take meta flag as true, else it will be false.
339 - If meta is set to true then consumer will get the value of message
340 offset along with message.
344 .. |image1| image:: images/image1.png
348 - Pretty is a Boolean value.
350 - DMaaP Message Router reads the value of pretty from
351 MRConfiguration.properties file at the time of startup.
353 - If the value of pretty is not null and if value of pretty is one of
354 these values true, yes, on, 1, y, checked then DMaaP Message Router
355 will take pretty flag as true, else it will be false.
357 - If pretty is set to true then different sets of messages will be
358 printed in next line separated by comma (,).
362 .. |image2| image:: images/image2.png
366 - A filter examine an event and decides if it matches or doesn't.
367 - Filters are mainly used in rules to decide if the processing entries
368 should be executed on the given event. They're also used for settings,
369 and systems like the Graph Correlator re-use Highland Park's filter
370 mechanism to specify which alarms fit in a correlation.
371 - The standard library includes a number of simple filters. The Equals
372 filter, for example, compares a field's value with some other value
373 and returns true if the values match.
374 - The standard library also includes filter classes called And, Or, and
375 Not, so you can compose more complex filters. For example, written in
376 the standard JSON config format:
378 +-----------------------------------------+
383 | { "class":"Equals", "foo":"abc" }, |
384 | { "class":"Assigned", "field":"bar" } |
387 +-----------------------------------------+
389 - This filter matches events in which the field "foo" has value "abc" and
390 the field "bar" is assigned to some value (as opposed to not being
391 present on the event).
392 - Filters are used by the consumers to filter out data and consume only
393 specific set of data which matches the conditions mentioned in filter.
394 - Filters can be passed as a query parameter by the consumer in consume
395 request as mentioned below:
396 - **http://localhost:8080/DMaaP/dmaaprest/events/DMaaP/consumergroup/mHOeNFY4XiWx4CBa?filter=\\{"class":"Equals","field":"email", "value":"`test@abc.com <mailto:test@att.com>`__" }**
397 - Filters can be applied only on data in JSON format i.e. if applied, filters will automatically ignore any non-json data.
398 - While consuming, request CONTENT\_TYPE is not relevant to filter.
400 The MR API allows a subscriber pass a Highland Park filter as part of
401 the GET request. This will filter the stream of messages sent back to
402 the subscriber, but for this to work, there are some requirements:
404 - The message payload must be JSON
406 - Only a filter built from Highland Park's Standard Library may be
407 used. (The Cambria API server doesn't have access to plugged in
410 - The filter must be encoded properly in the URL path.
412 Server-side filtering can also be setup in the Java client as illustrated below
414 **Filtering Consumer**
416 You can also provide a Highland Park filter to your consumer instance,
417 and this filter is passed on to the server in the GET request. One way
418 to create the filter is programmatically. In your code, instantiate a
419 filter from the Highland Park Standard Library Then create a String
420 representation of the filter using the FilterIo.write utility. This
421 String can then be passed to the Cambria client instance for use on the
424 Remember, only Highland Park standard library filter components can be
425 used -- no plug-ins are available in the Cambria server context.
429 package org.onap.sa.highlandPark.integration;
431 import java.io.IOException;
433 import java.util.UUID;
435 import org.onap.nsa.cambria.client.CambriaClientFactory;
437 import org.onap.nsa.cambria.client.CambriaConsumer;
439 import org.onap.sa.highlandPark.processor.HpEvent;
441 import org.onap.sa.highlandPark.stdlib.filters.FilterIo;
443 import org.onap.sa.highlandPark.stdlib.filters.OneOf;
445 public class ExampleFilteringConsumer
449 public static void main ( String[] args ) throws IOException,
454 // Cambria clients take a set of 1 or more servers to use in round-robin
457 // If a server becomes unreachable, another in the group is used.
460 serverGroup="ueb01hydc.it.att.com,ueb02hydc.it.att.com,ueb03hydc.it.att.com";
464 final String topic = "TEST-TOPIC";
466 // Cambria clients can run in a cooperative group to handle high-volume
469 // Here, we create a random group name, which means this client is not
472 final String consumerGroup = UUID.randomUUID ().toString ();
474 final String consumerId = "0";
476 // Cambria clients can sit in a tight loop on the client side, using a
479 // to wait for messages, and a limit to tell the server the most to send
482 final int longPollMs = 30\*1000;
484 final int limit = -1;
486 // The Cambria server can filter the returned message stream using
489 // Highland Park system. Here, we create a simple filter to test for the
492 // value being one of the Mobility power alarms.
494 final OneOf oneOf = new OneOf ( "AlarmId", kPowerAlarms );
496 // create the consumer
498 final CambriaConsumer cc = CambriaClientFactory.createConsumer (
501 consumerGroup, consumerId, longPollMs, limit, FilterIo.write ( oneOf )
504 // now loop reading messages. Note that cc.fetch() will wait in its HTTP
507 // method for up to 30 seconds (longPollMs) when nothing's available at
516 for ( String msg : cc.fetch () )
520 System.out.println ( "" + (++count) + ": " + msg );
528 private static final String[] kPowerAlarms =
532 "HUB COMMERCIAL POWER FAIL\_FWD",
534 "HUB COMMERCIAL POWER FAIL",
536 "RBS COMMERCIAL POWER FAIL - Fixed\_FWD",
538 "RBS COMMERCIAL POWER FAIL\_FWD",
540 "RBS COMMERCIAL POWER FAIL - No Generator\_FWD",
542 "RBS COMMERCIAL POWER FAIL - Portable\_FWD",
544 "RBS COMMERCIAL POWER FAIL - Shared\_FWD",
546 "RBS COMMERCIAL POWER FAIL - Yes\_FWD",
548 "RBS COMMERCIAL POWER FAIL - YES\_FWD",
550 "RBS COMMERCIAL POWER FAIL - Fixed",
552 "RBS COMMERCIAL POWER FAIL - No Generator",
554 "RBS COMMERCIAL POWER FAIL - Portable",
556 "RBS COMMERCIAL POWER FAIL - Shared",
558 "RBS COMMERCIAL POWER FAIL - YES",
560 "RBS COMMERCIAL POWER FAIL - Yes",
562 "RBS COMMERCIAL POWER FAIL",
564 "HUB COMMERCIAL POWER FAIL - Fixed",
566 "HUB COMMERCIAL POWER FAIL - No Generator",
568 "HUB COMMERCIAL POWER FAIL - Portable",
570 "HUB COMMERCIAL POWER FAIL - Shared",
572 "HUB COMMERCIAL POWER FAIL - Fixed\_FWD",
574 "HUB COMMERCIAL POWER FAIL - No Generator\_FWD",
576 "HUB COMMERCIAL POWER FAIL - Portable\_FWD",
578 "HUB COMMERCIAL POWER FAIL - Shared\_FWD",
588 MR server-side filtering allows a consumer to filter the stream of
589 messages returned from the GET call. The following link provide details
590 of building some of the filter to illustrate Filter Builder. It is not
591 meant to cover and provide examples of every filter