1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. http://creativecommons.org/licenses/by/4.0
11 According to **ONAP** logging policy, **HV-VES** logs contain all required markers as well as service and client specific Mapped Diagnostic Context (later referred as MDC)
13 Default console log pattern:
17 | %date{"yyyy-MM-dd'T'HH:mm:ss.SSSXXX", UTC}\t| %thread\t| %highlight(%-5level)\t| %msg\t| %marker\t| %rootException\t| %mdc\t| %thread
19 A sample, fully qualified message implementing this pattern:
23 | 2018-12-18T13:12:44.369Z | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | DEBUG | Client connection request received | ENTRY | | RequestID=d7762b18-854c-4b8c-84aa-95762c6f8e62, InstanceID=9b9799ca-33a5-4f61-ba33-5c7bf7e72d07, InvocationID=b13d34ba-e1cd-4816-acda-706415308107, PartnerName=C=PL, ST=DL, L=Wroclaw, O=Nokia, OU=MANO, CN=dcaegen2-hvves-client, StatusCode=INPROGRESS, ClientIPAddress=192.168.0.9, ServerFQDN=a4ca8f96c7e5 | reactor-tcp-nio-2
26 For simplicity, all log messages in this section are shortened to contain only:
31 Error and warning logs contain also:
35 **Do not rely on exact log messages or their presence, as they are often subject to change.**
37 Deployment/Installation errors
38 ------------------------------
40 **Missing required parameters**
44 Unexpected error when parsing command line arguments
45 usage: java org.onap.dcae.collectors.veshv.main.MainKt
46 Required parameters: s, p, c
47 -c,--config-url <arg> URL of ves configuration on consul
48 -d,--first-request-delay <arg> Delay of first request to consul in
50 -H,--health-check-api-port <arg> Health check rest api listen port
51 -I,--request-interval <arg> Interval of consul configuration
53 -i,--idle-timeout-sec <arg> Idle timeout for remote hosts. After
54 given time without any data exchange
56 connection might be closed.
57 -k,--key-store <arg> Key store in PKCS12 format
58 -kp,--key-store-password <arg> Key store password
59 -l,--ssl-disable Disable SSL encryption
60 -m,--max-payload-size <arg> Maximum supported payload size in
62 -p,--listen-port <arg> Listen port
63 -s,--kafka-bootstrap-servers <arg> Comma-separated Kafka bootstrap
64 servers in <host>:<port> format
65 -t,--trust-store <arg> File with trusted certificate bundle
67 -tp,--trust-store-password <arg> Trust store password
68 -u,--dummy If present will start in dummy mode
69 (dummy external services)
70 All parameters can be specified as environment variables using
71 upper-snake-case full name with prefix `VESHV_`.
74 This log message is printed when you do not specify the required parameters (via command line, or in environment variables).
75 As described in the above log message, there are a few required parameters:
76 **listen port**, **config url**, **kafka-bootstrap-servers** and either **trust store password** and **key store password** if you want to use SSL, or only **ssl disable** if not.
78 To get rid of this error, specify the required parameters. For example:
84 <hv-ves run command> --listen-port 6061 --config-url http://consul-url/key-path --kafka-bootstrap-servers message-router-kafka:9092 --key-store-password password --trust-store-password password
86 - By defining environment variables:
90 export VESHV_LISTEN_PORT=6061
91 export VESHV_CONFIG_URL=http://consul-url/key-path
92 export VESHV_KAFKA_BOOTSTRAP_SERVERS=message-router-kafka:9092
93 export VESHV_KEY_STORE_PASSWORD=password
94 export VESHV_TRUST_STORE_PASSWORD=password
98 Command line arguments have priority over environment variables. If you configure a parameter in both ways, **HV-VES** applies the one from the command line.
100 For more information about **HV-VES** configuration parameters, see :ref:`deployment`.
105 **Consul service not responding**
109 | ap.dcae.collectors.veshv.impl.adapters.HttpAdapter | ERROR | Failed to get resource on path: http://invalid-host:8500/v1/kv/veshv-config?raw=true (consul-server1: Temporary failure in name resolution)
110 | ap.dcae.collectors.veshv.impl.adapters.HttpAdapter | DEBUG | Nested exception: | | java.net.UnknownHostException: consul-server1: Temporary failure in name resolution
111 at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
112 at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
113 at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
114 at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
115 at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
116 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
117 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
118 at java.base/java.net.InetAddress.getByName(InetAddress.java:1248)
119 at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146)
120 at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143)
121 at java.base/java.security.AccessController.doPrivileged(Native Method)
122 at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
123 at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
124 at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
125 at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
126 at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
127 at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)
128 at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108)
129 at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208)
130 at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49)
131 at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188)
132 at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174)
133 at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
134 at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
135 at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
136 at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
137 at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
138 at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978)
139 at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512)
140 at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423)
141 at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482)
142 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
143 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
144 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:315)
145 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
146 at java.base/java.lang.Thread.run(Thread.java:834)
147 | rs.veshv.impl.adapters.ConsulConfigurationProvider | WARN | Could not load fresh configuration | java.net.UnknownHostException: consul-server1: Temporary failure in name resolution
148 at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
149 at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
150 at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
151 at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
152 at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
153 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
154 at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
155 at java.base/java.net.InetAddress.getByName(InetAddress.java:1248)
156 at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146)
157 at io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143)
158 at java.base/java.security.AccessController.doPrivileged(Native Method)
159 at io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143)
160 at io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43)
161 at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63)
162 at io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55)
163 at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57)
164 at io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32)
165 at io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108)
166 at io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208)
167 at io.netty.bootstrap.Bootstrap.access$000(Bootstrap.java:49)
168 at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:188)
169 at io.netty.bootstrap.Bootstrap$1.operationComplete(Bootstrap.java:174)
170 at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
171 at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485)
172 at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
173 at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:103)
174 at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
175 at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetSuccess(AbstractChannel.java:978)
176 at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:512)
177 at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423)
178 at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482)
179 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
180 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
181 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:315)
182 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
183 at java.base/java.lang.Thread.run(Thread.java:834)
184 | ors.veshv.healthcheck.factory.HealthCheckApiServer | DEBUG | HV-VES status: OUT_OF_SERVICE, Consul configuration not available. Retrying.
188 The above three logs indicate that **HV-VES** cannot connect to the Consul service under url given in ``--consul-url`` parameter.
189 Make sure Consul is up and running and the **ip + port** combination is correct.
193 **Missing configuration on Consul**
197 | ap.dcae.collectors.veshv.impl.adapters.HttpAdapter | ERROR | Failed to get resource on path: http://consul-server:8500/v1/kv/invalid-resource?raw=true (http://consul-server:8500/v1/kv/invalid-resource?raw=true 404 Not Found)
198 | ap.dcae.collectors.veshv.impl.adapters.HttpAdapter | DEBUG | Nested exception: | java.lang.IllegalStateException: http://consul-server:8500/v1/kv/invalid-resource?raw=true 404 Not Found
199 at org.onap.dcae.collectors.veshv.impl.adapters.HttpAdapter$get$2.apply(HttpAdapter.kt:46)
200 at org.onap.dcae.collectors.veshv.impl.adapters.HttpAdapter$get$2.apply(HttpAdapter.kt:34)
201 at reactor.netty.http.client.HttpClientFinalizer.lambda$responseSingle$7(HttpClientFinalizer.java:95)
202 at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)
203 at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81)
204 at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:147)
205 at reactor.netty.http.client.HttpClientConnect$HttpObserver.onStateChange(HttpClientConnect.java:383)
206 at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:501)
207 at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:443)
208 at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:494)
209 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
210 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
211 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
212 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
213 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
214 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
215 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
216 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
217 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
218 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
219 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
220 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
221 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
222 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
223 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
224 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)
225 at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410)
226 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310)
227 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
228 at java.base/java.lang.Thread.run(Thread.java:834)
229 | rs.veshv.impl.adapters.ConsulConfigurationProvider | WARN | Could not load fresh configuration | java.lang.IllegalStateException: http://consul-server:8500/v1/kv/invalid-resource?raw=true 404 Not Found
230 at org.onap.dcae.collectors.veshv.impl.adapters.HttpAdapter$get$2.apply(HttpAdapter.kt:46)
231 at org.onap.dcae.collectors.veshv.impl.adapters.HttpAdapter$get$2.apply(HttpAdapter.kt:34)
232 at reactor.netty.http.client.HttpClientFinalizer.lambda$responseSingle$7(HttpClientFinalizer.java:95)
233 at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:118)
234 at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.onNext(FluxRetryPredicate.java:81)
235 at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:147)
236 at reactor.netty.http.client.HttpClientConnect$HttpObserver.onStateChange(HttpClientConnect.java:383)
237 at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:501)
238 at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:443)
239 at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:494)
240 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
241 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
242 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
243 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
244 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
245 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
246 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
247 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
248 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
249 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
250 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
251 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
252 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
253 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
254 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
255 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)
256 at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410)
257 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310)
258 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
259 at java.base/java.lang.Thread.run(Thread.java:834)
260 | ors.veshv.healthcheck.factory.HealthCheckApiServer | DEBUG | HV-VES status: OUT_OF_SERVICE, Consul configuration not available. Retrying.
263 **HV-VES** logs this information when connected to Consul, but cannot find any JSON configuration under given key which in this case is **invalid-resource**.
264 For more information, see :ref:`run_time_configuration`
268 **Invalid configuration format**
272 | rs.veshv.impl.adapters.ConsulConfigurationProvider | INFO | Obtained new configuration from consul:
273 { "invalidKey": "value" }
274 | 2018-12-20T15:38:14.543Z | rs.veshv.impl.adapters.ConsulConfigurationProvider | WARN | Could not load fresh configuration | org.onap.dcae.collectors.veshv.impl.adapters.ParsingException: Failed to parse consul configuration
275 at org.onap.dcae.collectors.veshv.impl.adapters.ConsulConfigurationProvider.createCollectorConfiguration(ConsulConfigurationProvider.kt:125)
276 at org.onap.dcae.collectors.veshv.impl.adapters.ConsulConfigurationProvider.access$createCollectorConfiguration(ConsulConfigurationProvider.kt:48)
277 at org.onap.dcae.collectors.veshv.impl.adapters.ConsulConfigurationProvider$invoke$4.invoke(ConsulConfigurationProvider.kt:80)
278 at org.onap.dcae.collectors.veshv.impl.adapters.ConsulConfigurationProvider$invoke$4.invoke(ConsulConfigurationProvider.kt:48)
279 at org.onap.dcae.collectors.veshv.impl.adapters.ConsulConfigurationProvider$sam$java_util_function_Function$0.apply(ConsulConfigurationProvider.kt)
280 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:100)
281 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
282 at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmitScalar(FluxFlatMap.java:449)
283 at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:384)
284 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:275)
285 at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:849)
286 at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
287 at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
288 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476)
289 at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
290 at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:123)
291 at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:113)
292 at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:287)
293 at reactor.core.publisher.FluxUsing$UsingFuseableSubscriber.onNext(FluxUsing.java:350)
294 at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:113)
295 at reactor.core.publisher.FluxPeekFuseable$PeekFuseableConditionalSubscriber.onNext(FluxPeekFuseable.java:486)
296 at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1476)
297 at reactor.core.publisher.MonoReduceSeed$ReduceSeedSubscriber.onComplete(MonoReduceSeed.java:156)
298 at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136)
299 at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:378)
300 at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:202)
301 at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:343)
302 at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:325)
303 at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:372)
304 at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:522)
305 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
306 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
307 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
308 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
309 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
310 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
311 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
312 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
313 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
314 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
315 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
316 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
317 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
318 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
319 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
320 at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808)
321 at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410)
322 at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310)
323 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
324 at java.base/java.lang.Thread.run(Thread.java:834)
325 | ors.veshv.healthcheck.factory.HealthCheckApiServer | DEBUG | HV-VES status: OUT_OF_SERVICE, Consul configuration not available. Retrying.
328 This log is printed when you upload a configuration in an invalid format (for example, with missing fields). In the first log you can see that configuration on Consul is:
333 "invalidKey": "value"
336 The above is not a valid **HV-VES** configuration, therefore **HV-VES** does not apply it and becomes **unhealthy**.
337 For more information on **Consul configuration**, see :ref:`run_time_configuration`.
340 Message handling errors
341 -----------------------
343 **Handling messages when invalid Kafka url is specified**
347 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | DEBUG | Client connection request received
348 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | INFO | Handling new connection
349 | org.apache.kafka.clients.ClientUtils | WARN | Removing server invalid-kafka-host:9092 from bootstrap.servers as DNS resolution failed for invalid-kafka-host
350 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | WARN | Error while handling message stream: org.apache.kafka.common.KafkaException (Failed to construct kafka producer)
351 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | DEBUG | Detailed stack trace | org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
352 at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:66)
353 at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:406)
354 ... 49 common frames omitted
355 Wrapped by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
356 at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:457)
357 at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:289)
358 at reactor.kafka.sender.internals.ProducerFactory.createProducer(ProducerFactory.java:33)
359 at reactor.kafka.sender.internals.DefaultKafkaSender.lambda$new$0(DefaultKafkaSender.java:96)
360 at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:56)
361 at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
362 at reactor.core.publisher.Mono.subscribe(Mono.java:3590)
363 at reactor.core.publisher.MonoProcessor.add(MonoProcessor.java:531)
364 at reactor.core.publisher.MonoProcessor.subscribe(MonoProcessor.java:444)
365 at reactor.core.publisher.MonoFlatMapMany.subscribe(MonoFlatMapMany.java:49)
366 at reactor.core.publisher.FluxPeek.subscribe(FluxPeek.java:83)
367 at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62)
368 at reactor.core.publisher.FluxPeek.subscribe(FluxPeek.java:83)
369 at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
370 at reactor.core.publisher.FluxPeek.subscribe(FluxPeek.java:83)
371 at reactor.core.publisher.FluxOnErrorResume.subscribe(FluxOnErrorResume.java:47)
372 at reactor.core.publisher.FluxDoFinally.subscribe(FluxDoFinally.java:73)
373 at reactor.core.publisher.MonoIgnoreElements.subscribe(MonoIgnoreElements.java:37)
374 at reactor.netty.tcp.TcpServerHandle.onStateChange(TcpServerHandle.java:64)
375 at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:226)
376 at reactor.netty.channel.ChannelOperationsHandler.channelActive(ChannelOperationsHandler.java:112)
377 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
378 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
379 at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
380 at reactor.netty.tcp.SslProvider$SslReadHandler.userEventTriggered(SslProvider.java:720)
381 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
382 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
383 at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
384 at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1530)
385 at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:937)
386 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1360)
387 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
388 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
389 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
390 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
391 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
392 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
393 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
394 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
395 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
396 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
397 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
398 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
399 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
400 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
401 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:528)
402 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:482)
403 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
404 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
405 at java.base/java.lang.Thread.run(Thread.java:834)
406 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | DEBUG | Released buffer memory after handling message stream
409 **HV-VES** responds with the above when it handles a message and specified DmaaP MR Kafka bootstrap server is invalid.
410 Restart with different ``--kafka-bootstrap-servers`` command line option value is required.
411 For more information, see: :ref:`deployment`
415 **Kafka service became unavailable after producer has been created**
417 **HV-VES** lazily creates Kafka consumer after first successfully handled event.
418 If Kafka service becomes unreachable after consumer initialization, it is removed from bootstrap.servers list on next connection.
420 Following information is logged:
424 | org.apache.kafka.clients.NetworkClient | WARN | [Producer clientId=producer-1] Connection to node 1001 could not be established. Broker may not be available.
425 | org.apache.kafka.clients.NetworkClient | WARN | [Producer clientId=producer-1] Connection to node 1001 could not be established. Broker may not be available.
426 | org.apache.kafka.clients.NetworkClient | WARN | [Producer clientId=producer-1] Connection to node 1001 could not be established. Broker may not be available.
427 | org.apache.kafka.clients.NetworkClient | WARN | [Producer clientId=producer-1] Connection to node 1001 could not be established. Broker may not be available.
428 | org.apache.kafka.clients.NetworkClient | WARN | [Producer clientId=producer-1] Error connecting to node message-router-kafka:9092 (id: 1001 rack: null) | | java.nio.channels.UnresolvedAddressException: null
429 at java.base/sun.nio.ch.Net.checkAddress(Net.java:130)
430 at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:675)
431 at org.apache.kafka.common.network.Selector.doConnect(Selector.java:233)
432 ... 9 common frames omitted
433 Wrapped by: java.io.IOException: Can't resolve address: message-router-kafka:9092
434 at org.apache.kafka.common.network.Selector.doConnect(Selector.java:235)
435 at org.apache.kafka.common.network.Selector.connect(Selector.java:214)
436 at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:864)
437 at org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:64)
438 at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1035)
439 at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:920)
440 at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:508)
441 at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
442 at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
443 at java.base/java.lang.Thread.run(Thread.java:834)
444 | | kafka-producer-network-thread | producer-1
445 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | INFO | Handling new connection
446 | org.apache.kafka.clients.ClientUtils | WARN | Removing server message-router-kafka:9092 from bootstrap.servers as DNS resolution failed for message-router-kafka
447 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | WARN | Error while handling message stream: org.apache.kafka.common.KafkaException (Failed to construct kafka producer)
448 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | DEBUG | Detailed stack trace
449 at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:66)
450 at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:406)
451 ... 48 common frames omitted
452 Wrapped by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
453 at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:457)
454 at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:289)
455 at reactor.kafka.sender.internals.ProducerFactory.createProducer(ProducerFactory.java:33)
456 at reactor.kafka.sender.internals.DefaultKafkaSender.lambda$new$0(DefaultKafkaSender.java:96)
457 at reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:56)
458 at reactor.core.publisher.MonoPeekFuseable.subscribe(MonoPeekFuseable.java:74)
459 at reactor.core.publisher.Mono.subscribe(Mono.java:3590)
460 at reactor.core.publisher.MonoProcessor.add(MonoProcessor.java:531)
461 at reactor.core.publisher.MonoProcessor.subscribe(MonoProcessor.java:444)
462 at reactor.core.publisher.MonoFlatMapMany.subscribe(MonoFlatMapMany.java:49)
463 at reactor.core.publisher.FluxPeek.subscribe(FluxPeek.java:83)
464 at reactor.core.publisher.FluxMap.subscribe(FluxMap.java:62)
465 at reactor.core.publisher.FluxPeek.subscribe(FluxPeek.java:83)
466 at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
467 at reactor.core.publisher.FluxPeek.subscribe(FluxPeek.java:83)
468 at reactor.core.publisher.FluxOnErrorResume.subscribe(FluxOnErrorResume.java:47)
469 at reactor.core.publisher.FluxDoFinally.subscribe(FluxDoFinally.java:73)
470 at reactor.core.publisher.MonoIgnoreElements.subscribe(MonoIgnoreElements.java:37)
471 at reactor.netty.tcp.TcpServerHandle.onStateChange(TcpServerHandle.java:64)
472 at reactor.netty.tcp.TcpServerBind$ChildObserver.onStateChange(TcpServerBind.java:226)
473 at reactor.netty.channel.ChannelOperationsHandler.channelActive(ChannelOperationsHandler.java:112)
474 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
475 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
476 at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
477 at reactor.netty.tcp.SslProvider$SslReadHandler.userEventTriggered(SslProvider.java:720)
478 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)
479 at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)
480 at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)
481 at io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1530)
482 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1368)
483 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
484 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
485 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
486 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
487 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
488 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
489 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
490 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
491 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
492 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
493 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
494 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
495 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
496 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
497 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:528)
498 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:482)
499 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
500 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
501 at java.base/java.lang.Thread.run(Thread.java:834)
502 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | DEBUG | Released buffer memory after handling message stream
505 To resolve this issue, you can either wait for that Kafka service to be available, or just like in previous paragraph, restart **HV-VES** with different value of ``--kafka-bootstrap-servers`` option.
509 **Message with too big payload size**
513 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | DEBUG | Client connection request received
514 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | INFO | Handling new connection
515 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | Got message with total size of 16384 B
516 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | WARN | Error while handling message stream: org.onap.dcae.collectors.veshv.impl.wire.WireFrameException (PayloadSizeExceeded: payload size exceeds the limit (1048576 bytes))
517 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | DEBUG | Detailed stack trace | org.onap.dcae.collectors.veshv.impl.wire.WireFrameException: PayloadSizeExceeded: payload size exceeds the limit (1048576 bytes)
518 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$onError$1$1.invoke(WireChunkDecoder.kt:72)
519 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$onError$1$1.invoke(WireChunkDecoder.kt:41)
520 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:33)
521 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:27)
522 at arrow.effects.IORunLoop.step(IORunLoop.kt:49)
523 at arrow.effects.IO.unsafeRunTimed(IO.kt:115)
524 at arrow.effects.IO.unsafeRunSync(IO.kt:112)
525 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$generateFrames$1.accept(WireChunkDecoder.kt:66)
526 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$generateFrames$1.accept(WireChunkDecoder.kt:41)
527 at reactor.core.publisher.FluxGenerate.lambda$new$1(FluxGenerate.java:56)
528 at reactor.core.publisher.FluxGenerate$GenerateSubscription.slowPath(FluxGenerate.java:262)
529 at reactor.core.publisher.FluxGenerate$GenerateSubscription.request(FluxGenerate.java:204)
530 at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:1849)
531 at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:68)
532 at reactor.core.publisher.FluxGenerate.subscribe(FluxGenerate.java:83)
533 at reactor.core.publisher.FluxOnErrorResume.subscribe(FluxOnErrorResume.java:47)
534 at reactor.core.publisher.FluxDoFinally.subscribe(FluxDoFinally.java:73)
535 at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
536 at reactor.core.publisher.Flux.subscribe(Flux.java:7734)
537 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442)
538 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244)
539 at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
540 at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
541 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
542 at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211)
543 at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327)
544 at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310)
545 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
546 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
547 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
548 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
549 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
550 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
551 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
552 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
553 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1429)
554 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
555 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
556 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
557 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
558 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
559 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
560 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
561 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
562 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
563 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
564 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
565 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
566 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
567 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
568 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:528)
569 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:482)
570 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
571 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
572 at java.base/java.lang.Thread.run(Thread.java:834)
573 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | WARN | Error while handling message stream: org.onap.dcae.collectors.veshv.impl.wire.WireFrameException (PayloadSizeExceeded: payload size exceeds the limit (1048576 bytes))
574 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | DEBUG | Detailed stack trace | org.onap.dcae.collectors.veshv.impl.wire.WireFrameException: PayloadSizeExceeded: payload size exceeds the limit (1048576 bytes)
575 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$onError$1$1.invoke(WireChunkDecoder.kt:72)
576 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$onError$1$1.invoke(WireChunkDecoder.kt:41)
577 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:33)
578 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:27)
579 at arrow.effects.IORunLoop.step(IORunLoop.kt:49)
580 at arrow.effects.IO.unsafeRunTimed(IO.kt:115)
581 at arrow.effects.IO.unsafeRunSync(IO.kt:112)
582 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$generateFrames$1.accept(WireChunkDecoder.kt:66)
583 at org.onap.dcae.collectors.veshv.impl.wire.WireChunkDecoder$generateFrames$1.accept(WireChunkDecoder.kt:41)
584 at reactor.core.publisher.FluxGenerate.lambda$new$1(FluxGenerate.java:56)
585 at reactor.core.publisher.FluxGenerate$GenerateSubscription.slowPath(FluxGenerate.java:262)
586 at reactor.core.publisher.FluxGenerate$GenerateSubscription.request(FluxGenerate.java:204)
587 at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:1849)
588 at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:68)
589 at reactor.core.publisher.FluxGenerate.subscribe(FluxGenerate.java:83)
590 at reactor.core.publisher.FluxOnErrorResume.subscribe(FluxOnErrorResume.java:47)
591 at reactor.core.publisher.FluxDoFinally.subscribe(FluxDoFinally.java:73)
592 at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54)
593 at reactor.core.publisher.Flux.subscribe(Flux.java:7734)
594 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:442)
595 at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:244)
596 at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
597 at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:192)
598 at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114)
599 at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211)
600 at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327)
601 at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310)
602 at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141)
603 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
604 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
605 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
606 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
607 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
608 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
609 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
610 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1429)
611 at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199)
612 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243)
613 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
614 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
615 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
616 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
617 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
618 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
619 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
620 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
621 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
622 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
623 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
624 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:628)
625 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:528)
626 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:482)
627 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
628 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
629 at java.base/java.lang.Thread.run(Thread.java:834)
630 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | DEBUG | Released buffer memory after handling message stream
633 The above log is printed when the message payload size is too big. **HV-VES** does not handle messages that exceed specified payload size. Default value is **1048576 bytes (1MiB)**, but it can be configured via cmd or by environment variables.
639 Messages with invalid Google Protocol Buffers data encoded are omitted. **HV-VES** responds as follows:
643 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | DEBUG | Client connection request received
644 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | INFO | Handling new connection
645 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | Got message with total size of 28 B
646 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | Wire payload size: 16 B
647 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | TRACE | Wire frame header is valid
648 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | WARN | Failed to decode ves event header, reason: Protocol message tag had invalid wire type.
649 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | End of data in current TCP buffer
653 **Invalid Wire Frame**
655 Messages with invalid Wire Frame, just like those containing invalid GPB data, will be dropped. The exact reason can be found in logs.
659 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | DEBUG | Client connection request received
660 | p.dcae.collectors.veshv.impl.socket.NettyTcpServer | INFO | Handling new connection
661 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | Got message with total size of 322 B
662 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | Wire payload size: 310 B
663 | org.onap.dcae.collectors.veshv.impl.VesHvCollector | WARN | Invalid wire frame header, reason: Invalid major version in wire frame header. Expected 1 but was 2
664 | p.dcae.collectors.veshv.impl.wire.WireChunkDecoder | TRACE | End of data in current TCP buffer
670 For more information, see the :ref:`hv_ves_behaviors` section.
673 Authorization related errors
674 ----------------------------
676 **WARNING: SSL/TLS authorization is a part of an experimental feature for ONAP Casablanca release and should be treated as unstable and subject to change in future releases.**
677 **For more information, see** :ref:`ssl_tls_authorization`.
679 **Key or trust store missing**
683 | org.onap.dcae.collectors.veshv.main | ERROR | Failed to start a server | java.io.FileNotFoundException: /etc/ves-hv/server.p12 (No such file or directory)
684 at java.io.FileInputStream.open0(Native Method)
685 at java.io.FileInputStream.open(FileInputStream.java:195)
686 at java.io.FileInputStream.<init>(FileInputStream.java:138)
687 at org.onap.dcae.collectors.veshv.ssl.boundary.UtilsKt$streamFromFile$1.invoke(utils.kt:79)
688 at org.onap.dcae.collectors.veshv.ssl.boundary.UtilsKt$streamFromFile$1.invoke(utils.kt)
689 at org.onap.dcae.collectors.veshv.ssl.impl.SslFactories.loadKeyStoreFromFile(SslFactories.kt:50)
690 at org.onap.dcae.collectors.veshv.ssl.impl.SslFactories.keyManagerFactory(SslFactories.kt:43)
691 at org.onap.dcae.collectors.veshv.ssl.boundary.ServerSslContextFactory.jdkContext(ServerSslContextFactory.kt:42)
692 at org.onap.dcae.collectors.veshv.ssl.boundary.SslContextFactory.createSslContextWithConfiguredCerts(SslContextFactory.kt:49)
693 at org.onap.dcae.collectors.veshv.ssl.boundary.SslContextFactory.createSslContext(SslContextFactory.kt:39)
694 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer.configureServer(NettyTcpServer.kt:61)
695 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer.access$configureServer(NettyTcpServer.kt:46)
696 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1$ctx$1.invoke(NettyTcpServer.kt:52)
697 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1$ctx$1.invoke(NettyTcpServer.kt:46)
698 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$sam$java_util_function_Consumer$0.accept(NettyTcpServer.kt)
699 at reactor.ipc.netty.tcp.TcpServer.<init>(TcpServer.java:149)
700 at reactor.ipc.netty.tcp.TcpServer$Builder.build(TcpServer.java:278)
701 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1.invoke(NettyTcpServer.kt:53)
702 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1.invoke(NettyTcpServer.kt:46)
703 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:28)
704 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:22)
705 at arrow.effects.IORunLoop.step(IORunLoop.kt:50)
706 at arrow.effects.IO.unsafeRunTimed(IO.kt:109)
707 at arrow.effects.IO.unsafeRunSync(IO.kt:106)
708 at org.onap.dcae.collectors.veshv.utils.arrow.EffectsKt.unsafeRunEitherSync(effects.kt:50)
709 at org.onap.dcae.collectors.veshv.main.MainKt.main(main.kt:41)
712 The above error is logged when key store is not provided. Similarly, when trust store is not provided, **/etc/ves-hv/trust.p12** file missing is logged.
713 **server.p12** and **trust.p12** are default names of key and trust stores. They can be changed by specifying ``--trust-store`` or ``--key-store`` command line arguments on deployment.
717 **Invalid credentials**
721 | org.onap.dcae.collectors.veshv.main | ERROR | Failed to start a server | java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
722 ... 23 common frames omitted
723 Wrapped by: java.io.IOException: keystore password was incorrect
724 at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2059)
725 at java.security.KeyStore.load(KeyStore.java:1445)
726 at org.onap.dcae.collectors.veshv.ssl.impl.SslFactories.loadKeyStoreFromFile(SslFactories.kt:51)
727 at org.onap.dcae.collectors.veshv.ssl.impl.SslFactories.keyManagerFactory(SslFactories.kt:43)
728 at org.onap.dcae.collectors.veshv.ssl.boundary.ServerSslContextFactory.jdkContext(ServerSslContextFactory.kt:42)
729 at org.onap.dcae.collectors.veshv.ssl.boundary.SslContextFactory.createSslContextWithConfiguredCerts(SslContextFactory.kt:49)
730 at org.onap.dcae.collectors.veshv.ssl.boundary.SslContextFactory.createSslContext(SslContextFactory.kt:39)
731 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer.configureServer(NettyTcpServer.kt:61)
732 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer.access$configureServer(NettyTcpServer.kt:46)
733 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1$ctx$1.invoke(NettyTcpServer.kt:52)
734 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1$ctx$1.invoke(NettyTcpServer.kt:46)
735 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$sam$java_util_function_Consumer$0.accept(NettyTcpServer.kt)
736 at reactor.ipc.netty.tcp.TcpServer.<init>(TcpServer.java:149)
737 at reactor.ipc.netty.tcp.TcpServer$Builder.build(TcpServer.java:278)
738 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1.invoke(NettyTcpServer.kt:53)
739 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1.invoke(NettyTcpServer.kt:46)
740 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:28)
741 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:22)
742 at arrow.effects.IORunLoop.step(IORunLoop.kt:50)
743 at arrow.effects.IO.unsafeRunTimed(IO.kt:109)
744 at arrow.effects.IO.unsafeRunSync(IO.kt:106)
745 at org.onap.dcae.collectors.veshv.utils.arrow.EffectsKt.unsafeRunEitherSync(effects.kt:50)
746 at org.onap.dcae.collectors.veshv.main.MainKt.main(main.kt:41)
749 Key or trust store password provided in configuration is invalid.
753 **Invalid key store file**
757 | org.onap.dcae.collectors.veshv.main | ERROR | Failed to start a server | java.io.IOException: DerInputStream.getLength(): lengthTag=111, too big.
758 at sun.security.util.DerInputStream.getLength(DerInputStream.java:599)
759 at sun.security.util.DerValue.init(DerValue.java:391)
760 at sun.security.util.DerValue.<init>(DerValue.java:332)
761 at sun.security.util.DerValue.<init>(DerValue.java:345)
762 at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1938)
763 at java.security.KeyStore.load(KeyStore.java:1445)
764 at org.onap.dcae.collectors.veshv.ssl.impl.SslFactories.loadKeyStoreFromFile(SslFactories.kt:51)
765 at org.onap.dcae.collectors.veshv.ssl.impl.SslFactories.keyManagerFactory(SslFactories.kt:43)
766 at org.onap.dcae.collectors.veshv.ssl.boundary.ServerSslContextFactory.jdkContext(ServerSslContextFactory.kt:42)
767 at org.onap.dcae.collectors.veshv.ssl.boundary.SslContextFactory.createSslContextWithConfiguredCerts(SslContextFactory.kt:49)
768 at org.onap.dcae.collectors.veshv.ssl.boundary.SslContextFactory.createSslContext(SslContextFactory.kt:39)
769 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer.configureServer(NettyTcpServer.kt:61)
770 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer.access$configureServer(NettyTcpServer.kt:46)
771 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1$ctx$1.invoke(NettyTcpServer.kt:52)
772 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1$ctx$1.invoke(NettyTcpServer.kt:46)
773 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$sam$java_util_function_Consumer$0.accept(NettyTcpServer.kt)
774 at reactor.ipc.netty.tcp.TcpServer.<init>(TcpServer.java:149)
775 at reactor.ipc.netty.tcp.TcpServer$Builder.build(TcpServer.java:278)
776 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1.invoke(NettyTcpServer.kt:53)
777 at org.onap.dcae.collectors.veshv.impl.socket.NettyTcpServer$start$1.invoke(NettyTcpServer.kt:46)
778 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:28)
779 at arrow.effects.IO$Companion$invoke$1.invoke(IO.kt:22)
780 at arrow.effects.IORunLoop.step(IORunLoop.kt:50)
781 at arrow.effects.IO.unsafeRunTimed(IO.kt:109)
782 at arrow.effects.IO.unsafeRunSync(IO.kt:106)
783 at org.onap.dcae.collectors.veshv.utils.arrow.EffectsKt.unsafeRunEitherSync(effects.kt:50)
784 at org.onap.dcae.collectors.veshv.main.MainKt.main(main.kt:41)
786 The above is logged when provided keystore has invalid or corrupted content.
787 This log also appears when you try to use key store/trust store in archive format other than **PKCS12** (the only supported by **HV-VES** store type).