Platform > Operations > OpenTelemetry

Troubleshooting¶

This page lists OpenTelemetry export problems by symptom, with the exact log message each one produces and how to fix it.

Where to look¶

Export problems are reported only in the server's startup log, log/AmiCore.log, never at the collector. Search it for OTel export, OpenTelemetry, and ami.log.stats.otel.

The startup log also lists the export settings in use, each exactly as it was read. See step 5 of Sending metrics to a collector for the command and the expected output.

Each failed request also logs a general warning just before the export message, followed by a stack trace:

1	`Exception occured when sending rest request:`

Nothing arrives and the log has no export warnings¶

If the startup log lists only ami.log.stats.otel.enabled=false, the server did not read the setting that turns export on. Check that:

The property is spelled ami.log.stats.otel.enabled. The statistics file's own properties begin ami.amilog.stats, and an ami.amilog.stats.otel... property is ignored without a warning.
It is in config/local.properties of the installation that is running.
The server was restarted after the change. Settings are read only at startup.

Nothing arrives and the log shows "Failed to initialize OpenTelemetry stats exporter"¶

This warning at startup means one of the ami.log.stats.otel.* values could not be read, and export is off until the next restart:

1	`Failed to initialize OpenTelemetry stats exporter, export disabled.`

The lines that follow name the property and the file and line it came from, and the startup log's settings listing shows the value with the comment attached. The commonest cause is a comment on the same line as a value, e.g: ami.log.stats.otel.max.retries=2 # test. A # starts a comment only at the beginning of a line. Move the comment to a line of its own and restart.

Nothing arrives and the log shows "Invalid value for ami.log.stats.otel.enabled"¶

ami.log.stats.otel.enabled accepts only true or false. Any other value, such as yes, turns export off:

1	`Invalid value for ami.log.stats.otel.enabled; expected true or false. OpenTelemetry export disabled.`

Nothing arrives and the log shows "Could not create rest request"¶

The endpoint is missing its scheme. Set ami.log.stats.otel.endpoint to a URL that begins with http://, e.g: http://collector.example.com:4318, and restart.

With a value such as localhost:4318, nothing is sent, export never turns itself off, and no message mentions OpenTelemetry. The log shows this twice per interval:

1 2	`Could not create rest request: java.net.MalformedURLException:unknown protocol: localhost`

The collector rejects requests with HTTP 401¶

The collector requires credentials. Set ami.log.stats.otel.headers to a JSON object, e.g: {"Authorization":"Bearer <token>"}, and restart. Each rejected request logs this, where <endpoint> is the configured endpoint:

1	`OTel export request failed because java.io.IOException: Server returned HTTP response code: 401 for URL: <endpoint>/v1/metrics <response body> (1/5 consecutive failures)`

If headers is already set, check the startup log for this warning, which means the value is not a JSON object and was ignored:

1	`Ignoring ami.log.stats.otel.headers: value is not a valid JSON object (expected e.g. {"Authorization":"Basic <base64>"}); StringFormatException`

The HTTP form, Authorization: Bearer <token>, is the usual mistake. See Authenticating to a collector, including the warning there about the value being written to the startup log.

Requests to an HTTPS collector fail with "PKIX path building failed"¶

The server does not trust the collector's certificate:

OTel export request failed because sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (1/5 consecutive failures)

Add the certificate authority that issued the collector's certificate to the bundled Java runtime's trust store, then restart. See Sending to an HTTPS collector.

Export stopped and the log shows "OTel export disabled"¶

After 5 failed requests in a row, export turns off for as long as the server runs, even once the collector recovers. That takes 30 seconds or more after the first failure; see When the collector is unavailable:

1	`OTel export disabled after 5 consecutive failed requests, most recently because <reason>; no further OpenTelemetry export attempts will be made for the remainder of this process.`

Fix the cause shown as <reason>, then restart the server. Common reasons:

Reason in the log	Cause
`Server returned HTTP response code: <code> for URL: ...`	The collector answered with an error. The collector's response body follows the URL
`Connection refused: connect`	Nothing is listening at the endpoint
`the request timed out`	The collector did not respond within `ami.log.stats.otel.request.timeout`

Each request's data is dropped when it fails; it is not resent.

Metrics stop arriving but log records still arrive¶

The collector is accepting one kind of data and rejecting the other. Export keeps running and the rejected data is lost at every interval, because each successful request resets the failure count. The log repeats the same count every 15 seconds, where <endpoint> is the configured endpoint:

1	`OTel export request failed because java.io.IOException: Server returned HTTP response code: <code> for URL: <endpoint>/v1/metrics <response body> (1/5 consecutive failures)`

Check that the collector has a pipeline for both metrics and logs. The same applies with the roles reversed.

The statistics file is written less often than configured¶

Requests to the collector are sent from the thread that writes most of the statistics file, so a collector that is slow to respond delays the file too. With a 2-second timeout and a collector that did not respond, Process statistics were recorded every 19 seconds instead of every 15. Fix the collector, or lower ami.log.stats.otel.request.timeout.

A metric is missing¶

The statistic has no metric: component lifecycle and WebBalancer statistics arrive only as log records. See Statistics sent only as log records.
There is no data for it yet: AmiCenterTimers.* and AmiCenterTriggers.* appear only after the Center has a timer or a trigger.
It is not sent in every request: each request carries only what was recorded since the one before. OperatingSystem.* and FileSystem.* arrive once a minute and Environment.* once per start. See How often each metric arrives.

A metric has many more series than expected¶

Text columns are sent as data-point attributes. AmiCenterTables.count has one series per table, named in its type attribute, and Environment.* metrics carry 16 host and JVM attributes, including license dates. See Attributes on data points.

Data stops arriving from a server¶

There is no health or "up" metric. A server that stops, and a server whose export has turned itself off, both look the same at the collector: data stops arriving. A normal stop sends one final request first.

To tell the two apart, check whether the server is still running, and search log/AmiCore.log for OTel export disabled. If the server is running and the message is there, export has turned itself off; see Export stopped and the log shows "OTel export disabled".