Platform > Operations > OpenTelemetry

Metrics reference

This page lists every metric 3forge Enterprise sends over OpenTelemetry, the attributes on each data point, the statistics that arrive only as log records, and the format of the log records and resource attributes.

Overview

Each metric is named <statistic type>.<column>, e.g: Process.freeMem or AmiCenterEvents.logins. Names have no prefix and keep their mixed case.

• Gauges are sent as OTLP gauges: the current value at each interval.
• Counters are sent as OTLP sums: cumulative and monotonic, counting from the moment export started in that process. Each point carries startTimeUnixNano set to that moment.
• All values are integers, sent as asInt.
• Units use the UCUM codes OpenTelemetry expects: By is bytes, ms is milliseconds, 1 is a plain number, and a unit in braces, such as {thread}, names what is counted.

There is no health, status, or "up" metric.

How often each metric arrives

Each request carries only the statistics recorded since the previous request, so not every metric is in every request. On a new installation:

Metrics	Arrive
Process.*	In every request, every 15 seconds
Partition.*, AmiCenterEvents.*, AmiWebHttpServer.*	In every request from the second on
AmiCenterTables.count, Topic.*	In most requests, not all
OperatingSystem.*, FileSystem.*	Once a minute, set by ami.amilog.stats.os.period
Environment.*	Once per start, in the second request only
AmiCenterStoredProcs.*	In the second request; not in later ones on an idle installation
AmiCenterTimers.*, AmiCenterTriggers.*	Only once the Center has a timer or a trigger

The first request, about 2 seconds after startup, carries only the 10 Process.* metrics. The second carries the most: 48 of the 54 on a new installation. Later requests carried 33 to 40.

A dashboard that expects a value every 15 seconds shows gaps for the metrics that arrive less often. Environment.* in particular is one point per process start.

Metrics by statistic type

The description column is the text sent in each metric's description field.

Process

JVM memory, garbage collection, and threads. Sent by every process.

Metric	Kind	Unit	Description (as sent)
Process.freeMem	gauge	By	Free JVM heap memory
Process.maxMem	gauge	By	Max JVM heap the process can allocate
Process.totMem	gauge	By	Total JVM heap currently allocated
Process.popsGcMem	gauge	By	Heap used after the most recent GC
Process.gcCount	counter	{gc}	Cumulative garbage-collection cycles
Process.threadBlocked	gauge	{thread}	Threads in BLOCKED state
Process.threadsNew	gauge	{thread}	Threads in NEW state
Process.threadsRunnable	gauge	{thread}	Threads in RUNNABLE state
Process.threadsTimedWaiting	gauge	{thread}	Threads in TIMED_WAITING state
Process.threadsWaiting	gauge	{thread}	Threads in WAITING state

Environment

Host and JVM facts, recorded once at startup, so each arrives once per process start. Sent by every process.

Metric	Kind	Unit	Description (as sent)
Environment.cores	gauge	{cpu}	Available CPU cores
Environment.pid	gauge	1	Process ID (identifier, not a real measurement)
Environment.maxHeap	gauge	By	Max JVM heap (same value as Process.maxMem)
Environment.jvmStartTime	gauge	ms	JVM start time (epoch millis)

Operating system and disk

Collected every ami.amilog.stats.os.period (1 minute by default). Sent by every process. OperatingSystem.cpuBps is the load of the whole machine, not of the 3forge process, and its first value after each start is 0.

Metric	Kind	Unit	Description (as sent)
OperatingSystem.cpuBps	gauge	1	System CPU load as a fraction x10000 (e.g. 3500 = 35%)
OperatingSystem.totalMem	gauge	By	Total physical memory
OperatingSystem.freeMem	gauge	By	Free physical memory
FileSystem.totalSize	gauge	By	Total filesystem capacity
FileSystem.usedSize	gauge	By	Used filesystem space

Center

Tables, timers, triggers, procedures, and events. Sent by the Center.

Metric	Kind	Unit	Description (as sent)
AmiCenterTables.count	gauge	{row}	Current row count per center table
AmiCenterTimers.count	counter	{execution}	Cumulative center timer executions
AmiCenterTimers.millis	counter	ms	Cumulative center timer execution time
AmiCenterTimers.errors	counter	{error}	Cumulative center timer errors
AmiCenterTriggers.count	counter	{execution}	Cumulative center trigger executions
AmiCenterTriggers.millis	counter	ms	Cumulative center trigger execution time
AmiCenterTriggers.errors	counter	{error}	Cumulative center trigger errors
AmiCenterStoredProcs.count	counter	{execution}	Cumulative stored-procedure executions
AmiCenterStoredProcs.millis	counter	ms	Cumulative stored-procedure execution time
AmiCenterStoredProcs.errors	counter	{error}	Cumulative stored-procedure errors
AmiCenterEvents.rowsCount	gauge	{row}	Current total rows across the center
AmiCenterEvents.strPoolSize	gauge	{entry}	Current string-pool size
AmiCenterEvents.valPoolSize	gauge	{entry}	Current value-pool size
AmiCenterEvents.logins	counter	{login}	Cumulative user logins
AmiCenterEvents.logouts	counter	{logout}	Cumulative user logouts
AmiCenterEvents.events	counter	{event}	Cumulative events processed
AmiCenterEvents.relayEvents	counter	{event}	Cumulative relay events processed
AmiCenterEvents.dsQueries	counter	{query}	Cumulative datasource queries
AmiCenterEvents.snapshots	counter	{snapshot}	Cumulative snapshots taken
AmiCenterEvents.objExpires	counter	{expiration}	Cumulative object expirations

Queues and topics

Processing queues and messaging topics, one series per queue or topic. Sent by the Center, the Relay, and the Web.

Metric	Kind	Unit	Description (as sent)
Partition.processed	counter	{action}	Cumulative actions processed
Partition.queued	gauge	{action}	Actions currently queued
Partition.totExecTime	counter	ms	Cumulative partition execution time
Topic.msgCount	counter	{message}	Cumulative messages on the topic
Topic.queued	gauge	{message}	Messages currently queued on the topic
Topic.connections	gauge	{connection}	Active connections on the topic

Web

Web server sessions, connections, and cached rows. Sent by the Web.

Metric	Kind	Unit	Description (as sent)
AmiWebHttpServer.activeSessions	gauge	{session}	Active web sessions
AmiWebHttpServer.httpServices	gauge	{service}	Registered HTTP services
AmiWebHttpServer.activeConnections	gauge	{connection}	Active web connections
AmiWebHttpServer.userRows	gauge	{row}	Rows visible to users
AmiWebHttpServer.userHiddenRows	gauge	{row}	Rows hidden from users
AmiWebHttpServer.userHiddenAlwaysRows	gauge	{row}	Rows always hidden from users
AmiWebHttpServer.cachedRows	gauge	{row}	Rows currently cached
AmiWebHttpServer.centersConnected	gauge	{center}	Centers currently connected
AmiWebHttpServer.openedConnections	counter	{connection}	Cumulative web connections opened

Attributes on data points

Statistics that have text columns send them as attributes on every data point, so one metric can have several series.

Attribute	On	Meaning
component	Center, queue, topic, and Web metrics	The component the value belongs to, e.g: center, relay, web
name	AmiCenterTimers.*, AmiCenterTriggers.*, AmiCenterStoredProcs.*, Partition.*, Topic.*, FileSystem.*	The timer, trigger, procedure, queue, topic, or filesystem
type	AmiCenterTables.count	The table name, e.g: __ANNOTATION, despite the attribute's name
dir, connection	Topic.*	The topic's direction and connection
16 host and JVM attributes	Environment.*	See below

The Environment.* metrics carry these attributes: osName, osVersion, osArch, hostname, hostIp, workingDir, jvmName, jvmVersion, javaHome, user, timezone, amiVersion, licenseApp, licenseHost, licenseStartDate, and licenseEndDate. These values leave the machine with every export, including the host's IP address, the operating-system user, the installation path, and license details.

Statistics sent only as log records

Some statistics have no metric, and arrive only as log records:

• Component lifecycle: an AmiComponent log record for each component state change, with attributes type, name, pwd (the component's working directory), and state. At startup each component reports START and START_COMPLETE, then all components report ADD.
• Partition thread-pool column: the inThreadPool column of each Partition record.
• WebBalancer statistics: when a WebBalancer runs, it sends 3 kinds of log record at every interval:
  • AmiWebBalancerStats: one summary, with clientsActive, clientsIdle, serversUp, serversDown, activeConnections, totBytesToServers, totBytesToClients, and stickyCount.
  • AmiWebBalancerServer: one per server in its routes, with address, status (e.g: DOWN), activeSessions, bytesToServer, and avgResponseMillis.
  • AmiWebBalancerClient: one per client that has connected through it, with address, target, startTime, IdleSince, connections, bytesToServer, and bytesToClient.

Log records

Every statistics row is also sent to /v1/logs as one log record. Log records carry the same statistics as the metrics, not the server's log file.

Field	Value
severityNumber	9
severityText	INFO
body	The statistic type, as a string, e.g: Process
attributes	Every column of the row: text columns as stringValue, numbers as intValue
timeUnixNano	When the row was recorded

Resource attributes

Every request carries these resource attributes, all as strings:

Attribute	Value
service.name	ami.log.stats.otel.service.name, or AmiCore by default
service.instance.id	A new random id each time the process starts, e.g: F1-Rwv9YkPP4QUZ8464aAV5Oz
host.name	The machine's host name
process.pid	The process id
telemetry.source	runtime

The instrumentation scope is com.f1.ami.log.stats, with no version.