Platform > Operations

f1 Console

The f1 console is a command line administration interface into a running AmiCore instance, reached over SSH.

Overview

Each 3forge Enterprise component registers a set of console objects when it starts, and the console exposes the methods on those objects as commands. Use the console to read runtime state that the Web interface does not surface, to run the administration commands that exist only here, and to measure how deeply work is queued inside a component.

AmiCore opens 2 separate console interfaces on different ports. They are not the same service:

• f1.console.port, default 3285: the f1 console, described on this page.
• ami.db.console.port, default 3290: the Center's SQL console. See Center.

Note

"Partition" carries 2 unrelated meanings in AMI. On this page a partition is a single lane execution queue inside a running container. The storage sense divides a historical table across files on disk, and is covered in Partitioning.

Connecting

The console is served over SSH on f1.console.port, which defaults to 3285. Startup prints the command to connect. Connect with any standard SSH client and enter the password of an admin account when prompted:

ssh -p 3285 user@hostmachine

Set the port in local.properties:

# Console port
f1.console.port=3285

Setting f1.console.port=-1 stops AmiCore starting, so the listener cannot be disabled this way.

Host key

The SSH host key is held in a keystore, which defaults to the keystore shared across the installation:

# Keystore holding the SSH host key. Defaults to keys/3forge.jks
f1.console.keystore.file=${ami.keyfile}

# Keystore password. Required, and startup fails without it
f1.console.keystore.password=${ami.keypass}

# Entry in the keystore that holds the host key
f1.console.keystore.alias=<alias of the host key entry>

AmiCore generates the key pair if the keystore does not yet exist, so no manual step is needed to create one.

Leaving f1.console.keystore.file blank does not disable SSH. The blank path resolves to the installation directory, and startup fails with an error creating the keystore.

Warning

SSH authentication accepts a password only. Public key authentication is not supported, so the keys an operations team normally uses for SSH access will not work here. Credentials are the same admin accounts used for the Web admin interface, and a session is closed after 3 failed attempts.

Logging in

SSH authenticates the user when the connection is made, and the console then opens at the prompt:

Welcome to the 3forge admin console - Version 3.0
AmiCore>

Console credentials come from the same admin user store as the Web admin interface, and permissions are resolved the same way. The console has no login command, because SSH has already authenticated the user against the same store.

The session itself is controlled by 2 properties: f1.console.prompt, which sets the prompt string, and f1.console.history.file, which sets the command history file. History is written to .console_history.txt in the installation directory by default.

Discovering Commands

Available commands can be found from inside a live session rather than from a fixed list.

Run SHOW OBJECTS to list every registered object with its type and a short description:

AmiCore>show objects
+------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                               SHOW OBJECTS (9 objects)                                                               |
+-----------+-----------+------------------------------------------------------------------------------------------------------------------------------+
|Object     |Type       |Help                                                                                                                          |
+-----------+-----------+------------------------------------------------------------------------------------------------------------------------------+
|aidoc      |AmiDocs    |AMI reference material -- docs (aidoc_getDocumentation()) and dashboard patterns (aidoc_getPattern(), aidoc_search_patterns())|
|ami        |AmiManager |Add/Remove components. View Plugins, Security Warnings, etc                                                                   |
|ami_ZF_gw  |AmiGateway |Diagnose and manage this GATEWAY component                                                                                    |
|ami_ZF_web2|AmiWeb     |Diagnose and manage this WEB component                                                                                        |
|ami_center |AmiCenter  |Diagnose and manage this CENTER component                                                                                     |
|ami_relay  |AmiRelay   |Diagnose and manage this RELAY component                                                                                      |
|ami_web    |AmiWeb     |Diagnose and manage this WEB component                                                                                        |
|bs         |Bootstrap  |Tools and services for bootstapping the application                                                                           |
|log        |SpeedLogger|Manage logger configurations and view logs                                                                                    |
+-----------+-----------+------------------------------------------------------------------------------------------------------------------------------+

(EXECUTED IN 0.55 MILLISECONDS, 120.30 MB IN USE)

The listing returns 3 columns: Object, Type, and Help. To list the console's own commands, run help.

These commands are available in every session:

Command	Purpose
help	List every available command. Add a command name for detail on one
show objects	List every object that can be invoked from the console
show env	Environment overview: resolved properties, classpath, license, and JVM arguments
show cpu	Recent CPU usage
show vm	JVM options and boot classpath
show threads <pattern>	Thread dump, grouped by thread state
show history	List the command history of the session
flame threads <pattern>	Sample threads over a period and render them as a flame graph
setlocal [key[=value]]	Get or set the local variables of the session, such as timeout and limit
<expression>	Run an AmiScript expression, such as a method call on a console object
quit	Close the connection

Call a method by typing an expression on the object, such as ami_center.showQueues().

show threads and flame threads both require a search pattern and report Please provide a pattern to search for without one. Use * to match everything:

AmiCore>show threads *
AmiCore>flame threads *

show threads also accepts a thread state as a filter, as in show runnable threads *. flame threads accepts an optional sampling window written as total and period in milliseconds, as in flame 1000/5 threads *, which is also the default.

Some administration methods exist only in the console. The per component showPorts and showFiles methods are examples, reporting the server ports and configured file paths owned by one component and its plugins.

The Web exposes its session management commands on its component object, ami_web, including showLogins() and showSessions(). See Users and Sessions.

Component objects

Every component registers a console object named ami_ followed by the component's name in the components file. The Center's object is ami_center, the Relay's is ami_relay, and the Web's is ami_web. A component named ZF_gw registers ami_ZF_gw.

The object type depends on the component: AmiCenter, AmiRelay, AmiWeb, or AmiGateway. Each type carries methods specific to that component, and each returns its container from getContainer(). The container object is the same class for every component, so each container exposes an identical set of methods. What differs between components is the object name typed before the dot, and the partitions each container is running.

Other objects are registered alongside the component objects:

• ami: component management, plugins, and security warnings
• bs: bootstrap services
• log: logger configuration and logs
• aidoc: reference material, including documentation and dashboard patterns

Monitoring Queue Depth

A partition processes its work in a single lane, so a partition that cannot keep up accumulates a backlog rather than consuming more threads. Reading that backlog is the fastest way to tell whether a component is saturated and which part of it is responsible.

Run showQueues() on a component's object for a summary of every partition's backlog, then showPartitions() on its container for full detail. Both work on ami_center, ami_web, and ami_relay:

AmiCore>ami_center.showQueues()
AmiCore>ami_center.getContainer().showPartitions()

showQueues() returns 6 columns: Name, Actions, Queued, TimeMillis, Errors, and Locked. They match id, actions, queued, Time(ms), errors, and Locked in the table below.

The following captures were taken from a Center while the AMI_CENTER partition had a backlog:

AmiCore>ami_center.showQueues()
+----------------+-------+------+----------+-------+------+
|Name            |Actions|Queued|TimeMillis|Errors |Locked|
|String          |Long   |Long  |Long      |Integer|String|
+----------------+-------+------+----------+-------+------+
|AMI_CENTER_RR   |1      |0     |1         |0      |      |
|AMI_CENTER_AR   |2      |0     |1         |0      |      |
|MSG_AGENTS      |6      |0     |4         |0      |      |
|AMI_CENTERRQ    |76     |0     |4         |0      |      |
|AMI_CENTER_CC   |78     |0     |24        |0      |      |
|AMI_PREPROCESSOR|100    |0     |34        |0      |      |
|AMI_CENTER_CR   |103    |0     |7         |0      |      |
|MSG_CLIENTS     |126    |0     |5         |0      |      |
|AMI_CENTER      |2605   |21    |6832      |0      |WRITE |
+----------------+-------+------+----------+-------+------+


(EXECUTED IN 0.89 MILLISECONDS, 95.73 MB IN USE)

AmiCore>ami_center.getContainer().showPartitions()
+----------------+-----------------+------------+-------+-------+------+--------+------+-------+------+-------+
|id              |startTime        |threadpoolId|states |actions|queued|Time(ms)|visits|errors |Locked|InPool |
|String          |String           |String      |Integer|Integer|Long  |Long    |Long  |Integer|String|Boolean|
+----------------+-----------------+------------+-------+-------+------+--------+------+-------+------+-------+
|AMI_CENTER_RR   |20260929-10:02:51|F1POOL      |1      |1      |0     |1       |1     |0      |      |false  |
|AMI_CENTER_AR   |20260929-10:15:03|F1POOL      |1      |2      |0     |1       |1     |0      |      |false  |
|MSG_AGENTS      |20260929-10:02:51|F1POOL      |1      |6      |0     |4       |6     |0      |      |false  |
|AMI_CENTER_CC   |20260929-10:02:50|F1POOL      |1      |78     |0     |24      |77    |0      |      |false  |
|AMI_CENTERRQ    |20260929-10:19:59|F1POOL      |1      |83     |0     |4       |83    |0      |      |false  |
|AMI_PREPROCESSOR|20260929-10:19:57|F1POOL      |1      |107    |0     |35      |107   |0      |      |false  |
|AMI_CENTER_CR   |20260929-10:02:53|F1POOL      |1      |110    |0     |8       |110   |0      |      |false  |
|MSG_CLIENTS     |20260929-10:02:53|F1POOL      |1      |126    |0     |5       |66    |0      |      |false  |
|AMI_CENTER      |20260929-10:02:50|F1POOL      |1      |2613   |21    |7936    |2559  |0      |WRITE |true   |
+----------------+-----------------+------------+-------+-------+------+--------+------+-------+------+-------+


(EXECUTED IN 0.95 MILLISECONDS, 132.82 MB IN USE)

Reading the output

showPartitions() returns one row per partition, with 11 columns:

Column	Type	Meaning
id	String	The partition identifier
startTime	String	When the partition was created
threadpoolId	String	The thread pool the partition runs on
states	Integer	Number of state types the partition holds
actions	Integer	Actions processed since the partition started
queued	Long	Actions currently waiting to be processed
Time(ms)	Long	Total time spent processing actions
visits	Long	Number of times the queue has been run
errors	Integer	Errors recorded across the partition's states
Locked	String	Current lock state
InPool	Boolean	Whether the partition has actions to process. false means it is idle

queued is the column to read first. A value that is consistently above zero, or that climbs between two samples taken a few seconds apart, means the partition is receiving work faster than it can process it.

Locked is a string rather than a flag, and it reports read locks as well as write locks. It holds WRITE while a write lock is held, <n> READ while n read locks are held, and an empty value when the partition is unlocked. A partition showing WRITE alongside a rising queued value is being held up by a table mutation. A trigger, procedure, or timer locks the database while it runs, so a write lock is expected during execution and is worth investigating only when it persists.

In the captures above, AMI_CENTER shows queued 21 and Locked WRITE, so work is waiting behind a write in progress.

Rows are sorted by actions, so the busiest partitions appear at the bottom of the table.

Partitions to watch

Partition names identify the mechanism that owns the queue. The set depends on which component is inspected. Names ending in <n> carry a number assigned at runtime rather than fixed.

Center PartitionsWeb PartitionsRelay Partitions

• AMI_CENTER: the database itself. Actions here cover a wide range of activity, from Relay messages, replication, and write queries through triggers, procedures, and timers, down to routine status and connection checks.
• AMI_PREPROCESSOR: query parsing. A deep queue means queries are arriving faster than they can be parsed, or that the incoming queries are complex and slow to parse.
• AMI_CENTER_CC: client connection.
• AMI_CENTER_CR: client routing.
• AMI_CENTER_AR: agent routing. Messages arriving from the agent suite, which is the Relay connection carried on MSG_AGENTS, are routed on this partition. A deep queue means either that inbound data is arriving faster than the Center can process it, or that the Center has just connected and is taking a large initial snapshot.
• AMI_CENTER_RR: result routing. Responses are routed back to their caller on this partition.
• AMI_CENTER_RQ: translation of an incoming request, such as datasource management or a passthrough to the Relay, into an itinerary.
• AMI_CENTERRQ: translation of an incoming query into an itinerary. The name has no underscore before RQ.
• MSG_AGENTS: the message transport between the Relay and the Center, carrying traffic in both directions.
• MSG_CLIENTS: the message transport between the Web and the Center, carrying traffic in both directions.
• HDBP_<table>: one partition per historical table, named with the table name.

• SNAPSHOT_PARTITION: the lane that pushes snapshots and deltas from the database out to the realtime visualizations in the Web. A deep queue means either that there are too many subscriptions, or that logic in the tables or datamodels, such as a column formula, is heavy enough that the interface cannot keep up with the push rate.
• USERLOGIN: carries user login requests to the Web interface. Every login is authenticated on this single lane, so a deep queue here delays logging in for all users. This is not the path the console login takes: Web logins are authenticated against the Web interface user store, while the console authenticates against the admin user store.
• PORTLET_BACKEND: a single fixed partition that routes backend responses into per Session processing. One instance exists per Web process.
• AMI_CENTER_CLIENT_<n>: one partition per Center connection opened by a client process, numbered in the order the connections are made. The Web and the Gateway both open these, so AMI_CENTER_CLIENT_1 is the first Center connection that process opened.
• <session ID>: one partition per active Session, named with the session ID. To map a session ID back to a user, run ami_web.showSessions().

• RELAY_TRANSFORM: application of the Relay's transforms to incoming messages.
• RELAY_CUSTOM_METHOD: execution of custom methods on messages after their transforms.
• AMI_RELAY: the main Relay lane. Messages reach it after any transforms and custom methods, or directly when neither is configured.
• RELAY_FILE_MONITOR: the periodic check of the Relay's watched configuration files, such as transforms, routes, and feed handlers, for changes.
• MSG_CENTER_<n>: one partition per Center that a Relay connects to, numbered from zero in the order the connections are added.

What an action means

An action is one unit of queued work, and it is processed as a single transaction. An action is not one message. The Center batches messages arriving from the Relay, up to the limit set by ami.center.relay.batch.messages.max, which defaults to 1000.

This means a partition can show a low actions count while carrying a high message volume. Read actions as a count of transactions, and size the batch limit against it when tuning.

Warning

Do not confuse ami.center.relay.batch.messages.max with ami.center.snapshot.batchsize or idfountain.batchsize. They control unrelated batching and have very different defaults.

Investigating a deep queue

Once showPartitions() identifies a partition with a backlog, inspect the queue itself:

AmiCore>ami_center.getContainer().showQueue("AMI_CENTER")

The method takes the partition ID and returns the queued actions, with columns Type, Position, Action, Source, and Target. Target names the processor that handles the action, which turns a queue depth number into an identifiable path through the system. Source is empty for the actions in the capture below.

A partition with nothing waiting returns the column headers and no rows. That is the normal result for an idle partition, not a failure. Because a single lane drains quickly, run this while showPartitions() still reports a non-zero queued value for that partition.

The following capture, abbreviated, was taken while the AMI_CENTER partition had a backlog:

AmiCore>ami_center.getContainer().showQueue("AMI_CENTER")
+---------------------------------+-------------------------------+------+------+---------------------------+
|Type                             |Position                       |Action|Source|Target                     |
|String                           |Integer                        |String|String|String                     |
+---------------------------------+-------------------------------+------+------+---------------------------+
|HighPriorityPartitionActionRunner|AmiCenterStartItineraryMessage0|0     !null  |AmiCenterItineraryProcessor|
|HighPriorityPartitionActionRunner|AmiCenterStartItineraryMessage0|1     !null  |AmiCenterItineraryProcessor|
|HighPriorityPartitionActionRunner|AmiCenterStartItineraryMessage0|2     !null  |AmiCenterItineraryProcessor|
|HighPriorityPartitionActionRunner|AmiCenterStartItineraryMessage0|3     !null  |AmiCenterItineraryProcessor|
...
|QueuedPartitionActionRunner      |CountMessage0                  |0     !null  |AmiCenterMonitorProcessor  |
+---------------------------------+-------------------------------+------+------+---------------------------+

The same container object carries further inspection methods:

Method	Returns
diagnose()	A diagnostic summary of the container
showStates()	The states held by each partition
showThreads()	The container's threads
showServices()	The services registered in the container
showContainerScopes()	The container's scopes and their start status

Investigating CPU and Thread Use

Queue depth shows how much work is waiting. These 3 commands show what the JVM's threads are doing, and they are most useful run in sequence.

Confirming CPU is the problem

Start with show cpu, which reports recent process CPU use as a percentage:

AmiCore>show cpu
CPU Usage: %0.42

A low figure means a slow instance is not CPU bound, and queue depth or lock contention is the more likely cause. Check that before reading any stacks.

Listing threads by state

show threads <pattern> returns a thread dump with a section for each thread state that has matching threads: WAITING, TIMED_WAITING, NEW, BLOCKED, TERMINATED, and RUNNABLE. Threads sharing an identical stack are collapsed into a single entry with a count, the shared name prefix, and the differing name suffixes.

Name a state before threads to filter to it:

AmiCore>show runnable threads *

Runnable threads are the only ones able to consume CPU, so that filter is usually the fastest way to narrow a CPU investigation.

Aggregating stacks with flame threads

flame threads <pattern> samples every matching thread repeatedly over a window, then merges the stacks into a tree. Each frame carries the number of samples in which it appeared. The tree is split into a WAITING/TIMED_WAITING section and a RUNNABLE section.

Set the window by placing a total and a period in milliseconds before threads:

AmiCore>flame 5000/10 threads *

That samples for 5 seconds at 10 millisecond intervals, giving 500 runs. The default is 1000/5.

Counts aggregate across every matching thread, so a frame's count can exceed the number of runs. A count of 3000 results from 6 threads parked on the same stack across 500 runs.

Warning

flame threads is not a CPU profile. It counts how often each frame was seen, including on threads that were blocked or idle, and unlike show threads it has no state filter. On a healthy instance the largest frames are usually threads parked waiting for work, consuming no CPU at all, and a thread waiting in native code such as an event loop or a socket read is still listed as RUNNABLE. Read the tree for what is doing work, not for what is biggest.

The following output, abbreviated, was taken from an idle instance:

AmiCore>flame 5000/10 threads *
Evaluated 500 Runs.
WAITING/TIMED_WAITING: All Matching Stack Traces [47993]
+-java.lang.Thread.run(Thread.java:748) [35000]
| +-com.f1.utils.concurrent.FastThreadPool.run(Unknown Source) [15500]
| |   java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) [15500]
| +-com.f1.msgdirect.MsgDirectConnection.run(Unknown Source) [3000]
| |   java.lang.Object.wait(Native Method) [3000]
| ...

RUNNABLE: All Matching Stack Traces [507]
+-java.lang.Thread.run(Thread.java:748) [500]
|   sun.awt.windows.WToolkit.run(WToolkit.java:316) [500]
|     sun.awt.windows.WToolkit.eventLoop(Native Method) [500]
+-com.f1.ami.one.AmiOneMain$3.run(Unknown Source) [6]
|   javax.swing.JLabel.setText(JLabel.java:330) [6]
|     ...
|               javax.swing.plaf.basic.BasicHTML.updateRenderer(BasicHTML.java:207) [6]

Nearly all of the 48500 samples sit in the WAITING/TIMED_WAITING section, in thread pools and message socket readers parked waiting for work. The RUNNABLE section holds 507 samples: 500 in the Windows AWT event loop, which is a native wait rather than computation, and 6 beneath JLabel.setText, where the interface is rendering a status label. That is an idle instance, not a busy one. A genuinely CPU bound instance looks different: the deep, high count branches in RUNNABLE sit in query, script, or application code rather than in a wait.