AMIScript: Scripting¶
AMI Script Introduction¶
Overview¶
AMI has an embedded, versatile language which is a combination of the well-known C/Java style, SQL and String Templating languages. Here is a very quick example of the incredible versatility of the language which does a cross-database join:
In this introductory language you can see the interoperability of SQL and a procedural style language. The ${...}
syntax is used for easily creating dynamic strings and SQL statements.
Key Features¶
AMI Script is a versatile language designed for optimized "on the fly" compilation, meaning expressions can be re-compiled and evaluated in an instant. Some key features of the language:
- Type Safe - All variables have well defined types, ex: String vs. Integer
- Compile Time Binding - Code is compiled before execution to minimize runtime issues
- Procedural - ** **Custom methods (with overloading), along with a set of predefined procedures
- Object Oriented - Objects in AMI are represented as such in Ami Script
- Event Driven - Executes based on events, such as running a datamodel or clicking a button
- Implicit Casting - Casting from one type to another can be explicitly or implicitly defined
- Embedded SQL - For accessing external sources and tables directly in the language
- String Templating - Ability to template code for reuse
- Task Swapping - Tasks can be exported runtime for out-of-band execution. (separate thread)
Embedded SQL is an SQL-style language extension to AMI Script with near-full SQL support along with several enhancements geared towards data preparation for the visualization of data. All of these functions act on an in-memory database. It also supports "calling out" to external datasources. Some key features are:
- Query Clauses - For filtering, joining, and grouping data on the in-memory database
- Data Modification - For adding, updating, and deleting data on the in-memory database
- Schema Modification - For adding, editing, and deleting tables/columns on the in-memory database
- Use ... Execute - For seamlessly running code on external datasources and storing results to the in-memory database
- Advanced Data Preparation and Analysis - For complex data calculations, not typically available on a standard SQL database
- Lambda Support -The ability to run AMI script code inside SQL. For example, to run a code snippet per row returned in a table
- Nested Queries - The in(...) clause supports full nested queries, and also supports multiple column in-clauses
- Intelligent Indexing - As joins, in-clauses, etc. are evaluated, indexes will automatically be spun up depending on data size and cardinality
AMI Script Template syntax is an extension of AMI Script with the aim of simplifying dynamic text generation, useful for creating dynamic HTML, dynamic SQL, etc.
- Embed Ami Script inside Text - Embed AMI script inside text allows for easy dynamic text generation
- Embed Ami Script inside SQL - Embed AMI script inside SQL allows for easy dynamic SQL generation
- Injection Protection - Properly escaping control characters, quotes, etc.
- Conditional Templating - Ability to do conditional & loop based template construction
Examples¶
Syntax¶
Same support as java for loops, if/else statements, white space, comments and expressions:
Data Structures¶
General purpose List, Map, Set & Table data structures are provided. The constructors can be used to prepopulate data:
Type Safety & Coercion¶
Type names are case insensitive, ex double and DOUBLE are the same. For compatibility, int and integer mean the same thing as well as char and character.
All types support null value. Dereferencing null, results in null. Expressions with null result in null
Flexible type coercion. When coercion is not possible the value is coerced to null.
All types extend Object and casting is not necessary for assignment.
All types can be thrown / caught and all blocks can have catch clauses (try key word is not necessary)
String Templates¶
If strings contain a \${...} the content of the curly brackets is interpreted
String Template Example¶
In this example, we will use string templates to access and evaluate variables from amiscript and pass them to a div field.
Alternatively, you could also use string template inside the div field:
You can use triple quotes for literals:
SQL¶
There are several reserved words that indicate an sql statement, including CREATE, SELECT, INSERT, DELETE, etc.
Variables and templates can be used inside sql:
Table coercion works for Table cells to objects, Columns to Lists and Rows to maps.
To query external data sources run the USE ds= ... EXECUTE command.
To upload data to external data sources run the USE ds= ... INSERT command.
Custom Methods¶
Methods are static and must return a type. If a return is clause is not reached, the function returns null.
Variable Naming¶
Case-sensitive combination of a-z, A-Z, _, 0-9. The first character must not be a number. Alternatively, variables can be wrapped in back ticks (`), in which case all characters are supported, allowing for accessibility with external language references. Note: `test`
is equivalent to test
. Variables are declared with the following syntax (the value will default to null if an expression is not supplied):
type variable_name [ = expression][, variable_name [ = expression] ... ];
Literals (Constants)¶
digits | Int constant |
0xdigits | Hex Int constant |
0digits | Octal Int constant |
digitsL | Long constant |
0xdigitsL | Hex long constant |
0digitsL | Octal long constant |
digits.digits | Float constant. Note, scientific (aka 'E') notation is supported |
digits.digitsD | Double constant. Note, scientific (aka 'E') notation is supported |
"chars" | String constant. Use backslash (\ to escape quotes, backslash (\) and other control chars, ex: \n. A string can be defined over multiple lines using a trailing backslash on all but the last line, note that preceding white space is trimmed out for subsequent lines. Please see String Templating for details on \${...} syntax |
\n\r\t |
White space (token delimiter) |
Reserved Type¶
All reserved types' values are immutable. Null is a valid value
Long | 64 bit signed whole number |
Double | 64 bit signed float |
String | Variable length string of UTF 8 characters |
Integer | 32 bit signed whole number |
Float | 32 bit signed float |
Byte | 8 bit signed whole number |
Boolean | boolean (true or false) |
Number | Base class for Long,Double,Integer,Float,Byte |
UTC | Timestamp with millisecond precision (since unix epoch) |
Examples
Casting¶
Variables are strongly typed, but casting can be explicit or implicit. If a cast fails, the result evaluates to null. Also, equality operations (==, <=, >=, >, <, !=) will also auto cast when possible as well.
Examples
Variable Scoping¶
Scopes are defined using statement blocks, which are denoted by curly brackets: {...}. Within a statement block, variables are only visible after they are declared. When statement blocks are nested inside other statement blocks, all variables declared in the outer block are visible within the inner block as long as the variable is declared before the inner block. A variable in the same scope cannot be re-declared when another variable with the same name is visible.
Examples
Standard Operators¶
The operators below are ordered by significance.
math_expressions¶
(evaluates to num_var type)
num_expr | * | num_expr | multiplication |
num_expr | / | num_expr | division |
num_expr | % | num_expr | modulus (remainder) |
num_expr | + | num_expr | addition for numbers |
num_expr | - | num_expr | subtraction |
binary_expressions¶
(evaluates to num_int_expr type)
int_expr | ~ | int_expr | bitwise NOT |
int_expr | << | int_expr | signed bit shift left |
int_expr | >> | int_expr | signed bit shift right |
int_expr | >>> | int_expr | unsigned bit shift right |
int_expr | & | int_expr | bitwise AND |
int_expr | ^ | int_expr | bitwise XOR |
int_expr | ! | int_expr | bitwise OR |
boolean_expressions¶
(evaluates to boolean)
! | bool_expr | boolean (logical) NOT | |
expr | < | expr | less than (strings are case sensitive) |
expr | > | expr | greater than (strings are case sensitive) |
expr | <= | expr | less than or equal to (strings are case sensitive) |
expr | >= | expr | greater than or equal to (strings are case sensitive) |
expr | == | expr | equal to (strings are case sensitive) |
expr | != | expr | is not equal to (strings are case sensitive) |
expr | =~ | regex | matches regular expression (see Java Pattern for details) |
expr | !~ | regex | does not match regular expression (see Java Pattern for details) |
expr | ~~ | simple_pattern | simplified text matching. See Simplified Text Matching section. |
expr | && | expr | logical and. Short circuits if left expr returns false. Short circuits if right expr is const and false |
expr | || | expr | logical or. Short circuits if left expr returns true. Short circuits if right expr is const and true |
string_expressions¶
(evaluates to boolean)
string_expr+ expr | String concatenation. Null values equate to empty string ("") |
special_expressions¶
(expression) | Force Order of operations |
(type)statement | Explicit cast to type. Note that a failed cast returns null. Also, casting is automatic, so it's not necessary |
bool_expr ? tr_expr : fl_expr | If bool_expr returns true, then evaluates to tr_expr, otherwise evaluates to fl_expr. |
Examples
Assignment & Declaration¶
special (evaluates to type)¶
new type(param[, param ...]) | Object Creation |
type var_name | Variable declaration (Value is defaulted to null) |
type var_name = expression | Variable declaration and assignment |
Variable declaration and assignment | Variable declaration and assignment |
assignment (evaluates to num_var type)¶
num_var | ++ | combined post-increment assignment | |
++ | num_var | combined pre-increment assignment | |
num_var | -- | combined post-decrement assignment | |
-- | num_var | combined pre-decrement assignment | |
num_var | *= | num_expr | combined multiplication assignment |
num_var | /= | num_expr | combined division assignment |
num_var | %= | num_expr | combined modulus assignment (remainder) |
num_var | += | num_expr | combined addition assignment |
num_var | -= | num_expr | combined subtraction assignment |
Flow Control¶
Complex AMI scripts can be achieved by combining multiple statements into statement blocks. A statement block is wrapped in curly brackets {...} and each statement is separated with a semi-colon;. Then, the execution of statement blocks can be determined using a variety of flow control statements including if, if/else, for, while, and for-each-loops. Note that for and while loops can use the continue and break directives as well.
Examples
statement block¶
Evaluates to return value if a return_statment exists, otherwise it evaluates to the last statement's evaluation. If no statements, evaluates to Void
for loop¶
initialization
- statement that is called first, used to declare variables with local scope
termination_expression
- statement that must evaluate to boolean, false terminates
increment
- statement called after each completion of the loop's statements
statement
-
called once per loop. Note:
continue_statement
will exit the statement and proceed to increment statementbreak_statement
will exit the for loop entirely
Evaluates to Void
for each loop¶
iterable_or_array
- values to iterate over in order
statement
-
called once per loop. Note:
continue_statement
will exit the statement and proceed to increment statementbreak_statement
will exit the for loop entirely
Evaluates to Void
while loop¶
Same as for(;bool_expression;) statement
Evaluates to Void
if statement¶
if bool_expression
evaluates to true then execute statement
Evaluates to Void
if else statement¶
if bool_expression
evaluates to true then execute statement
otherwise execute statement2
Evaluates to Void
return statement¶
Evaluates to statement
's value. Will cause call stack to be popped out to caller function.
break statement¶
See for_loop, for_each_loop and while_loop.
continue statement¶
See for_loop, for_each_loop and while_loop.
function call¶
Evaluates to return_type
value
member method call¶
Evaluates to return_type
value
Objects¶
There is a predefined set of object types (classes). These are used to facilitate the interaction between AMI Script and various backend components, user inputs, dashboards, etc. Additionally, data structures are represented as objects and can be instantiated using the new operator. Objects can only be accessed via methods (not member variables).
The Session Object¶
session
- each user that is logged in has a session object (which is of type Session). This is the "starting point" for access to the user's information, along with various dashboard components.
Attributes and Class Methods for a Session Object¶
-
session values: A hash map that stores user defined variables in the format of Variable_name_key=variable_value. Initially it is an empty map.
- session.setValue(string key, object value) && session.getValue(string key) && session.getValues()
-
session variables: A list of variables that are predefined by dashboard and webserver. They cam be viewed from Dashboard -> Session Variables
The dashboard editor can also define custom session variables by clicking Add variable. These are saved as part of the dashboard layout file
Now this variable that we just defined is accessible everywhere in the current session.
-
session global properties: these are server properties defined in the default.properties, local.properties and all other properties files
- session.getGlobalProperty(string propertyKey)
-
session Properties: another way to access session variables
Common Data Structures¶
List
: A zero-indexed Array List implementation
Set
: A set of unique values
Map
: A Linked Hash Map implementation
Table
: A 2-dimensional table with the same characteristics as a SQL style table. Tables have a title. Columns are named and strongly typed.
Tableset
: A map of tables. Can be loosely thought of as a mini database.
Examples
Commenting¶
Commenting is identical to the commenting syntax in Java & C++. There are two types of comments: single-line and multi-line.
comments (single-line)¶
comments (multi-line)¶
Examples
AMIScript Constants¶
AMI Script can reference constants just like other variables. Note the constant's values can not be changed using AMI Script. The standard naming convention for global variables is 2 under bars (__) followed by an all uppercase name. Constants can be defined in three ways:
-
At the server level (via a config file loaded at startup, typically local.properties)
Typically the syntax is:
amiscript.variable.varname=value
(note that if value is a string, is must be surrounded in double quotes, see literals for details)
For example, this could reside in your local.properties:
#My custom variables
amiscript.variable.__COMPANY_NAME="Acme Technologies"
amiscript.variable.__COPYRIGHT=2017
-
At the user level (using the authentication plugin, or the access.txt file)
Typically the syntax is:
{=html}amiscript.variable.varname=value
(note that if value is a string, is must be surrounded in double quotes, see literals for details)
For example, this could reside in your access.txt file for user demo (with password demo123):
demo|demo123|amiscript.variable.__GROUP="infosec"|amiscript.variable.__SECURITY_LEVEL=12
-
There are also pre-determined variables defined by the AMI Web system:
__USERNAME
: (String) the username of the logged in user__SESSIONID
: (String) a unique id for the web session__TIMEZONE
: (String) the timezone of the user's session__ADDRESS
: (String) the ip address of the user's browser__LOADTIME
: (Long) the time in milliseconds that the user started the sessionsession
: (Session) the session object representing the user's session
Hint: Through the front end in editor mode you can view all declared constants via: Menu Bar -> Dashboard -> Variable Table
Simplified Text Matching¶
Overview
The simplified text matching finds text entries using a simple pattern. When included in AMI script, it takes the form: var ~~ "my pattern"
Rule 1, the general case: the pattern supplied will match any text that contains the pattern, case insensitive. For example, assuming word is a variable let's consider the expression: word ~~ "AB"
. This will return true if word is cab or ABC or tabs but will return false if word is apple.
Rule 2, special symbols: If you need to look for symbols besides letters, numbers or white space, prefix with a back slash. For example, to look for the word P&L, the expression is word ~~ "P\\&L"
. Remember, we are in a string so a double back slash evaluates to a single backslash. Note this includes the following list of symbols: *^$.[]()\\|&!?:
Rule 3, Additional syntax: There are many special symbols that can be used for more advanced searches, for example the star (*) means anything, so "f*r"
will match any thing that has an f followed by r, such as 3FORGE. See the following sections for more advanced syntax.
Syntax The following syntaxes are used to do "basic" matching against expressions with some text.
-
Simple Matching Expression
Matches any text that contains the supplied pattern regardless of case. The following characters are reserved and may be embedded in the expression:
*
- Anything: Can be expanded to include any text.^
- Starts With: Symbolizes the "start" of the text. In other words, if an expression starts with^
, then the text must start with the following expression. Please note that carets (^
) not at the start of the expression will be treated as literals. (When used in conjunction with the tilde (~
), the hat should follow the tilde.)$
- Ends With: Symbolizes the "end" of the text. In other words, if an expression ends with a dollar sign($
) then the text must end with the preceding expression. Please note that dollar signs not at the end of the expression will be treated as literals..
- Any single char.: each dot will disregard one character and checks for the target character(s) in the remaining string, case INsensitive.
Examples:
.s
disregards the first character, checks for "s" OR "S" in the remaining string;s.
attempts to find "s" OR "S" in the string. If found, disregards the next character and returns a match. If not found, returns empty;..s
disregards the first TWO characters, checks for "s" OR "S" in the remaining string. Return a match if "s" OR "S" is found, empty otherwise;ss..
attempts to find "ss" OR either one of ("sS", "SS", "Ss") (must be consecutive "s") in the string. When found, disregards the next two characters and returns a match;..s.S..
disregards the first TWO characters, attempts to find one "s" OR "S", if found, disregard the next character, then attempts to find another "s" OR "S" in the remaining string, if found, disregards the next two characters and returns a match. If "s" was not found in either one of the two scenarios, return empty.- Note that the evluation order is always from left to right. Each character is checked at most once.
[charlist]
Any of Chars.: Match any one of the characters enclosed within the brackets.~
: Case Sensitive. Apply it before the first character. It indicates case sensitivity when matching the string.- Escaped chars: When referencing the following characters literally, they must be escaped with a back slash (
\
):\*\^\$.\[\]()\~\|&!?:
- Literals: These are simply letter-for-letter matches to the expression
-
'Exact Expression'
Matches the exact expression supplied in quotes (other than an escaped quote using \ ). This would be similar to using an equals ( == )
-
/regex/<options>/
This will use the java.util.regex package to match against regular expressions (see the java.util.regex.Pattern for details on building regular expressions). The
<options>
allow you to control behavior of how the regular expression operates. Simply supply the options you would like to apply in a continuous string between the 2nd and last forward slash (/). The following options are supported:- q - Canonical Equivalence: When this flag is specified, two characters will be considered to match if and only if their full canonical decompositions match. The expression "a\u030A", for example, will match the string "?" when this flag is specified. By default, matching does not take canonical equivalence into account.
- f - Full Line: When this flag is specified, the text must fully match the regular expression. If this option is not specified, any portion of the text matching the expression will be considered a match.
- i - Case Insensitive: By default, case-insensitive matching assumes that only characters in the US-ASCII charset are being matched. Unicode-aware case-insensitive matching can be enabled by specifying the u (Unicode case) flag in conjunction with this flag.
- c - Comments: White space is ignored, and embedded comments starting with # are ignored until the end of a line.
- l - Literal: White space is ignored, and embedded comments starting with # are ignored until the end of a line.
- d - Disable Dot All: If the disable dot all flag is set then the dot expression (.) will match any character except a line terminator. By default, the dot expression only matches line terminators.
- m - Multiline: In multiline mode the expressions ^ and $ match just after or just before, respectively, a line terminator or the end of the input sequence. By default, these expressions only match at the beginning and the end of the entire input sequence.
- u - unicode Case: When this flag is specified, case-insensitive matching, enabled by the i (case insensitive) flag, is done in a manner consistent with the Unicode Standard. By default, case-insensitive matching assumes that only characters in the US-ASCII charset are being matched.
- x - Unix Lines: In this mode, only the '\n' line terminator is recognized in the behavior of ., ^, and $.
- v - Not Matching: Reverses the matching result. Expressions that normally would match will not match and expressions that would not normally match will match.
- n - No Error: If the regular expression is invalid do not throw an exception, instead just always return false, unless the v flag is also set, in which case always return true.
-
<null>
Null String - providing null will result in a matcher which matches nothing.
-
<Empty String>
Providing an empty (zero length) string to the Matcher method will result in a matcher which matches only the empty string.
Conjugated Syntaxes
The following syntaxes allow for more complicated pattern matching. Nested syntaxes may also be conjugated syntaxes. For nested syntaxes which are basic syntaxes open parenthesis [ ( ] and close parenthesis [ ) ] must be escaped with a backslash [ \ ].
-
(Expression)
: The parenthesis can be used to force order of operations, important when mixing or [ | ] and and [ & ] clauses. -
Expression1\|Expression2
: If either of the nested expressions match this expression will return true. -
Expression1&Expression2
: If both of the nested expressions match this expression will return true. -
!Expression
: Inverts the result of the expression. -
(Expression) ? (ExpressionTrue) : (ExpressionFalse)
: If the Expression matches, return result of ExpressionTrue Otherwise return result of ExpressionFalse.
Sample Expressions
Expression | Examples Match | Examples Mismatch | Comments |
---|---|---|---|
FORG |
3Forge |
Frg |
Partial text, Case insensitive match |
~FORG |
FORGE |
forge |
Partial text, Case sensitive match |
^FORG$ |
FORG |
FORGE |
Exact text, Case insensitive match |
~^Forge$ |
Forge |
forge |
Exact text, Case sensitive match |
!FORG |
3FOR |
3FORGE |
Escape partial text, case insensitive |
!^FORG$ |
FORGE |
FORG |
Escape exact text, case insensitive |
[RB]ob |
Bobby |
lob |
Using char list |
[A-Z]ob |
mob |
7ob |
Char list range |
3F*ge |
3Forge llc |
Forge llc |
Wild chars |
3f.ge |
3Frge |
3Forge llc |
Match single char |
Acme llc. |
Acme llc. |
Acme llcc |
Escaping special chars |
Rob|bob|dan |
robert |
ted |
Or clause partial match |
^Rob$|^Dan$ |
rob |
robert |
Or clause, exact match |
bob&rob |
robob |
bobob |
And clause partial |
'Exact' |
Exact |
exact |
Exact case sensitive text match |
a |
Empty string, only matches empty string | ||
a?bb:s |
Abba |
acca |
''If contains 'a', must contain 'bb' |
a?bb:s |
test |
tent |
''If doesn't contain 'a', must contain 's' |
(a|b)&(e|f) |
af |
ab |
Must contain a or b. Also must contain e or f |
'abc'|'def' |
def |
DEF |
Must be exactly 'abc' or 'def' |
/reg/ |
regular |
Regular |
By default, regex must be case sensitive |
/reg/i/ |
Regular |
whatever |
I option makes case insensitive |
/req//|3F |
3F |
3Forge |
Mixing regex and non regex |
/req//|3F |
reg |
Regular |
Mixing regex and non regex |
String Templating¶
Goals
String templating is used to ease the dynamic creation of strings. Some key places that benefit from string templating:
- HTML generation
- SQL generation
- Content-rich messages for the end user
- Debugging / Logging
Basic Usage
As usual, strings are denoted using double quotes at the beginning and end, ex: "...". Within a string, ${...} is recognized as a special operator. The text between the curly braces is interpreted as code to be executed. For example, the code can be a simple variable name, or a complex set of expressions.
If you wish to include a literal dollar sign followed by a curly bracket, the dollar sign must be escaped with a backslash.
To prevent against SQL (and other forms) of injections, ${...} wrapped with quotes ('), double quotes (") or back ticks (`) will cause the evaluated text to be escaped with a backslash (\) accordingly:
For example:
Finally, null
will evaluate to an empty string ""
.
Examples
Templated Flow Control¶
The ${...}
syntax can also be used to dictate if/how text should be included. This includes all flow control statements, including if-statements, for-loops, for-each-loops, and while-loops. All flow control statements are closed by the ${}
token (empty curly brackets).
Flow Control Statement
Note: Definition is a multiline string for clarity
Single Line Flow Control Statement
Examples:
For if/else use the reserved ${else} token:
If/else Statements
Note: Definition is a multiline string for clarity.
Single Line If/else Statements
Example:
"Count the apples: ${if(5 < 3)} apple ${else} orange ${} There is one orange."
Flow control statements can be nested:
Nested Flow Control Statements
Note: Definition is a multiline string for clarity.
Single Line Nested Flow Control Statements
Example:
Examples
Syntax for Embedding External Languages¶
Goals
- Allow users familiar with other languages to embed familiar code into existing data models.
- Enable users to take advantage of existing libraries found in other languages
Syntax
Rules on Prefixing and terminating the custom code segment:
- The prefix must be at least 2 consecutive opening curly brackets ({).
- The termination must have the same number of closing curly brackets (}) as the prefix has.
- You can not include the suffix inside the embedded code. So for example, let's say you have a }} inside the embedded code, then you should wrap the embedded code in three curly brackets, ex: {{{custom_code_witha_double_curlys}}}
- The embedded code is evaluated literally, ex: all escapes, etc. are sent directly to the sub-language engine
Currently, python is supported out of the box.
Example
Formatting Timestamps¶
3forge provides two ways of handling timestamps when it comes to formatting:
- User Settings: If you like to maintain a unified date/time format across the dashboard, the formatting can be configured via user settings. Simply navigate to Account -> My Settings
- AMI Script Methods: If you like to have different formats in different places in the dashboard, AMI Script provides a handful of methods that can be utilized. To know more about their usage you can navigate to Help -> AMI Script Doc and the relevant methods will be available under AMI-Script Functions.
An example is provided below where an AMI Script Method has been used to format a table column.
Additionally, You can set the default timezone for all users by setting the property in local.properties:
AMI Reserved WORDS (alphabetical)¶
ADD ALTER ANALYZE AND AS ASC BEFORE BINARY BOOLEAN BREAK BY CALL CASE CATCH CREATE CONCURRENT CONTINUE DEFAULT DELETE DESC DESCRIBE DISABLE DO DOUBLE DROP ELSE ENABLE EXECUTE EXTERN FLOAT FOR FROM GROUP HAVING IF IN INDEX INSERT INT INTEGER INTO JOIN LEFT LIMIT LONG MODIFY NEAREST NEW OFTYPE ON ONLY OR ORDER PARTITION PREPARE PRIORITY PROCEDURE PUBLIC RENAME RETURN RIGHT SELECT SET SHOW STEP STRING SWITCH SYNC TABLE TABLES TIMER TRIGGER TO TRUNCATE UNION UNPACK UPDATE USE UTC UTCN VALUES WHERE WHILE WINDOW
Reserved words are case insensitive