Skip to content
Architecture > Authentication

Data Filter

As part of AMI's custom Java plugin functionality, data filters are highly customizable Java plugins that can be used to set user permissions for data access in AMI applications.

Overview

The data filter plugin controls what a given user can access based on some parameters, e.g; region. This stops users from accessing data they are not entitled to.

Generally, the Web filter is deployed one of two ways:

  1. Filtering real-time data as rows are updated.
  2. Filtering the resulting table of a datamodel query.

Typically, a data filter is invoked after some internal authentication system which will assign system variables to a user, such as region. The instructions for writing a custom entitlements plugin can be found here.

Note

A data filter is initialized per session of AMI -- if multiple users are using the same AMI dashboard, each user will have a separate instance of the data filter applied to them.

Requirements

You will need to follow the general steps for setting up a custom plugin. Once your Java project has been set up, ensuring autocode.jar and out.jar are included in your build path; you will need to include the following factories in your project classes:

Java Interface

1
2
com.f1.ami.web.datafilter.AmiWebDataFilterPlugin 
com.f1.ami.web.datafilter.AmiWebDataFilterinstance

These factories contain all the methods to needed for creating custom code.

Properties

You will also need to include the completed plugin in your AMI local.properties:

1
ami.web.data.filter.plugin.class=fully_qualified_class_name

General Implementation

  1. Use the com.f1.ami.web.auth.AmiAuthenticator plugin to authenticate a user and return a set of variables that are assigned to the user's session. These variables will be used to set the flags for the data filter.

  2. This user-session is passed into the com.f1.ami.web.datafilter.AmiWebDataFilterPlugin which then returns a com.f1.ami.web.datafilter.AmiWebDataFilterinstance.

  3. As data is passed from the backend to the frontend, the user's AmiWebDataFilter is called first which determines whether to suppress the data.

Data filters can be applied to both real-time and static datamodel queries. The methods to implement are below.

Realtime

As data is streamed into AMI, individual records are transported to the frontend for display on a per-row basis.

  • Rows can be added using AmiWebDataFilter::evaluateNewRow(...)
  • Rows can be updated using AmiWebDataFilter::evaluateUpdateRow(...)

Deciding how rows are added or updated is determined by flags. The flags determines when the data filter is run and the output.

Visibility Flags

Flag Behavior
HIDE_ALWAYS The row is always hidden after the filter is run (the filter does not re-run)
SHOW_ALWAYS The row is always shown after filter is run (the filter does not re-run)
HIDE The row is hidden until updated, then the filter is re-run
SHOW The row is shown until updated, then the filter is re-run

Static Query Results

When the user invokes a query (generally via the EXECUTE command within a datamodel), a query object is constructed and sent to the backend for execution. Then, the backend responds with a table (or multiple tables) of data.

  • Query request is sent as a message via AmiWebDataFilter::evaluateQueryRequest(...).
  • Apply the logic for filtering the response/output from the datamodel with AmiWebDataFilter::evaluateQueryResponse(...)

Example

This example implements a simple data filter that queries a table in AMI when a user logs in and shows/hides rows depending on the permissions in that table. For more information on how to build a plugin see here.

AMI Center commands

First, launch AMI and create the following table for storing which "batches" a user gets to see:

1
2
CREATE PUBLIC TABLE entitlements(user String,batch String,permissions String) USE PersistEngine="FAST";
INSERT INTO entitlements VALUES ("demo", "demo_batch", "access");

Properties

In local.properties, add the following:

1
2
3
4
5
6
7
8
ami.web.data.filter.plugin.class=customdatafilter.SampleDataFilterPlugin
SampleDataFilter.entitlements.url=jdbc:amisql:localhost:3280?username=demo&password=demo123 # jdbc url to query
SampleDataFilter.entitlements.table=entitlements # jdbc table name to query
SampleDataFilter.entitlements.column.user=user # user column name in table above
SampleDataFilter.entitlements.column.key=batch # entitlement key column name in table above
SampleDataFilter.entitlements.column.permissions=permissions # entitlement permission column name in table above
SampleDataFilter.filter.columns=batch # column names on which to apply the filter
SampleDataFilter.debug=false # debug mode

AmiWebDataFilterPlugin

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
package customdatafilter;

import com.f1.ami.web.datafilter.AmiWebDataFilter;
import com.f1.ami.web.datafilter.AmiWebDataFilterPlugin;
import com.f1.ami.web.datafilter.AmiWebDataSession;
import com.f1.container.ContainerTools;
import com.f1.utils.PropertyController;
import com.f1.utils.SH;
import java.util.HashMap;

public class SampleDataFilterPlugin implements AmiWebDataFilterPlugin {
        public String url;
        public String table;
        public HashMap<String, String> columns = new HashMap<String, String>();
        public boolean enabled;
        public String[] filterColumns;
        public boolean debug;

        @Override
        public void init(ContainerTools tools, PropertyController props) {
                this.url = props.getRequired("SampleDataFilter.entitlements.url", String.class);
                this.table = props.getRequired("SampleDataFilter.entitlements.table", String.class);
                this.columns.put("user", props.getRequired("SampleDataFilter.entitlements.column.user", String.class));
                this.columns.put("key", props.getRequired("SampleDataFilter.entitlements.column.key", String.class));
                this.columns.put("permissions", props.getRequired("SampleDataFilter.entitlements.column.permissions", String.class));

                this.filterColumns = SH.split(",", props.getRequired("SampleDataFilter.filter.columns", String.class));
                this.debug = props.getOptional("SampleDataFilter.debug", false);
        }

        @Override
        public String getPluginId() {
                return "DATAFILTER_PLUGIN";
        }

        @Override
        public AmiWebDataFilter createDataFilter(AmiWebDataSession session) {
                return new SampleDataFilter(this, session);
        }
}

AmiWebDataFilter

The implementation of the filter log itself is contained in a separate Java class that implements the actual AmiWebDataFilter factory.

  • Realtime feeds are filtered using evaluateNewRow and evaluateUpdatedRow.
  • Datamodel quieres are filtered with evaluateQueryRequest and evaluateQueryResponse.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
package customdatafilter;

import com.f1.ami.web.AmiWebObject;
import com.f1.ami.web.datafilter.AmiWebDataFilter;
import com.f1.ami.web.datafilter.AmiWebDataFilterQuery;
import com.f1.ami.web.datafilter.AmiWebDataSession;
import com.f1.base.Column;
import com.f1.base.Row;
import com.f1.utils.structs.table.columnar.ColumnarTable;
import com.f1.utils.LH;
import java.sql.*;
import java.util.HashMap;
import java.util.Objects;
import java.util.logging.Logger;

public class SampleDataFilter implements AmiWebDataFilter {

        private static final Logger log = LH.get();
        private AmiWebDataSession userSession;
        private SampleDataFilterPlugin plugin;
        private HashMap <String, String> permissions = new HashMap<String, String>();

        public SampleDataFilter(SampleDataFilterPlugin plgn, AmiWebDataSession session) {
                this.userSession = session;
                this.plugin = plgn;
        }

        @Override
        public byte evaluateNewRow(AmiWebObject realtimeRow) {
                debugLog("New Realtime Row evaluated for ", userSession.getUsername());
                debugLog("Row: ", realtimeRow);
                debugLog("Permissions: ", permissions);
                for (String filterColumn: plugin.filterColumns) {
                        String key = (String) realtimeRow.getParam(filterColumn);
                        if (key == null) {
                                debugLog("Filter column not found: ", filterColumn);
                                continue;
                        }
                        debugLog("Filter column ", filterColumn, " found with value ", key, " which has been ", (permissions.containsKey(key) ? "SHOWN": "HIDDEN"));
                        return permissions.containsKey(key) ? SHOW_ALWAYS : HIDE_ALWAYS;
                }
                debugLog("No Filter column found, row shown");
                return SHOW_ALWAYS;
        }

        @Override
        public byte evaluateUpdatedRow(AmiWebObject realtimeRow, byte currentStatus) {
                debugLog("New Realtime Row evaluated for ", userSession.getUsername());
                debugLog("Row: ", realtimeRow);
                debugLog("Permissions: ", permissions);
                for (String filterColumn: plugin.filterColumns) {
                        String key = (String) realtimeRow.getParam(filterColumn);
                        if (key == null) {
                                debugLog("Filter column not found: ", filterColumn);
                                continue;
                        }
                        debugLog("Filter column ", filterColumn, " found with value ", key, " which has been ", (permissions.containsKey(key) ? "SHOWN": "HIDDEN"));
                        return permissions.containsKey(key) ? SHOW_ALWAYS : HIDE_ALWAYS;
                }
                debugLog("No Filter column found, row shown");
                return SHOW_ALWAYS;
        }

        @Override
        public AmiWebDataFilterQuery evaluateQueryRequest(AmiWebDataFilterQuery query) {
                return query;
        }

        @Override
        public void evaluateQueryResponse(AmiWebDataFilterQuery query, ColumnarTable table) {
                debugLog("New Query evaluated for ", userSession.getUsername());
                debugLog("Query: ", query);
                debugLog("Response Table: ", table);
                for (String filterColumn: plugin.filterColumns) {
                        Column keyColumn = table.getColumnsMap().get(filterColumn);
                        if (keyColumn == null) {
                                debugLog("Filter column ", filterColumn, " not found");
                                continue;
                        }
                        debugLog("Filter column ", filterColumn, " found, filtering rows");
                        for (int i = table.getSize() - 1; i >= 0; i--) {
                                Row row = table.getRow(i);
                                String key = row.getAt(keyColumn.getLocation(), String.class);
                                if (!permissions.containsKey(key)) {
                                        debugLog("Row hidden: ", row);
                                        table.removeRow(row);
                                } else {
                                        debugLog("Row shown: ", row);
                                }
                        }
                }
                debugLog("Query evaluation complete");
        }

        @Override
        public void onLogin() {
                debugLog("Login start for user: ", userSession.getUsername());
                try {
                        getPermissions();
                } catch (Exception e) {
                        LH.severe(log, "SampleDataFilter Error in method onLogin ", e);
                }
                debugLog("Login complete for user: ", userSession.getUsername(), " with permisions: ", permissions);
        }

        @Override
        public void onLogout() {
                return;
        }

        private void getPermissions() throws SQLException, ClassNotFoundException {
                Class.forName("com.f1.ami.amidb.jdbc.AmiDbJdbcDriver");
                final Connection conn = DriverManager.getConnection(plugin.url);
                debugLog("Established connection for permissions: ", conn);
                try {
                        String user = this.userSession.getUsername();
                        String query = "\"" + user.toLowerCase().replace("\\", "\\\\").replace("\"","\\\"") + "\"";
                        debugLog("Sending query: select * from ", plugin.table, " where strLower(", plugin.columns.get("user"), ") == ", query);
                        final ResultSet rs = conn.createStatement().executeQuery("select * from " + plugin.table + " where strLower(" + plugin.columns.get("user") + ") == " + query);
                        debugLog("Recieved response: ", rs);
                        while (rs.next()) {
                                String key = rs.getString(plugin.columns.get("key"));
                                String perm = rs.getString(plugin.columns.get("permissions"));
                                addKey(key, perm);
                        }
                        debugLog("Permissions: ", permissions);
                } catch (Exception e) {
                        LH.severe(log, "SampleDataFilter Error in method getPermissions ", e);
                } finally {
                        conn.close();
                }
                this.userSession.putVariable("permissions",permissions,HashMap.class);
        }

        private void addKey(String key, String permission) {
                if (permissions.containsKey(key) && Objects.equals("view", permission)) {
                        return;
                }
                permissions.put(key, permission);
        }

        private void debugLog(Object... data) {
                if (plugin.debug)
                        LH.info(log, data);
        }
}

Testing

To test this, create a new realtime table table in the center and insert some rows:

1
2
CREATE PUBLIC TABLE orders(batch String,sym String,qty Long,px Double);
INSERT INTO orders VALUES ("demo_batch", "MSFT", 100, 250), ("demo_batch", "AAPL", 100, 300), ("other_batch", "MSFT", 200, 240);

Now create a realtime table in the Web on the orders table. If you are logged in as demo then you should see the demo_batch values but no the other_batch values.

Blank WebDataFilter Java File

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
package com.f1.ami.web.datafilter;

import com.f1.ami.web.AmiWebObject;
import com.f1.ami.web.datafilter.AmiWebDataFilter;
import com.f1.ami.web.datafilter.AmiWebDataFilterQuery;
import com.f1.ami.web.datafilter.AmiWebDataSession;
import com.f1.base.Column;
import com.f1.base.Row;
import com.f1.utils.structs.table.columnar.ColumnarTable;

public interface AmiWebDataFilter {
    public static final byte HIDE_ALWAYS = 1;
    public static final byte SHOW_ALWAYS = 2;
    public static final byte HIDE = 3;
    public static final byte SHOW = 4;

    /**
     * Called the user logs
     * 
     */
    public void onLogin();

    /**
     * Called when the user logs out
     * 
     */
    public void onLogout();

    /**
     * Called when a new row is received from the Center. Note that during login, all rows for display will pass through this method
     * 
     * @param realtimeRow
     *            the row to filter
     * @return {@link #HIDE_ALWAYS} - The row will be hidden from the user, regardless of any subsequent updates to the row's data (less overhead)<BR>
     *         {@link #HIDE} - The row will be hidden from the user, but future updates to the row's data will be re-evaluated via
     *         {@link #evaluateUpdatedRow(AmiWebObject) }(greater overhead)<BR>
     *         {@link #SHOW_ALWAYS} - The row will be visible to the user, regardless of any subsequent updates to the row's data (less overhead)<BR>
     *         {@link #SHOW} - The row will be visible the user, but future updates to the row's data will be re-evaluated via {@link #evaluateUpdatedRow(AmiWebObject)} (greater
     *         overhead)<BR>
     * 
     */
    public byte evaluateNewRow(AmiWebObject realtimeRow);

    /**
     * Called when a new row is updated from the Center. Note that this is only called for realtimeRows whose prior evaluation either returned {@link #HIDE} or {@link #SHOW}
     * 
     * @param realtimeRow
     *            the row to filter
     * @param currentStatus
     *            Either {@link #HIDE} or {@link #SHOW}
     * @return {@link #HIDE_ALWAYS} - The row will be hidden from the user, regardless of any subsequent updates to the row's data (less overhead)<BR>
     *         {@link #HIDE} - The row will be hidden from the user, but future updates to the row's data will be re-evaluated via {@link #evaluateUpdatedRow(AmiWebObject)}(greater
     *         overhead)<BR>
     *         {@link #SHOW_ALWAYS} - The row will be visible to the user, regardless of any subsequent updates to the row's data (less overhead)<BR>
     *         {@link #SHOW} - The row will be visible the user, but future updates to the row's data will be re-evaluated via {@link #evaluateUpdatedRow(AmiWebObject)} (greater
     *         overhead)<BR>
     * 
     */
    public byte evaluateUpdatedRow(AmiWebObject realtimeRow, byte currentStatus);

    /**
     * This method is called after each EXECUTE completes. In the general case, the Rows of the Table of the will be evaluated and certain rows may be deleted.
     * 
     * @param query
     *            - The query passed to the backend.
     * @param table
     *            - the resulting table from the query
     */
    public void evaluateQueryResponse(AmiWebDataFilterQuery query, ColumnarTable table);

    /**
     * Called before the request is sent to the backend. If the query is permitted as is, simply return the query param. To reject the query return null. Or create a new
     * {@link AmiWebDataFilterQueryImpl} object and set the various parameters that should actually be executed
     * 
     * @param query
     *            the query that the user would like to run
     * @return the actually query to run, or null if the query should not be executed.
     */
    AmiWebDataFilterQuery evaluateQueryRequest(AmiWebDataFilterQuery query);

}