================= Language Concepts ================= .. contents:: Sections: Metrics FPL Concepts ------------------------------------ Metrics FPL is a programming language designed to analyze and manipulate Metrics and API data streams. .. highlight:: javascript :: function example() { search {from="-8h"} sContent("@tags","fpl-example-data") let {id, isx5, isprime, odd, even, divisors} = f("@fields") aggregate v=values(id) by divisors let num_of_ints = listcount(v) } stream demo_table=example() Basics ^^^^^^ Primitives """"""""""""""" .. glossary:: String Collection of alpha-numeric and other characters. Enclosed in rouble quotes. Int64 and Float64 Numeric data can be represented using either Integers or Floats. Boolean The conditional 'true' and 'false' is represented by the Boolean (bool). 'null' and 'undefined' Special values, such as 'null' and 'undefined' are also specified in the language. Array and Maps (of primitives) Basic data structures, such as arrays and maps are supported. Object Types """""""""""" .. glossary:: Lambda A block of code, function, or expression that may or may not have a declared name. A lambda can be a simple expression: ``(x, y) ⇒ x+y`` Or a block of statements: ``(x, y) ⇒ { let a = x + 1 return a + y }`` Note: an explicit 'return' is required in the above case FPLTable A table with Rows and Columns, used in the majority of FPL scripts to pass and return data. FPLStream A data stream with defined ``from``, ``to``, ``interval``, and ``dimensions`` values, specifically catered to 'Metrics' streaming data. Tuples and Maps Similar to the above primitives, basic data structures of Objects are supported. Terms ^^^^^ Task """" A 'task' is the largest unit of execution within FPL. it is comprised of one or more code blocks, which are themselves collections functions, methods, and commands, each of which may be on separate lines. In concept, this is analogous to a 'program' or 'script'. Each run or execution of the FPL is considered a new 'task'. The task history is recorded in the software for reference and review. Control Blocks """""""""""""" The logical flow of a code block can be controled using logical statements like ``to``, ``interval``, and ``dimensions``. The ``return `` command is also supported. :: function example() // pipes end or :: function example(){ // pipes } The stream block is self-contained. At the end of execution, the stream block will return the results of the inner FPL pipe commands in a 'table' of columns. To execute a stream block, use the following syntax: :: stream demo_table=example() This will execute the example stream block, and store the resulting table in "demo_table". The stream block can also accept arguments, as seen the following example: :: function example(from_date, to_date, magic_str) search {from=from_date, to=to_date} sContains("",magic_str) let event_source = f("@source") aggregate total_by_src=count() by event_source end stream demo_table1=example("20220501", "20220601", "test") Pipe """" The smallest unit of execution is a 'pipe'. In FPL, each pipe is defined on their own separate line. A collection of 'pipe' commands make up a 'stream' block, but 'pipe' commands can also be used outside of a block, within the FPL task itself. The input / output of a pipe can vary greatly, depending on the type and function. The details of each available pipe of command is found in the rest of this documnet. **Please visit the relevant sub-section(s) for detailed information on a particular command of the Fluency Processing Language.** Logic Flow ^^^^^^^^^^ The goal of any FPL is to transform a collection of log / resource data into a table format. During the transformation process, additional actions can be performed to process the data in various, and potentially more meaningful ways. Data Selection """""""""""""" The FPL task begins with data selection. The selection of data is based on an expression that searches the entire dataset. Often, it is done with the `search` command as the first line. `Search` selects all the data matching the 'query' criteria within a specified 'time' range. Data Extraction """"""""""""""" The data extraction step chooses and 'extracts' the desired data field from the selected data. Normally this is done via the `let` assignment command, used in onjunction with the field-extraction function `f()`. Notice that `f()` has parentheses. It is a function, not a command. It returns an object of the selected field(s). Data extraction transforms the fields of the data object into "variables" or "columns", in preparation for the table data transformation. Data Processing """"""""""""""" The data processing portion encompasses a wide range of actions that can manipulate / process the raw data to more useful forms. This is also the main difference between FPL and a simple query expression. An analytic capability here allows the user to performs complex calculations over the dataset. With the data in a "columns", following the data extraction step, `aggregation` functions, such as `sum`, `min/max`, `distinct`, and `stdev` can be performed. Advanced functions, such as table `join` are also available. Additional utility functions, such as `toString`, `len`, and `regexp` can also be used. Results and Graphs ^^^^^^^^^^^^^^^^^^ Running an FPL task produce one or more results tables. These tables can use used as-in, and exported as CSV files. Or they can be enhanced into graphical elements, such as Charts and Graphs, and even combined into published Reports. The default number of rows is set to ten (10). This can be changed with the `sort` command: :: search { from="-7d Page last updated: 2023 May 25