NOTE: tnt4j-streams version 2.x migrated to Java 11 and bumped SLF4J from 1.7.x to 2.x.

Latest Java 8 compliant tnt4j-streams version is 1.15.x.

NOTE: TNT4J-Streams version 1.13.0 migrated logger to log4j2. In case you have changed default TNT4J-Streams logger configuration using previous TNT4J-Streams versions, please check new logger configuration file log4j2.xml.

1. Log4j2 supports configuration defined in XML, JSON, YAML and properties formats (NOTE: Log4j2 XML and properties formats differs from ones used by Log4j12). Previous Log4j12 log4j.properties file defined configuration shall be migrated into one of these new configuration definition formats.
2. Log4j2 changed configuration file definition System property name from log4j.configuration to log4j2.configurationFile.
3. See Log4j 2.x migration guide and Log4j 2.x configuration reference for more details
4. Log4j2 used asynchronous logging techniques noticeably increases application performance comparing to Log4j12. See Log4j 2.x Asynchronous Logging Performance report as reference.

TNT4J Streams allows data streaming, parsing from various data sources into TNT4J event sinks.

• TNT4J-Streams can be run out of the box for a large set of data streaming without writing no additional code. All you need is to define your data format mapping to TNT4J event mapping in TNT4J-Streams configuration.

• Supports the following data sources:

  • File
  • Characters/bytes feed stream from file or over TCP/IP
  • HDFS
  • MQTT
  • HTTP
  • JMS
  • Apache Kafka (as Consumer and as Producer/Consumer interceptor)
  • Apache Flume
  • Logstash
  • WMQ (IBM MQ)
  • OS pipes
  • Zipped files (also applies for Hdfs)
  • Standard Java InputStream/Reader
  • JAX-RS service (JSON/XML)
  • JAX-WS service
  • System command
  • MS Excel document
  • Elastic Beats
  • FileSystem (JSR-203 compliant) provided files (accessing remote files over SCP/SSH, SFTP, etc.)
  • JDBC
  • Chronicle Queue
  • Artemis Broker or Producer/Consumer interceptor
  • Compressed binary data or input stream
  • Protobuf messages

• Files (including provided by HDFS and JSR-203 FileSystem) can be streamed:

  • as "whole at once" - when a stream starts, it reads file contents line by line meaning a single file line holds data of a single activity event. After file reading completes - the stream stops.
  • using file polling - when some application uses file to write data at runtime, stream waits for file changes. When file changes, changed (appended) lines are read by stream and interpreted as single line is single activity event. Stream stops only when application gets terminated or some critical runtime error occurs.

• Customized parser for Apache Access Logs.

• Customized IBM MQ Trace Events stream (and parser).

• Customized IBM MQ Error log parser (also supports JSON formatted logs).

• It can be integrated with:

  • Logstash
  • Apache Flume
  • Angulartics
  • AJAX
  • Node.js
  • Collectd
  • Nagios
  • Elastic Beats
  • AWS CloudWatch metrics over Kinesis FireHose
  • Prometheus Remote-Write over Snappy compressed protobuf messages

  just by applying configuration and without additional coding.

• Redirect streamed data from different TNT4J based producer APIs like tnt4j-stream-* - to be TNT4J based streams concentrator.

• Run TNT4J-Streams as system daemon service.

• Run TNT4J-Streams as HTTP POST requests consumer servlet.

• Select File->Import...->Maven->Existing Maven Projects
• Click 'Next'
• In 'Root directory' field select path of directory where you have downloaded (checked out from git) TNT4J-Streams project
• Click 'OK'
• Dialog fills in with project modules details
• Click 'Finish'

• As standalone application

  • write streams configuration file. See 'Streams configuration' chapter for more details
  • configure your loggers
  • use bin/tnt4j-streams.bat or bin/tnt4j-streams.sh to run standalone application

  NOTE: in case you are using Java 9 as your runtime JVM and getting java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException, add java command parameter --add-modules java.xml.bind to add JAXB classes to java classpath.

• As API integrated into your product

  • Use Maven dependency:

        <dependency>
            <groupId>com.jkoolcloud.tnt4j.streams</groupId>
            <artifactId>tnt4j-streams-core</artifactId>
            <version>2.3.0</version>
        </dependency>

  • Write streams configuration file. See 'Streams configuration' chapter for more details
  • use StreamsAgent.runFromAPI(new CfgStreamsBuilder().setConfig(configFileName)) in your code

• As system daemon service. See 'TNT4J-Streams as System Service configuration' for details how to run TNT4J-Streams as your system service.

• Utility operations:

  • to encrypt plain text password use bin/utils/su.cmd or bin/utils/su.sh, e.g:

    *nix

    ./bin/utils/su.sh -e somePass

    MS Windows

    \bin\utis\su.cmd -e somePass

    Output of this command is like this: ENCRYPTED PASSWORD: v5OmKh7o3bHWRibcb4bnLg==. Now you can paste your encrypted password value (e.g. v5OmKh7o3bHWRibcb4bnLg) into your stream configuration.

Mapping of streamed data to activity event fields are performed by parser. To map field value you have to define field tag in parser configuration:

• attributes:

  • name - defines activity event field name

  • locator - defines location of data value from streamed data

  • locator-type - defines type of attribute locator. Set of supported values:

    • StreamProp
    • Index
    • Label
    • Range
    • Cache
    • Activity
    • Expression
    • ParserProp
    • SystemProp
    • EnvVariable

  • format - defines format type or representation that value is expected to be in (e.g. binary, date/time string, decimal format, etc.) . Set of supported values:

    • base64Binary

    • hexBinary

    • string

    • any decimal or date-time format pattern, e.g. #####0.000. Can be defined multiple date-time patterns delimited using | symbol.

    • one of number type enumerators: integer/int, long, double, float, short, byte, biginteger/bigint/bint, bigdecimal/bigdec/bdec and any. any will resolve any possible numeric value out of provided string, e.g. string "30hj00" will result value 30. It also allows defining casting mode, by adding prefix to number type enumerator (except enumerator any):

      • default casting mode without prefix is EXACT: numeric casting without significant value loss. In case number can't be cast to target type, original value is kept.
      • ~ - API casting mode: numeric casting shall be performed using plain Java API and in some cases resulting significant value loss.
      • ^ - UP_BOUND casting mode: if value can't be cast to target type, the closest upper bound type shall be used to maintain value without significant loss. Upper bound sequences:
        • For floating point numbers: Float, Double, BigDecimal
        • For non-floating point numbers: Byte, Short, Integer, Long, Float, Double, BigInteger, BigDecimal

      For example:

      <field name="NumericCastExact" locaotr="number1" locator-type="Label" datatype="Number" format="long"/>
      <field name="NumericCastApi" locaotr="number1" locator-type="Label" datatype="Number" format="~long"/>
      <field name="NumericCastUpBound" locaotr="number1" locator-type="Label" datatype="Number" format="^int"/>

  • value - defines predefined (hardcoded) value of field

  • datatype - defines how to interpret field resolved value. Set of supported values:

    • String
    • Binary
    • Number
    • DateTime
    • Duration
    • Timestamp
    • Generic
    • AsInput

  • radix - for numeric values, defines the radix that the value is specified in (ignored if format is specified)

  • units - defines the units of value to be represented. Set of supported values:

    • Days
    • Hours
    • Minutes
    • Seconds
    • Milliseconds
    • Microseconds
    • Nanoseconds

  • timezone - defines the time zone that a date/time string is represented in (when not specified, date/time string is assumed to be in local time zone)

  • locale - defines locale for data formatter to use

  • required - indicates whether a non-null value resolution is required for this field. If present, it takes precedence over the parser property RequireDefault.

  • id - field identifier

  • unwrapSingleValue - flag indicating an item value shall be unwrapped if array or collection containing single item.

  • charset - defines a Java supported charset/encoding name (in string format) for field Raw binary data; used to convert between Unicode and a number of other character encodings. Default value is the running streams JVM default charset (in most cases UTF8).

  • emptyAsNull - flag indicating an "empty" parser resolved value, e.g.:

    • string ""
    • 0 size array/collection,
    • array/collection having all null or "" elements

    shall be treated as null value, and consequently not added to produced activity entity values set. In most cases, some initialized object having no "payload" - is "useless" same way as it is not defined at all (equals null). Default value - true.

• tags:

  • field-map - tag is used to perform manual mapping from streamed data value source to field value target.

Sample:

<parser name="TokenParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityTokenParser">
    <.../>
    <field name="StartTime" locator="1" format="dd MMM yyyy HH:mm:ss" locale="en-US"/>
    <field name="ServerIp" locator="2"/>
    <field name="ApplName" value="orders"/>
    <field name="Correlator" locator="3"/>
    <field name="UserName" locator="4"/>
    <field name="EventName" locator="5"/>
    <field name="EventType" locator="5">
        <field-map source="Order Placed" target="START"/>
        <field-map source="Order Received" target="RECEIVE"/>
        <field-map source="Order Processing" target="OPEN"/>
        <field-map source="Order Processed" target="SEND"/>
        <field-map source="Order Shipped" target="END"/>
    </field>
    <field name="MsgValue" locator="8"/>
</parser>

/**
 * Name of application associated with the activity.
 */
ApplName(String.class),

/**
 * Host name of server to associate with activity.
 */
ServerName(String.class),

/**
 * IP Address of server to associate with activity.
 */
ServerIp(String.class),

/**
 * Name to assign to activity entry. Examples are operation, method, API call, event, etc.
 */
EventName(String.class),

/**
 * Type of activity - value must match values in {@link com.jkoolcloud.tnt4j.core.OpType} enumeration.
 */
EventType(Enum.class),

/**
 * Status of activity - value must match values in {@link com.jkoolcloud.tnt4j.core.ActivityStatus} enumeration.
 */
EventStatus(Enum.class),

/**
 * Time action associated with activity started.
 */
StartTime(UsecTimestamp.class),

/**
 * Time action associated with activity ended.
 */
EndTime(UsecTimestamp.class),

/**
 * Elapsed time of the activity in microseconds.
 */
ElapsedTime(Long.class),

/**
 * Identifier of process where activity event has occurred.
 */
ProcessId(Integer.class),

/**
 * Identifier of thread where activity event has occurred.
 */
ThreadId(Integer.class),

/**
 * Indicates completion status of the activity - value must match values in {@link com.jkoolcloud.tnt4j.core.OpCompCode} enumeration.
 */
CompCode(Enum.class),

/**
 * Numeric reason/error code associated with the activity.
 */
ReasonCode(Integer.class),

/**
 * Error/exception message associated with the activity.
 */
Exception(String.class),

/**
 * Indicates completion status of the activity - value can either be label from {@link com.jkoolcloud.tnt4j.core.OpLevel} enumeration 
 * or a numeric value.
 */
Severity(Enum.class),

/**
 * Location that activity occurred at.
 */
Location(String.class),

/**
 * Identifier used to correlate/relate activity entries to group them into logical entities.
 */
Correlator(String[].class),

/**
 * User-defined label to associate with the activity, generally for locating activity.
 */
Tag(String[].class),

/**
 * Name of user associated with the activity.
 */
UserName(String.class),

/**
 * Name of resource associated with the activity.
 */
ResourceName(String.class),

/**
 * User data to associate with the activity.
 */
Message(String.class),

/**
 * Identifier used to uniquely identify the data associated with this activity.
 */
TrackingId(String.class),

/**
 * Length of activity event message data.
 */
MsgLength(Integer.class),

/**
 * MIME type of activity event message data.
 */
MsgMimeType(String.class),

/**
 * Encoding of activity event message data.
 */
MsgEncoding(String.class),

/**
 * CharSet of activity event message data.
 */
MsgCharSet(String.class),

/**
 * Type of log entry.
 */
LogType(String.class),

/**
 * Age of activity event message.
 */
MessageAge(Long.class),

/**
 * Activity event category name.
 */
Category(String.class),

/**
 * Identifier used to uniquely identify parent activity associated with this activity.
 */
ParentId(String.class),

/**
 * Identifier used to globally identify the data associated with this activity.
 */
Guid(String.class),

/**
 * Stream activity entity time-to-live attribute.
 */
TTL(Long.class);

NOTE: EventType field is mandatory and can't have value null. If this field value resolves to null then streams automatically sets value to EVENT.

NOTE: Custom fields values can be found as activity event properties:

Sample:

<field name="Transport" locator="ActivityTransport" locator-type="Label"/>
<field name="Topic" locator="TopicName" locator-type="Label"/>

/**
 * Indicates that raw data value is the value of a named property of the current stream.
 */
StreamProp(String.class),

/**
 * Indicates that raw data value is at a specified index location, offset, etc. This is a generic index/offset value whose 
 * interpretation is up to the specific parser applying the locator. It also can be index of RegEx group.
 */
Index(Integer.class),

/**
 * Indicates that raw data value is the value of a particular key or label. Examples of this are XPath expressions for XML elements, 
 * and where each element of a raw activity data string is a name/value pair. It also can be name of RegEx group.
 */
Label(String.class),

/**
 * Indicates that raw data value is the value of a specific regular expression match, for parsers that interpret the
 * raw activity data using a regular expression pattern defined as a sequence of repeating match patterns. Match
 * identifier can be group sequence number or name.
 *
 * @deprecated use {@link #Label} instead. 
 */
@Deprecated
REMatchId(String.class),

/**
 * Indicates that data value is the value from stream stored cache with specified cache entry key.
 */
Cache(String.class),

/**
 * Indicates that data value is the value from currently processed activity data entity with specified entity field name.
 */
Activity(String.class),

/**
 * Indicates that data value shall be evaluated by using provided expression for parser context data.
 */
Expression(String.class),

/**
 * Indicates that raw data value is the value of a named property of the current parser.
 */
ParserProp(String.class),

/**
 * Indicates that raw data value is the value of a named JVM System property.
 */
SystemProp(String.class),

/**
 * Indicates that raw data value is the value of a named OS Environment variable.
 */
EnvVariable(String.class);

NOTE: Index is default value and may be suppressed in field/locator definition:

this:

<field name="UserName" locator="4"/>

is same as:

<field name="UserName" locator="4" locator-type="Index"/>

NOTE: obsolete (yet backward compatible) locator types:

• REGroupName - now should be changed to Label
• REGroupNum - now should be changed to Index
• REMatchId - now should be changed to Label
• REMatchNum - now should be changed to Index

In stream parsers configuration you are allowed to use stacked parsers technique: it is when some field data parsed by one parser can be forwarded to another parser to make more detailed parsing: envelope-message approach.

To define stacked parser(s) you have to define parser-ref tag(s) under parser field or embedded-activity definition.

NOTE: embedded-activity is tag field alias, used to define set of locator resolved data transparent to parent activity, but useful to make separate set of related child activities.

NOTE: if upper level parser resolved data is incompatible with stacked parser, stacked parser is not applied to parse that data.

Sample:

<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="tnt-data-source.xsd">

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <.../>
    </parser>
    <parser name="AccessLogParserExt" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <.../>
    </parser>

    <parser name="SampleJMSParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJMSMessageParser">
        <.../>
        <field name="MsgBody" locator="ActivityData" locator-type="Label">
            <parser-ref name="AccessLogParserCommon"/>
        </field>
        <.../>
        <embedded-activity name="InternalActivity" locator="OtherActivityData" locator-type="Label">
            <parser-ref name="AccessLogParserExt" aggregation="Merge"/>
            <parser-ref name="AccessLogParserCommon" aggregation="Relate"/>
        </embedded-activity>
        <.../>
    </parser>

    <stream name="SampleJMStream" class="com.jkoolcloud.tnt4j.streams.inputs.JMSStream">
        <.../>
        <parser-ref name="SampleJMSParser"/>
    </stream>
</tnt-data-source>

See 'JMS text message' sample for full configuration definition.

In this sample stream named SampleJMStream has primary parser reference SampleJMSParser. It parses data received as JMS message ( envelope). Field MsgBody carries JMS message payload data (message). In this sample consider we are sending Apache Access log entry as JMS message payload. So to parse that Apache access log entry we use stacked parser named AccessLogParserCommon.

After processing one JMS message TNT4J activity event will contain fields mapped by both SampleJMSParser and AccessLogParserCommon in the end.

See Parser matching data or parsing context for parser reference configuration details.

• ^.fieldName - stacked parser Activity type locator prefix ^. can be used to access parent parser (one stacked parser was invoked from) produced activity entity field value, where field name is defined by locator token fieldName. E.g.:

  <tnt-data-source
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:noNamespaceSchemaLocation="tnt-data-source.xsd">
  
      <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
          <.../>
          <field name="ValueFromParent" locator="^.StaticValue" locator-type="Activity"/>
      </parser>
  
      <parser name="SampleJMSParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJMSMessageParser">
          <.../>
          <field name="StaticValue" value="SampleJMSParser_Value" transparent="true"/>
          <.../>
          <embedded-activity name="InternalActivity" locator="OtherActivityData" locator-type="Label">
              <parser-ref name="AccessLogParserCommon" aggregation="Relate"/>
          </embedded-activity>
          <.../>
      </parser>
      <.../>
  </tnt-data-source>

  AccessLogParserCommon uses field ValueFromParent locator ^.StaticValue to fill in value defined in SampleJMSParser parser field StaticValue. After parsing, field ValueFromParent will have value SampleJMSParser_Value.

• ^.child[groupName.chIndex].fieldName (short form ^.child[chIndex].fieldName or ^.child[groupName].fieldName) - stacked parser aggregating activity entities using Relate method, Activity type locators can resolve field values from these child entities using this relative locator. Child activity entities are grouped by stacked parser names within parent entity, thus it is possible to access child entity field values from different stacked parsers under same parent activity. Locator pattern tokens means this:

  • ^. - is locator prefix to access parent activity entity
  • child - is predefined value to resolve child entity
  • groupName - is activity children group name, or if this token is missing - then group name is current stacked parser name
  • childIndex - is child index in defined group, or if this token is missing - then child index is ordinal index of current stacked parser produced activity entity
  • fieldName - is child activity entity field name

• child[groupName.matchExpression].fieldName (can be prefixed ^. to access parent activity entity children) - resolves child activity entity matching expression matchExpression field fieldName value. Match expression can be field value comparison with variable or static value, like this: fieldName=${variable} e.g. guid=${txHash}

  Locator definition samples:

     <field name="mainReason" locator="^.child[0].reason" locator-type="Activity"/>
     <field name="mainReason" locator="^.child[OtherParser].reason" locator-type="Activity"/>
     <field name="mainReason" locator="^.child[InputsParser.0].reason" locator-type="Activity"/>
     <field name="mainReason" locator="${child[TransactionParser.guid=${txHash}].tokenName}" locator-type="Activity"/>

  Parsers sample:

      <parser name="tokenAttributesBySLP" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser" manualFieldsOrder="true">
          <.../>
          <field name="transactionType" locator="transactionType" locator-type="Label" datatype="AsInput"/>
          <.../> 
      </parser>
  
      <parser name="voutParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser" manualFieldsOrder="true">
          <field name="EventType" value="Snapshot"/> 
          <.../> 
          <field name="slpTxAttributes" locator="slpAttributes" locator-type="Label" transparent="true">
              <parser-ref name="tokenAttributesBySLP" aggregation="Merge"/>
          </field>
          <field name="transactionIndex" locator="n" datatype="Number" locator-type="Label"/>
          <field name="transactionType" locator="^.child[0].transactionType" locator-type="Activity"/> 
          <.../>
      </parser>
  
      <parser name="vinParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
          <field name="EventType" value="Snapshot"/>
          <.../>
          <field name="tinIndex" locator="^.child[voutParser.0].transactionIndex" locator-type="Activity"/>
          <.../>
      </parser>
  
      <parser name="TransactionParserEnhanced" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser" manualFieldsOrder="true">
          <field name="EventType" value="Event"/>
          <.../>
          <embedded-activity name="vin" locator="$.result.vin" locator-type="Label">
              <parser-ref name="vinParser" aggregation="Relate"/>
          </embedded-activity>
          <embedded-activity name="vout" locator="$.result.vout" locator-type="Label">
              <parser-ref name="voutParser" aggregation="Relate"/>
          </embedded-activity>
          <.../>
      </parser>

  Parent activity entity (event) is created by parser named TransactionParserEnhanced. This parser has two fields vin and vout resolving array of JSON maps and referring stacked parsers vinParser and voutParser and aggregating child activity entities (snapshots) using Relate method. voutParser itself has field slpTxAttributes referring stacked parser tokenAttributesBySLP appending field transactionType to voutParser parser created snapshot. Consider TransactionParserEnhanced passes different contents of maps into voutParser and first map (child) entry slpAttributes has value map entry transactionType, the rest - not. But we want all produced snapshots to have field transactionType and to have value defined in first child. Locator ^.child[0].transactionType allows us to access that wanted value from first child (in case current child entry is the first one, field value is simply remapped to itself). Meanwhile, parser vinParser can access voutParser produced snapshots field transactionIndex value since they both "are under" same parent parser TransactionParserEnhanced.

  When two or more stacked parsers produces related sets of activity entities (having same count of child entities), it may be useful to access corresponding (auto mapped by index) activity entity. In such case using child locator pattern ^.child[groupName].fieldName allows to perform such index auto mapping from current stacked parser children set to groupName defined group children set.

• ^.XPathExpression - ActivityXMLParser stacked parser field Label type locator prefix ^. can be used to access parent parser (one stacked parser was invoked from) parsed document/node value using XPath expression, defined by token XPathExpression. E.g.:

     <!-- parser handles XML having root tags <source> or <destination> -->
     <parser name="TransferSetParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
          <field name="LastModified" locator="/source/file/@last-modified" locator-type="Label" required="false"/>
          <.../>
          <!-- access some field values from parent context document -->
          <field name="SourceSystemName" locator="^./transaction/sourceAgent/@agent" locator-type="Label"/>
          <field name="DestinationSystemName" locator="^./transaction/destinationAgent/@agent" locator-type="Label"/>
          <field name="Correlator" locator="^./transaction/@ID" locator-type="Label" datatype="String" format="string"/>
          <.../>
      </parser>
  
      <!-- parser handles XML having root tag <item> -->
      <parser name="TransferSetItemParser" class="com.jkoolcloud.tnt4j.streams.parsers.MessageActivityXmlParser">
          <field name="ResultCode" locator="/item/status/@resultCode" locator-type="Label" datatype="Number" transparent="true"/>
          <.../>
          <!-- field passes '/item/source' or '/item/destination' node to be root for stacked parser -->
          <embedded-activity name="TS_ITEM" locator="/item/*[name(.) = 'source' or name(.) = 'destination']" locator-type="Label">
              <parser-ref name="TransferSetParser" aggregation="Relate"/>
          </embedded-activity>
          <.../>
      </parser>
  
     <!-- parser handles XML having root tag <transaction> -->
      <parser name="ProgressEventParser" class="com.jkoolcloud.tnt4j.streams.parsers.MessageActivityXmlParser">
          <field name="EventType" value="Event"/>
  
          <field name="UserName" locator="/transaction/originator/userID" locator-type="Label"/>
          <field name="Location" locator="/transaction/originator/hostName" locator-type="Label"/>
          <.../>
          <!-- field passes '/transaction/transferSet/item' node to be root for stacked parser -->
          <embedded-activity name="TRANSFERSET" locator="/transaction/transferSet/item" locator-type="Label">
              <parser-ref name="TransferSetItemParser" aggregation="Merge"/>
          </embedded-activity>
          <.../>
      </parser>

Stacked parsers sample configuration tag <parser-ref> has attribute aggregation. This attribute defines method of resolved activity data aggregation into parent activity. Attribute has two possible values:

• Merge - resolved activity entity fields are merged into parent activity. NOTE: Parent activity entity will contain all fields processed by all stacked parsers. This is default value when attribute aggregation definition is missing in configuration.

• Relate - resolved activity entities are collected as children of parent activity. As a result there will be one parent activity entity having collection of child activities resolved by stacked parsers. NOTE: this value has alias Join left for backward compatibility, but it is not recommended using it anymore and should be changed right away for existing configurations. Activity entities can have these relations:

  • ACTIVITY - can have any entity as child: ACTIVITY, EVENT, SNAPSHOT or DATASET
  • EVENT - can have only SNAPSHOT or DATASET as a child
  • SNAPSHOT/DATASET - can't have any child entities

  Be sure that each parser or sub-parser that creates an activity entity to be sent to jKool/AP Insight as a JSON record has a field line:

  <field name="EventType" value="vvvv"/>

  where vvvv is one of values: ACTIVITY, SNAPSHOT/DATASET or EVENT (EVENT also maps from values OTHER, CALL, START, STOP, OPEN, CLOSE, SEND, RECEIVE, INQUIRE, SET, BROWSE, ADD, UPDATE, REMOVE, CLEAR, DATAGRAM). NOTE: If none of the above values applies to your case, you may use NOOP value. Then, however, the activity entity JSON record will not be created and not sent to jKoolCloud, e.g.:

  <field name="EventType" value="NOOP"/>

• Relate_Flat - resolved activity entities are collected as children of root activity entity. It is useful when related activity entities hierarchy can become as tree structure, but because of EVENT and SNAPSHOT/DATASET hierarchy restrictions (see Relate definition), it can't be sent to JKool directly.

For a Relate and Relate_Flat type aggregations there is related stream output parameter SplitRelatives:

<stream name="SampleJMStream" class="com.jkoolcloud.tnt4j.streams.inputs.JMSStream">
    <property name="SplitRelatives" value="true"/>
    <.../>
    <parser-ref name="SampleJMSParser"/>

</stream>

or

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <java-object name="StreamOutput" class="com.jkoolcloud.tnt4j.streams.outputs.JKCloudActivityOutput">
        <property name="SplitRelatives" value="true"/>
    </java-object>
    <.../>
    <stream name="SampleJMStream" class="com.jkoolcloud.tnt4j.streams.inputs.JMSStream">
        <.../>
        <parser-ref name="SampleJMSParser"/>
        <reference name="StreamOutput"/>
    </stream>
</tnt-data-source>

It allows sending as many activity entities to jKool as there are child activity entities resolved by stacked parsers, merging those child activity entities data with parent activity entity data. E.g. when parser builds activity entities relations like this (4 entities in total):

ParentActivity
    + ChildEvent1
    + ChildEvent2
    + ChildEvent3

stream output will send such entities to jKool (3 entities in total):

Event1 having ChilEvent1 + ParentActivity data
Event2 having ChilEvent2 + ParentActivity data
Event3 having ChilEvent3 + ParentActivity data

This is useful when streamed data is aggregated in one data package, like JSON/XML data having some header values and array of payload entries (see mft-tracking sample XML's), but you need only those payload entries maintaining some header data contained values to be set to jKool. Or like in mft_fte sample, when MFT transaction progress event transferSet node having source and destination definitions, and you need to split them into separate events maintaining some transaction data values.

In streams configuration you can define field or locator resolved values transformations. In general, transformations performs resolved activity value post-processing before sending it to meshIQ: e.g., extracts file name from resolved activity file path.

To pass a resolved field/locator value to a transformation script/expression, use the predefined variable placeholder $fieldValue. You can also use parser defined field names as script/expression variables having the format ${FIELD_NAME} to access resolved activity entity fields like ${EventType} or ${Trace.HighResTime}. The referenced fields must be within the same parser definition.

To access transformation performing field/locator name/id - use predefined variable placeholder $fieldName.

To define transformations stream configuration token <field-transform> shall be used. Attributes:

• name - name of transformation (optional).
• lang - transformation script/expression language. Can be one of groovy, javascript or xpath. Default value - javascript.
• beanRef - transformation implementing bean reference.
• phase - defines activity data resolution phase, when transformation has to be applied. Can be one of raw, formatted or aggregated. raw and formatted values are applicable for field locator resolved value transformation, while aggregated is default for transformation having references to activity entity fields.

Token body is used to define transformation script/expression code.

Valid transformation configuration should define beanRef, or have script/expression code defined in token body data (<![CDATA[]]>).

To use TNT4J-Streams predefined functions namespace ts: shall be used.

Streams predefined custom XPath functions to be used in transformation expressions:

• ts:getFileName(filePath) - implemented by transformation bean com.jkoolcloud.tnt4j.streams.transform.FuncGetFileName. Retrieves file name from provided file path.
• ts:getObjectName(objectFQN, options) - implemented by transformation bean com.jkoolcloud.tnt4j.streams.transform.FuncGetObjectName. Retrieves desired object name from provided fully qualified object name. Function supported options:
  • resolution options: DEFAULT, BEFORE, AFTER, REPLACE, FULL. (Optional)
  • search symbols. (Optional)
  • replacement symbols. (Optional)
• ts:replaceRegex(text, regex, replacement) - implemented by transformation bean com.jkoolcloud.tnt4j.streams.transform.FuncReplaceRegex. Replaces regex matching segments within provided text with provided replacement.
• ts:findRegex(text, regex, group) - implemented by transformation bean com.jkoolcloud.tnt4j.streams.transform.FuncFindRegex. Finds regex matching segment within provided text.

You may also define your own customized XPath functions. To do this your API has to:

• implement interface javax.xml.xpath.XPathFunction
• register function by invoking com.jkoolcloud.tnt4j.streams.utils.StreamsXMLUtils.registerCustomFunction(functionName, function).

e.g.:

public class YourTransform implements XPathFunction {
    @Override
    public Object evaluate (List args) {
        // retrieve expression code provided arguments and make transformation here.
    }
}
...
    StreamsXMLUtils.registerCustomFunction("yourTransformation",new YourTransform());
    ...

then you can use it from stream configuration:

<field name="InvoiceFileFromFunction" locator="7">
    <field-transform name="fileNameF" lang="xpath">
        ts:yourTransformation($fieldValue, "arg2", "arg3")
    </field-transform>
</field>

NOTE: those functions can be also used in XPath expressions of ActivityXMLParser ( or MessageActivityXMLParser) field locators. For example:

<parser name="XMLParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
    <.../>
    <field name="MFT_SRC_FILE_NAME" locator="ts:getFileName(/transaction/transferSet/item/source/file)" locator-type="Label"/>
    <.../>
</parser>

Use samples of ts:getObjectName() (consider XPath /transaction/transferSet/item/source/queue points to queue named SomeQueueName@Agent1):

<parser name="XMLParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
    <.../>
    <!-- to get queue name without agent: 'SomeQueueName' -->
    <field name="MFT_SRC_QUEUE_NAME" locator="ts:getObjectName(/transaction/transferSet/item/source/queue)" locator-type="Label"/>
    <!-- to get queue name without agent (part before FQN delimiter): 'SomeQueueName'-->
    <field name="MFT_SRC_QUEUE_NAME" locator="ts:getObjectName(/transaction/transferSet/item/source/queue, 'BEFORE', '@')" locator-type="Label"/>
    <!-- to get agent name (part after FQN delimiter): 'Agent1'-->
    <field name="MFT_SRC_AGENT_NAME" locator="ts:getObjectName(/transaction/transferSet/item/source/queue, 'AFTER', '@')" locator-type="Label"/>
    <!-- to get queue fully qualified name with '@' replaced to '_': 'SomeQueueName_Agent1' -->
    <field name="MFT_SRC_QUEUE_FQN_REPLACED" locator="ts:getObjectName(/transaction/transferSet/item/source/queue, 'REPLACE', '@', '_')" locator-type="Label"/>
    <!-- to get queue fully qualified: 'SomeQueueName@Agent1'-->
    <field name="MFT_SRC_QUEUE_FQN" locator="ts:getObjectName(/transaction/transferSet/item/source/queue, 'FULL')" locator-type="Label"/>
    <.../>
</parser>

• Field value transformation

<field name="Field" separator=",">
    <field-transform name="toUpper" lang="groovy">
        $fieldValue.toUpperCase()
    </field-transform>
    <field-locator locator="loc1" locator-type="Label"/>
    <field-locator locator="loc2" locator-type="Label"/>
    <.../>
</field>

When transformation is defined for a field containing multiple locators, field value transformation is applied to all field locators aggregated field value.

Sample above states that field value combined from locators loc1 and loc2 and separated by , should be upper-cased. For example locator loc1 resolves value value1, locator loc2 resolves value vaLue2. Then field value before transformations is value1,vaLue2 and field value after transformation is VALUE1,VALUE2.

• Field locator value transformation

<field name="Field" separator=",">
    <field-locator locator="loc1" locator-type="Label">
        <field-transform name="toUpper" lang="groovy" phase="raw">
            $fieldValue.toUpperCase()
        </field-transform>
    </field-locator>
    <field-locator locator="loc2" locator-type="Label">
        <field-transform name="toUpperConcat" lang="groovy" phase="formatted">
            $fieldValue.toUpperCase() + "_transformed"
        </field-transform>
    </field-locator>
    <.../>
</field>

Sample above states that locator loc1 resolved value should be upper-cased and locator loc2 resolved value should be upper-cased and concatenated with string _transformed. Then those transformed values should be aggregated to field value separated by , symbol. For example locator loc1 resolved value value1, then after transformation value is changed to VALUE1. Locator loc2 resolved value value2, then after transformation value is changed to VALIUE2_transformed. Field aggregates those locators values to VALUE1,VALUE2_transformed.

• Field/locator value transformation using additional activity entity fields

<field name="EventName" locator="enLoc" locator-type="Label"/>
<field name="EventType" locator="etLoc" locator-type="Label"/>
<.../>
<field name="Field" value="activity">
<field-transform name="toUpperConcat" lang="groovy">
    $fieldValue.toUpperCase() + "=" + ${EventType} + " : " + ${EventName}
</field-transform>
<.../>
</field>

Sample above defines fields EventName and EventType resolving values from streamed activity data. Then field named Field uses transformation to build value from predefined initial value activity and fields EventName, EventType resolved values. For example field EventName resolved value Transcation_X and EventType resolved value is SEND. Then after transformation field Field value becomes as ACTIVITY=SEND:Transcation_X.

NOTE:

• locator defined transformations are applied before field value transformations.
• it is possible to define multiple transformations for same stream element (field/locator). In that case transformations are applied sequentially where input of applied transformation is output of previous transformation.
• it is allowed to combine field and locators transformations within same stream field scope setting transformations for both field and locators independently.
• when "short form" of field-locator configuration is used, transformation is bound to locator (because field-locator relation is 1:1 and resolved value is same for locator and for field). Example:

<field name="Correlator" locator="3">
    <field-transform name="concat">
        "corel_" + $fieldValue
    </field-transform>
</field>

• Groovy script/expression

<field name="ApplName" value="orders">
    <field-transform name="toUpper" lang="groovy">
        $fieldValue.toUpperCase().center(30)
    </field-transform>
</field>

This sample upper case resolved value and makes "centred" (padded and tailed with space symbols) 30 symbols length string.

• JavaScript script/expression

<field name="UserName" locator="4">
    <field-transform name="toUpper" lang="javascript">
        $fieldValue.toUpperCase()
    </field-transform>
</field>

This sample upper case resolved value.

Or (using default attribute lang value)

<field name="Correlator" locator="3">
    <field-transform name="concat">
        "corel_" + $fieldValue
    </field-transform>
</field>

This sample concatenates string corel_ and resolved value.

JavaScript/Groovy based transformations default import packages:

• com.jkoolcloud.tnt4j.core
• com.jkoolcloud.tnt4j.utils
• com.jkoolcloud.tnt4j.uuid
• com.jkoolcloud.tnt4j.streams.utils
• com.jkoolcloud.tnt4j.streams.fields
• org.apache.commons.lang3
• org.apache.commons.collections4

so you are allowed to directly use classes from those packages in script code, e.g.:

<field-transform lang="groovy" name="FieldSubstring10">
    <![CDATA[
        StringUtils.substring($fieldValue, 0, 10)
    ]]>
</field-transform>

• XPath expression

<field name="InvoiceFileFromFunction" locator="7">
    <field-transform name="fileNameF" lang="xpath">
        concat(ts:getFileName($fieldValue), "_cust")
    </field-transform>
</field>

This sample retrieves file name from resolved file path (contained in $fieldValue) and concatenates it with string _cust.

• Transformation bean

To make custom Java Bean based transformations your API should implement interface com.jkoolcloud.tnt4j.streams.transform.ValueTransformation<V, T>.

<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="tnt-data-source.xsd">
    <.../>
    <java-object name="getFileNameTransform" class="com.jkoolcloud.tnt4j.streams.transform.FuncGetFileName"/>
    <.../>
    <field name="InvoiceFileFromBean">
        <field-locator locator="7">
            <field-transform beanRef="getFileNameTransform"/>
        </field-locator>
    </field>
    <.../>

This sample shows how to invoke Java Bean defined transformation. Class com.jkoolcloud.tnt4j.streams.transform.FuncGetFileName implements custom XPath function to get file name from provided file path. Field/locator value mapping to function arguments is performed automatically and there is no need to define additional mapping.

• Building field value from other activity entity fields values using transformations

Sometimes field value may be some combination of other activity entity fields values. To achieve such goal, transformations comes in hand.

<field name="MQTrace_HighresTime" locator="MQGACF_ACTIVITY_TRACE.MQIAMO64_HIGHRES_TIME" locator-type="Label" datatype="Timestamp"
       units="Microseconds"/>
<field name="ElapsedTime" locator="MQGACF_ACTIVITY_TRACE.MQIAMO64_QMGR_OP_DURATION" locator-type="Label"
       datatype="Number" units="Microseconds"/>
<field name="StartTime" value="">
<field-transform lang="groovy" name="StartTimeTransform">
    ${MQTrace_HighresTime} / 1000
</field-transform>
</field>
<field name="EndTime" value="">
<field-transform lang="groovy" name="EndTimeTransform">
    ${MQTrace_HighresTime} + ${ElapsedTime}
</field-transform>
</field>

Sample shows how to make activity entity fields StartTime and EndTime values using IBM MQ trace resolved high resolution values (in microseconds) of operation start and elapsed times.

StartTime is evaluated using Groovy transformation making field MQTrace_HighresTime field value dividing by 1000 and such getting time value in milliseconds.

EndTime is evaluated using Groovy transformation adding MQTrace_HighresTime and ElapsedTime fields values having final value in microseconds.

NOTE: initial value for those fields is set to value="". But it is allowed to set there any default initial value or use value locator(s), if it may be useful for a transformation expression evaluation. In that case to access initial value use expression variable $fieldValue.

TODO

TNT4J-Streams allows you to dynamically define field/field-locator parameters. A dynamic reference variable placeholder is defined using ${XXXXX} format, where XXXXX is a name or identifier of another data source configuration entity within the same parser definition.

Defining dynamic field parameters sample:

<.../>
<parser name="CollectdStatsDataParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
<field name="EventType" value="SNAPSHOT"/>
<.../>
<field name="${FieldNameLoc}" locator="values" locator-type="Label" value-type="${ValueTypeLoc}" split="true">
    <field-locator id="FieldNameLoc" locator="dsnames" locator-type="Label"/>
    <field-locator id="ValueTypeLoc" locator="dstypes" locator-type="Label"/>
</field>
<.../>
</parser>
<.../>

Sample shows how to dynamically define field name and value-type parameters referencing values resolved by FieldNameLoc and ValueTypeLoc locators.

There is also field attribute split stating that if field/locator resolved value is array/collection, then it should make as many activity fields as there are array/collection items available. In this case if field name is static (or makes same name from dynamically resolved values), name gets appended with sequential numbering to make fields names unique.

Defining combined field parameters values:

<.../>
<parser name="AttributesParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
<field name="${FieldNameLoc}_attr" locator="/entry/*[2]/text()" locator-type="Label" split="true">
    <field-locator id="FieldNameLoc" locator="/entry/*[1]/text()" locator-type="Label"/>
</field>
</parser>
<.../>

In this sample field name value is combined from dynamic ${FieldNameLoc} and static _attr parts.

More complex sample involving field name resolution from map and using collection/array element index:

<field name="${ColumnNameLoc}.Column$index" locator="*.2" locator-type="Label" datatype="AsInput">
    <field-locator id="ColumnNameLoc" locator="ColumnNames" locator-type="Activity" datatype="AsInput"/>
</field>

Locator ColumnNames resolved map of e.g. some table column names, where map entry key string ColumnX (X stands for column index) and value is column name string. Consider locator *.2 resolves list/array of cell values for that table over RegEx. Dynamic field name definition ${ColumnNameLoc}.Column$index pre-fills token $index with cell index (from list/array), refers locator ColumnNameLoc resolved map, and finally picks map entry value having key ColumnX (Column0, Column1 and so on...).

Sample of using another field resolved value in locator definition:

<.../>
<parser name="TransferSetParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
<.../>
<field name="Direction" locator="name(//*[1])" locator-type="Label" transparent="true"/>

<field name="ResourceName" formattingPattern="{0}={1};Agent={2}">
    <!--resolves FILE or QUEUE-->
    <field-locator locator="name(//*/*)" locator-type="Label">
        <field-map-ref resource="MFT_MAPPINGS.Resource"/>
    </field-locator>
    <!--resolves file or queue name -->
    <field-locator locator="ts:getFileName(/${Direction}/file)" locator-type="Label" required="false"/>
    <field-locator locator="ts:getObjectName(/${Direction}/queue)" locator-type="Label" required="false"/>
    <!-- agent-->
    <field-locator locator="/transaction/${Direction}Agent/@agent" locator-type="Label" required="false"/>
</field>
</parser>
<.../>

Sample configuration defines parser field Direction resolving value e.g., source or destination. Then field ResourceName locators use this value when constructing actual XPath expression e.g., /transaction/${Direction}Agent/@agent to resolve value from XML data.

NOTE: when using field/field-locator attribute locator-type="Activity" define field name as locator value without ${}, e.g.:

<.../>
<parser name="TransferSetParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
<.../>
<field name="Checksum" locator="//*/checksum" locator-type="Label" required="false"/>
<field name="Correlator" locator="Checksum" locator-type="Activity" required="false"/>
<.../>
</parser>
<.../>

TNT4J-Streams provides temporary storage (e.g. cache) for a resolved activity fields values. It is useful when there are some related activities streamed and particular jKool prepared activity entity requires data values form previously streamed activities.

Sample streamed values caching configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="EventParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="EventType" value="EVENT"/>

        <field name="EventName" locator="$.event" locator-type="Label"/>
        <field name="LastEvent" locator="CachedEventName" locator-type="Cache"/>

        <field name="Correlator" locator="$.transaction" locator-type="Label"/>
        <field name="Transaction" locator="$.transaction" locator-type="Label"/>

        <field name="SecretValue" locator="$.secret" locator-type="Label"/>
        <field name="LastSecret" locator="SecretCache" locator-type="Cache"/>

        <field name="Message" locator="$.message" locator-type="Label"/>
        <field name="Resource" locator="$.resource" locator-type="Label"/>
    </parser>

    <cache>
        <property name="MaxSize" value="300"/>
        <property name="ExpireDuration" value="15"/>

        <entry id="CachedEventName">
            <key>EventName</key>
            <value>${EventName} in ${Transaction}</value>
        </entry>
        <entry id="SecretCache">
            <key>${Transaction}Secret</key>
            <value>${SecretValue}</value>
        </entry>
    </cache>

    <stream name="MultipleEvents" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="FileName" value="./tnt4j-streams-core/samples/cached-values/event*.json"/>

        <parser-ref name="EventParser"/>
    </stream>

</tnt-data-source>

Sample configuration defines stream MultipleEvents reading data from JSON files using filename mask event*.json and referencing parser EventParser. Stream defines two cache related properties MaxSize and ExpireDuration. Definitions of those properties can be found in chapter Stream cache related parameters.

Stream definition has cache section, defining stored cache entries. Entry key and value can be configured using static values (e.g., <key>EventName</key>), dynamic activity value references (e.g., <value>${SecretValue}</value>) referencing activity entity field name, or combined static and dynamic values (e.g., <value>${EventName} in ${Transaction}</value>). Using dynamic value references in <key> definition will result having multiple cache entries filled in with values resolved from streamed activity data.

Parser EventParser has two fields having locator-type="Cache". This means that values from those fields are resolved from stream cache entries referenced by locator="SecretCache" where locator value points to cache entry identifier. Having reference to cache entry, stream takes entry key definition and pre-fills it with current activity entity fields values making actual cache key value and thus mapping to particular cached value.

When release assemblies are built, samples are located in samples directory, e.g., build/tnt4j-streams-2.3.0/samples. To run particular sample:

• go to sample directory
• run run.bat or run.sh depending on your OS

For more detailed explanation of streams and parsers configuration and usage see chapter 'Configuring TNT4J-Streams' and JavaDocs.

This sample shows how to stream activity events (orders) data from single log file.

Sample files can be found in samples/single-log directory.

orders.log file contains set of order activity events. Single file line defines data of single order activity event.

NOTE: Records in this file are from year 2011, e.g. 12 Jul 2011, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="TokenParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityTokenParser">
        <property name="FieldDelim" value="|"/>
        <field name="StartTime" locator="1" format="dd MMM yyyy HH:mm:ss" locale="en-US"/>
        <field name="ServerIp" locator="2"/>
        <field name="ApplName" value="orders"/>
        <field name="Correlator" locator="3"/>
        <field name="UserName" locator="4"/>
        <field name="EventName" locator="5"/>
        <field name="EventType" locator="5">
            <field-map source="Order Placed" target="START"/>
            <field-map source="Order Received" target="RECEIVE"/>
            <field-map source="Order Processing" target="OPEN"/>
            <field-map source="Order Processed" target="SEND"/>
            <field-map source="Order Shipped" target="END"/>
        </field>
        <field name="MsgValue" locator="8"/>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="FileName" value="orders.log"/>
        <!--<property name="RangeToStream" value="1:"/>-->

        <parser-ref name="TokenParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that FileLineStream referencing TokenParser shall be used.

FileStream reads data from orders.log file.

TokenParser uses | symbol as fields delimiter and maps fields to TNT4J event fields using field index locator.

NOTE: StartTime fields defines format and locale to correctly parse field data string. EventType uses manual field string mapping to TNT4J event field value.

This sample shows how to stream activity events (orders) data from multiple log files using file name matching wildcard pattern.

Sample files can be found in samples/multiple-logs directory.

orders-in.log and orders-out.log files contains set of order activity events. Single file line defines data of single order activity event.

NOTE: Records in this file are from year 2011, e.g. 12 Jul 2011, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

Sample configuration and sample idea is same as 'Single Log file' with one single difference:

<property name="FileName" value="orders-*.log"/>

meaning that stream should process not one single file, but file set matching orders-*.log wildcard pattern.

This sample shows how to stream activity events (orders) data received over OS pipe from another application or OS command.

Sample files can be found in samples/piping-stream directory.

orders.log file contains set of order activity events. Single file line defines data of single order activity event.

NOTE: Records in this file are from year 2011, e.g. 12 Jul 2011, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

jk-pipe.bat or jk-pipe.sh files are wrappers to bin/tnt4j-streams executables to minimize parameters. All what you need is to pass file name of stream parsers configuration, e.g., parsers.xml

run.bat or run.sh files uses OS piping to run sample.

Sample parser configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="TokenParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityTokenParser">
        <property name="FieldDelim" value="|"/>
        <field name="StartTime" locator="1" format="dd MMM yyyy HH:mm:ss" locale="en-US"/>
        <field name="ServerIp" locator="2"/>
        <field name="ApplName" value="orders"/>
        <field name="Correlator" locator="3"/>
        <field name="UserName" locator="4"/>
        <field name="EventName" locator="5"/>
        <field name="EventType" locator="5">
            <field-map source="Order Placed" target="START"/>
            <field-map source="Order Received" target="RECEIVE"/>
            <field-map source="Order Processing" target="OPEN"/>
            <field-map source="Order Processed" target="SEND"/>
            <field-map source="Order Shipped" target="END"/>
        </field>
        <field name="MsgValue" locator="8"/>
    </parser>
</tnt-data-source>

For details on parser configuration see sample named 'Single Log file'.

This sample shows how to stream activity events (Apache access log records) data from zipped file entries.

Sample files can be found in samples/zip-stream directory.

sample.zip and sample.gz files contains set of compressed Apache access log files.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserExt" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>
        <property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+))?)|(-))"/>
        <property name="ConfRegexMapping" value="%*i=(.*?)"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
        <field name="ElapsedTime" locator="15" locator-type="Index" datatype="Number" format="#####0.000"
               locale="en-US" units="Seconds"/>
    </parser>

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>
        <property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+))?)|(-))"/>
        <property name="ConfRegexMapping" value="%*i=(.*?)"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
    </parser>

    <stream name="SampleZipFileStream" class="com.jkoolcloud.tnt4j.streams.inputs.ZipLineStream">
        <property name="FileName" value="./tnt4j-streams-core/samples/zip-stream/sample.zip"/>
        <!--<property name="FileName" value="./tnt4j-streams-core/samples/zip-stream/sample.zip!2/*.txt"/>-->
        <!--<property name="FileName" value="./tnt4j-streams-core/samples/zip-stream/sample.gz"/>-->
        <!--<property name="ArchType" value="GZIP"/>-->

        <parser-ref name="AccessLogParserExt"/>
        <parser-ref name="AccessLogParserCommon"/>
    </stream>
</tnt-data-source>

Stream configuration states that ZipLineStream referencing AccessLogParserExt and AccessLogParserCommon shall be used.

ZipLineStream reads all entries lines from sample.zip file.

To filter zip file entries use zip entry name wildcard pattern, e.g., sample.zip!2/*.txt. In this case stream will read just zipped files having extension txt from internal zip directory named 2 (from sub-directories also).

Details on AccessLogParserCommon (or ApacheAccessLogParser in general) can be found in samples section 'Apache Access log single file' and parsers configuration section 'Apache access log parser'.

This sample shows how to stream activity events (Apache access log records) data read from standard Java input.

Sample files can be found in samples/java-stream directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserExt" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>
        <property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+))?)|(-))"/>
        <property name="ConfRegexMapping" value="%*i=(.*?)"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
        <field name="ElapsedTime" locator="15" locator-type="Index" datatype="Number" format="#####0.000"
               locale="en-US" units="Seconds"/>
    </parser>

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>
        <property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+))?)|(-))"/>
        <property name="ConfRegexMapping" value="%*i=(.*?)"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
    </parser>

    <java-object name="SampleFileStream" class="java.io.FileInputStream">
        <param name="fileName" value="./tnt4j-streams-core/samples/zip-stream/sample.gz" type="java.lang.String"/>
    </java-object>
    <java-object name="SampleZipStream" class="java.util.zip.GZIPInputStream">
        <param name="stream" value="SampleFileStream" type="java.io.InputStream"/>
    </java-object>
    <!--java-object name="SampleFileReader" class="java.io.FileReader">
        <param name="fileName" value="./tnt4j-streams-core/samples/apache-access-single-log/access.log"
            type="java.lang.String"/>
    </java-object-->

    <stream name="SampleJavaInputStream" class="com.jkoolcloud.tnt4j.streams.inputs.JavaInputStream">
        <property name="HaltIfNoParser" value="false"/>

        <reference name="SampleZipStream"/>
        <!--reference name="SampleFileReader"/-->

        <reference name="AccessLogParserExt"/>
        <reference name="AccessLogParserCommon"/>
    </stream>
</tnt-data-source>

Stream configuration states that SampleJavaInputStream referencing AccessLogParserExt and AccessLogParserCommon shall be used.

SampleJavaInputStream reads all lines from SampleZipStream referring FileInputStream SampleFileStream which reads file sample.gz.

NOTE: that java-object/param@value value may be reference to configuration already defined object like SampleFileStream in this particular sample.

To use Reader as input you should uncomment configuration lines defining and referring SampleFileReader and comment out SampleZipStream reference.

Details on AccessLogParserCommon (or ApacheAccessLogParser in general) can be found in samples section 'Apache Access log single file' and parsers configuration section 'Apache access log parser'.

This sample shows how to stream Apache access log records as activity events from single log file.

Sample files can be found in samples/apache-access-single-log directory.

access.log is sample Apache access log file depicting some HTTP server activity.

NOTE: Records in this file are from year 2004, e.g. 07/Mar/2004, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserExt" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>
        <!--property name="Pattern"
                  value="^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] &quot;(((\S+) (.*?)( (\S+)|()))|(-))&quot; (\d{3}) (\d+|-)( (\S+)|$)"/-->
        <property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+)|()))|(-))"/>
        <property name="ConfRegexMapping" value="%*i=(.*?)"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
        <field name="ElapsedTime" locator="15" locator-type="Index" datatype="Number" format="#####0.000" locale="en-US" units="Seconds"/>

    </parser>

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>
        <property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+)|()))|(-))"/>
        <property name="ConfRegexMapping" value="%*i=(.*?)"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>

    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="access.log"/>
        <!--<property name="RangeToStream" value="1:"/>-->

        <parser-ref name="AccessLogParserExt"/>
        <parser-ref name="AccessLogParserCommon"/>
    </stream>
</tnt-data-source>

Stream configuration states that FileLineStream referencing AccessLogParserExt and AccessLogParserCommon shall be used. Note that multiple parsers can be used to parse stream entries data, meaning that activity event data will be made by first parser capable to parse entry data.

FileStream reads data from access.log file. HaltIfNoParser property states that stream should skip unparseable entries and don't stop if such situation occurs.

AccessLogParserCommon parser is dedicated to parse Apache access log entries made using default logging configuration.

LogPattern defines logger pattern used to log entries to log file. Using this property parser is capable to automatically build RegEx to parse log entry fields.

User is also allowed to manually define RegEx for log entry line using Pattern property.

ConfRegexMapping properties are used to allow user override default log pattern token-RegEx mappings and define those manually to improve automatically build entry line RegEx.

Activity event fields mapping is performed using locator identifying RegEx pattern group index.

AccessLogParserExt is differs from AccessLogParserCommon just by having one additional log token %D in LogPattern property.

So if for example half of log file was made using log pattern defined in AccessLogParserCommon parser LogPattern property and the second part using log pater defined in AccessLogParserExt parser LogPattern property - stream should be able to handle whole log file with no problems.

NOTE: StartTime fields defines format and locale to correctly parse field data string. CompCode uses manual field string mapping to TNT4J event field value.

Sample does all the same as Apache Access log single file, but uses named RegEx group mapping for a parser defined fields.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserExt" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>
        <property name="ConfRegexMapping"><![CDATA[%*r=(?<request>((?<method>\S+) (?<uri>.*?)( (?<version>\S+))?)|(-))]]></property>

        <field name="Location" locator="hostname" locator-type="Label"/>
        <field name="UserName" locator="user" locator-type="Label"/>
        <field name="StartTime" locator="time" locator-type="Label" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="method" locator-type="Label"/>
        <field name="ResourceName" locator="uri" locator-type="Label"/>
        <field name="CompCode" locator="status" locator-type="Label">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="status" locator-type="Label"/>
        <field name="MsgValue" locator="sizeClf" locator-type="Label"/>
        <field name="ElapsedTime" locator="reqTime" locator-type="Label" datatype="Number" format="#####0.000" locale="en-US"
               units="Seconds"/>
    </parser>

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>
        <property name="ConfRegexMapping"><![CDATA[%*r=(?<request>((?<method>\S+) (?<uri>.*?)( (?<version>\S+))?)|(-))]]></property>

        <field name="Location" locator="hostname" locator-type="Label"/>
        <field name="UserName" locator="user" locator-type="Label"/>
        <field name="StartTime" locator="time" locator-type="Label" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="method" locator-type="Label"/>
        <field name="ResourceName" locator="uri" locator-type="Label"/>
        <field name="CompCode" locator="status" locator-type="Label">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="status" locator-type="Label"/>
        <field name="MsgValue" locator="sizeClf" locator-type="Label"/>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="./tnt4j-streams-core/samples/apache-access-single-log/access.log"/>
        <property name="RestoreState" value="false"/>
        <!--<property name="RangeToStream" value="1:"/>-->

        <!--<property name="UseExecutors" value="true"/>-->
        <!--<property name="ExecutorThreadsQuantity" value="5"/>-->
        <!--<property name="ExecutorsTerminationTimeout" value="20"/>-->
        <!--<property name="ExecutorsBoundedModel" value="false"/>-->
        <!--<property name="ExecutorRejectedTaskOfferTimeout" value="20"/>-->

        <parser-ref name="AccessLogParserExt"/>
        <parser-ref name="AccessLogParserCommon"/>
    </stream>
</tnt-data-source>

See parsers configuration section 'Apache access log parser' for a default RegEx group names used by ApacheAccessLogParser.

This sample shows how to stream Apache access log records as activity events from multiple log files using file name matching wildcard pattern.

Sample files can be found in samples/apache-access-multi-log directory.

localhost_access_log.[DATE].txt is sample Apache access log files depicting some HTTP server activity.

NOTE: Records in this file are from year 2015 ranging from April until November, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

Sample configuration and sample idea is same as 'Apache Access log single file' with one single difference:

<property name="FileName" value="*_access_log.2015-*.txt"/>

meaning that stream should process not one single file, but file set matching *_access_log.2015-*.txt wildcard pattern.

This sample shows how to stream Apache access log records as activity events from file which is used for logging at runtime. File polling technique may be used for any text file. File rolling is also supported.

Sample files can be found in samples/log-file-polling directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
        <field name="ElapsedTime" locator="14" locator-type="Index" datatype="Number" format="#####0.000" locale="en-US" units="Seconds"/>

    </parser>

    <stream name="SampleFilePollingStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="FileName"
                  value="[PATH_TO_LOGS_REPOSITORY]/logs/localhost_access_log.*.txt"/>
        <property name="FilePolling" value="true"/>
        <property name="FileReadDelay" value="20"/>
        <property name="StartFromLatest" value="true"/>
        <parser-ref name="AccessLogParserCommon"/>
    </stream>
</tnt-data-source>

Stream configuration states that FilePollingStream referencing AccessLogParserCommon shall be used.

FileStream reads data from access.log file. HaltIfNoParser property states that stream should skip unparseable entries and don't stop if such situation occurs.

AccessLogParserCommon is same as in 'Apache Access log single file' sample, so refer it for more details.

FileName property defines that stream should watch for files matching localhost_access_log.*.txt wildcard pattern. This is needed to properly handle file rolling.

FilePolling property indicates that stream polls files for changes.

FileReadDelay property indicates that file changes are streamed every 20 seconds.

StartFromLatest property indicates that stream should start from the latest entry record in log file. Setting this property to false would stream all log entries starting from the oldest file matching wildcard pattern.

NOTE: Stream stops only when critical runtime error/exception occurs or application gets terminated.

These samples show how to read or poll HDFS files contents. Samples are very similar to 'Log file polling' or 'Apache Access log single file'. Difference is that specialized stream classes are used.

• Simple HDFS file streaming

Sample files can be found in tnt4j-streams/tnt4j-streams-hdfs/samples/hdfs-file-stream directory.

<stream name="SampleHdfsFileLineStream" class="com.jkoolcloud.tnt4j.streams.inputs.HdfsFileLineStream">
    <property name="FileName" value="hdfs://127.0.0.1:19000/log.txt*"/>
    <.../>
</stream>

To stream HDFS file lines HdfsFileLineStream shall be used. FileName is defined using URI starting hdfs://.

• HDFS file polling

Sample files can be found in tnt4j-streams/tnt4j-streams-hdfs/samples/hdfs-log-file-polling directory.

<stream name="SampleHdfsFilePollingStream" class="com.jkoolcloud.tnt4j.streams.inputs.HdfsFileLineStream">
    <property name="FileName"
              value="hdfs://[host]:[port]/[path]/logs/localhost_access_log.*.txt"/>
    <property name="FilePolling" value="true"/>
    <.../>
</stream>

To poll HDFS file HdfsFileLineStream shall be used with property FilePolling value set to true. FileName is defined using URI starting hdfs://.

NOTE: Stream stops only when critical runtime error/exception occurs or application gets terminated.

• Zipped HDFS file streaming

Sample files can be found in tnt4j-streams/tnt4j-streams-hdfs/samples/hdfs-zip-stream directory.

<stream name="SampleHdfsZipLineStream" class="com.jkoolcloud.tnt4j.streams.inputs.HdfsZipLineStream">
    <property name="FileName"
              value="hdfs://[host]:[port]/[path]/sample.zip!2/*.txt"/>
    <property name="ArchType" value="ZIP"/>
    <.../>
</stream>

To stream HDFS zipped file lines HdfsZipLineStream shall be used. FileName is defined using URI starting hdfs://.

This sample shows how to stream activity events from redirected Apache Flume output Raw data. Apache Flume output is configured to send Raw output data as JSON to localhost:9595. Sample also shows how to use stacked parsers technique to extract log entry data from JSON envelope.

Sample files can be found in samples/apache-flume directory.

How to install plugin into Apache Flume and configure, see samples/apache-flume/README.md

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
    </parser>

    <parser name="JSONEnvelopeParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOL"/>

        <field name="MsgBody" locator="$.body" locator-type="Label">
            <parser-ref name="AccessLogParserCommon"/>
        </field>
        <field name="sinkName" locator="$.sinkName" locator-type="Label"/>
        <field name="chanelName" locator="$.chanelName" locator-type="Label"/>
        <field name="headers" locator="$.headers" locator-type="Label"/>
    </parser>

    <stream name="SampleFlumeStream" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="9595"/>
        <parser-ref name="JSONEnvelopeParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that CharacterStream referencing JSONEnvelopeParser shall be used.

CharacterStream starts server socket on port defined using Port property. HaltIfNoParser property indicates that stream should skip unparseable entries.

JSONEnvelopeParser transforms received JSON data package to Map with entries MsgBody, sinkName, chanelName and headers. MsgBody entry value is passed to stacked parser named AccessLogParserCommon. ActivityDelim property indicates that every line in parsed string represents single JSON data package.

Details on AccessLogParserCommon (or ApacheAccessLogParser in general) can be found in samples section 'Apache Access log single file' and parsers configuration section 'Apache access log parser'.

NOTE: Stream stops only when critical runtime error/exception occurs or application gets terminated.

This sample shows how to stream activity events from Apache Flume parsed log entry output data. Apache Flume output is configured to send parsed log entry data as JSON to localhost:9595.

Sample files can be found in samples/apache-flume-parsed directory.

How to install plugin into Apache Flume and configure, see samples/apache-flume-parsed/README.md

messages.json file contains sample Apache Flume output JSON data package prepared using configuration of this sample. This sample JSON is for you to see and better understand parsers mappings. Do not use it as Apache Flume input!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="FlumeJSONParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOL"/>

        <field name="Location" locator="$.headers.clientip" locator-type="Label"/>
        <field name="UserName" locator="$.headers.auth" locator-type="Label"/>
        <field name="StartTime" locator="$.headers.logtime" locator-type="Label" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="$.headers.method" locator-type="Label"/>
        <field name="ResourceName" locator="$.headers.param" locator-type="Label"/>
        <field name="CompCode" locator="$.headers.response" locator-type="Label">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="$.headers.response" locator-type="Label"/>
        <field name="MsgValue" locator="$.headers.bytes" locator-type="Label"/>
        <field name="Message" locator="$.body" locator-type="Label"/>
        <field name="Tag" separator=",">
            <field-locator locator="$.sinkName" locator-type="Label"/>
            <field-locator locator="$.chanelName" locator-type="Label"/>
        </field>
    </parser>

    <stream name="SampleFlumeStream" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="Port" value="9595"/>
        <parser-ref name="FlumeJSONParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that CharacterStream referencing FlumeJSONParser shall be used.

CharacterStream starts server socket on port defined using Port property.

FlumeJSONParser transforms received JSON data package to Map entries. Note that some entries like headers in map has inner map as value. Fields of such entries can be accessed defining field name using . as field hierarchy separator. ActivityDelim property indicates that every line in parsed string represents single JSON data package.

Details on AccessLogParserCommon (or ApacheAccessLogParser in general) can be found in samples section 'Apache Access log single file' and parsers configuration section 'Apache access log parser'.

This sample shows how to stream activity events from redirected Logstash output Raw data. Logstash output is configured to send Raw output data as JSON to localhost:9595. Sample also shows how to use stacked parsers technique to extract log entry data from JSON envelope.

Sample files can be found in samples/logstash directory.

How to setup sample environment see samples/logstash/README.md

messages.json file contains sample Logstash output JSON data package prepared using configuration of this sample. This sample JSON is for you to see and better understand parsers mappings. Do not use it as Logstash input!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
    </parser>

    <parser name="JSONEnvelopeParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOL"/>

        <field name="MsgBody" locator="$.message" locator-type="Label">
            <parser-ref name="AccessLogParserCommon" tags="Normal server,Delayed server"/>
        </field>
        <field name="path" locator="$.path" locator-type="Label"/>
        <field name="Tag" locator="$.tags" locator-type="Label"/>
        <field name="host" locator="$.host" locator-type="Label"/>
    </parser>

    <stream name="SampleLogstashStream" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="RestartOnInputClose" value="true"/>
        <property name="Port" value="9595"/>
        <parser-ref name="JSONEnvelopeParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that CharacterStream referencing JSONEnvelopeParser shall be used.

CharacterStream starts server socket on port defined using Port property. HaltIfNoParser property indicates that stream should skip unparseable entries.

JSONEnvelopeParser transforms received JSON data package to Map with entries MsgBody, path, Tag and host. MsgBody entry value is passed to stacked parser named AccessLogParserCommon. ActivityDelim property indicates that every line in parsed string represents single JSON data package.

Details on AccessLogParserCommon (or ApacheAccessLogParser in general) can be found in samples section 'Apache Access log single file' and parsers configuration section 'Apache access log parser'.

NOTE: Stream stops only when critical runtime error/exception occurs or application gets terminated.

This sample shows how to stream activity events from parsed by Logstash. Logstash Grok output plugin is configured to send parsed Apache Access log entry data as JSON to localhost:9595.

Sample files can be found in samples/logstash-parsed directory.

How to setup sample environment see samples/logstash-parsed/README.md

messages.json file contains sample Logstash output JSON data package prepared using configuration of this sample. This sample JSON is for you to see and better understand parsers mappings. Do not use it as Logstash input!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="LogstashJSONParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOL"/>

        <field name="Location" locator="$.clientip" locator-type="Label"/>
        <field name="UserName" locator="$.auth" locator-type="Label"/>
        <field name="StartTime" locator="$.timestamp" locator-type="Label" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="$.verb" locator-type="Label"/>
        <field name="ResourceName" locator="$.request" locator-type="Label"/>
        <field name="CompCode" locator="$.response" locator-type="Label">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="$.response" locator-type="Label"/>
        <field name="MsgValue" locator="$.bytes" locator-type="Label"/>
        <field name="Message" locator="$.message" locator-type="Label"/>
        <field name="Tag" locator="$.tags" locator-type="Label"/>
    </parser>

    <stream name="SampleLogstashStream" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="Port" value="9595"/>
        <parser-ref name="LogstashJSONParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that CharacterStream referencing LogstashJSONParser shall be used.

CharacterStream starts server socket on port defined using Port property.

LogstashJSONParser transforms the received JSON data package to a Map data structure and maps map entries to activity event fields using map entry key labels. The ActivityDelim property with value EOL indicates that every line in the parsed stream represents a single JSON data package.

This sample shows how to stream activity events from Elastic Beats. NOTE: Elastic Beats environment has to be configured to output data to stream started Logstash server host and port, e.g.localhost:5044.

Sample files can be found in samples/elastic-beats directory.

dashboards directory contains exported jKool dashboard dedicated to visualize data for this sample. You can import it into your jKool repository.

How to set up Elastic Beats environment see samples/elastic-beats/readme.md

sampleMsg.json file contains sample Elastic Beats provided Logstash message as JSON data prepared using configuration of this sample. This sample JSON is for you to see and better understand parsers mappings. Do not use it as Logstash input!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://raw.githubusercontent.com/Nastel/tnt4j-streams/master/config/tnt-data-source.xsd">

    <parser name="BeatsMessageParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="EventType" value="EVENT"/>

        <field name="StartTime">
            <field-locator locator="@timestamp" locator-type="Label" datatype="DateTime" format="yyyy-MM-dd'T'HH:mm:ss.SSSX">
            </field-locator>
        </field>

        <field name="EventName" formattingPattern="{0} {1}">
            <field-locator locator="metricset.module" locator-type="Label"/>
            <field-locator locator="metricset.name" locator-type="Label"/>
        </field>

        <field name="all" locator="*" locator-type="Label"/>
    </parser>

    <stream name="SampleBeatsStream" class="com.jkoolcloud.tnt4j.streams.inputs.ElasticBeatsStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="RestartOnInputClose" value="true"/>

        <!--Binding host-->
        <!--<property name="Host" value="localhost"/>-->
        <property name="Port" value="5044"/>
        <!--Worker properties-->
        <!--<property name="Timeout" value="30"/>-->
        <!--<property name="ThreadCount" value="1"/>-->

        <!--SSL properties-->
        <!--<property name="SSLCertificateFilePath" value="/etc/pki/client/cert.key"/>-->
        <!--<property name="SSLKeyFilePath" value="/etc/pki/client/cert.pem"/>-->
        <!--<property name="PassPhrase" value="pass"/>-->

        <parser-ref name="BeatsMessageParser"/>
    </stream>
</tnt-data-source>