June 2nd 2020: closed for reference in future projects.

Discussion: desired or possibly useful PureScript language features, based my current UX and lessons learned using PureScript for this project.

Modules

• Extensible Modules (multiple non-overlapping definitions for the same module, multiple modules per file)
• Module Dependencies (quantify over module imports)
• Friend Modules (constrained contextual access to module exports in other modules)
• Typed Modules (quantify over module exports)
• Scoped Imports

Syntax

• Macros or Template PureScript
• S-Expression syntax
• Top-level definition stubs (type signatures without implementations that cannot be referenced at run-time / erased at compile time)
• Lawful abstraction stubs (type signatures with definitional equality type that can not be accompanied by a type definition)

Primitive Types

• Primitive type for working with foreign JSON objects.

Linting

• Automatic checking of unit test coverage according to lawful specification.

See also: Ideas which need proposals.

June 4th, 2020: closed for future revision and review.

Possible future project names.

concise

• I was hoping to e.g. follow The Flash comics for a project naming convention, perhaps somewhat arbitrarily.

flashpoint

• There appears to be existing projects and IT organizations with this name.

trajectory

• This could be a suitable name for the project that could be assigned following RELEASE-01?

xs

• A possible alternative suitable name for the project?

...

Steps to Resolution

e1f9080

• Create package of helper functions for repetition of PureScript parsers, based on e.g. range of number of occurrences and validation constraints on trailing input, with unit tests.
• Add as a separate project dependency.
• See purescript-parsing-repetition.

Follow-Up Issues

• Publish purescript-parsing-repetition?

Steps to Resolution

2102f7a

• Define a module, Text.Parsing.Linux.Audit, that exports a function entry :: Parser String Foreign, which parses raw Linux Auditing System log entries in JSON format.
• Implement a test module, Test.Text.Parsing.Linux.Audit, that exports a function suite :: Aff Unit, with an appropriate set of unit tests for the public exports of Text.Parsing.Linux.Audit.

Follow-Up Issues

• Define an entry point, Main.Text.Parsing.Linux.Audit, that reads a sample of raw Linux Auditing System entries from a log file passed as a process argument. It should write parsed log entries in JSON format to a file passed as a process argument.
• Define an entry point,Main.Control.FS.Linux.Audit, that acts as an event collector that continuously watches for changes to the input file passed as a process argument, applies SQLi risk mitigations to the properties of parsed log file entries in JSON format, and writes parsed/validated log entries in JSON format to a SQlite3 database (e.g. with a table for each type of log entry.
• Define an entry point Main.Control.{HTTP,TCP}.Linux.Audit that spawns an HTTP/HTTPS or TCP event collector that allows clients to forward raw Linux Auditing System entries, with a fresh authorization token for each new request (where authorization tokens are passed in the the body of the response to previous requests for each new request), storing the entries in a SQLite3 database.
• I'm not sure if it is possible to configure auditd to forward log entries to a UNIX domain socket, and would like to test this in as part of a follow-up issue, e.g. implementing an analogous event collector with an appropriate entry point.
• Conduct a 24hr test of parsing/validating forwarded Linux Auditing System log entries in JSON format, in a model production test scenario.

June 7th, 2020: closed, post-mortem will be applied to current lack of application-tier deployment guide for lessons learned for future project(s).

Referenced in Building & Installation

June 4th, 2020: closed for future revision and review.

Issue Tracking

• After RELEASE-01, should issue tracking policies & procedures be revised following scheduled monthly lessons learned assessments](8) for the project?

Issue Resolution

Proposed Policies & Procedures

Issue Expiration

• Issues should expire after 15-30 days.
• Expired issues should be closed regardless of resolution (but logged).
• Unresolved expired issues can be revised and re-opened as new issue.
• Unresolved issues should be audited as part of monthly lessons learned assessments.
• Unresolved issues may be revised and re-opened after proposing changes to the lessons learned repository.

Issue Branching

• Should a separate branch be created for each issue?
• Should resolved issues be paired with tagged commits (e.g. RELEASE-xx/TIER-xx/ISSUE-xx/ACTION-xx)?

...

Issue Repository

• _Should a separate repository of all tracked & resolved issues for the project be maintained (e.g. including a markdown file for the issue & comment history on GitHub)? _

Referenced in Building & Installation

June 7th, 2020: closed, post-mortem will be applied to current management of project unit tests & structure for lessons learned for future project(s).

Referenced in Building & Installation

June 4th, 2020: closed for revision at a later date.

Modules prefixed with:

• Data should only export type definitions and common type class instances for those definitions.
• Effect should only export functions that produce an Effect a, e.g. for a randomly-generated a.
• FFI should only export functions with a foreign import from corresponding *.jsfile.
• Text.Parsing should only export functions that produce a Parser a m b.
• ...

June 4th, 2020: pulp and bower appear to still be required to publish packages on Pursuit, including current project dependencies, so I am closing this issue for now.

According to purescript.org, the recommended build tool for PureScipt projects is now spago. This project currently uses pulp as its build tool, and bower for PureScript package management.

• I propose that a new branch be created to resolve this issue, to attempt to test & evaluate spago as an alternative & recommended choice of PureScript package manager and build tool for this project.

I would like to the project to be deployable as a self-contained binary executable in the future (or e.g. as self-contained binary executables for each tier).

• I encountered an issue using pkg with the versions of the software dependencies necessary to build and run the project shown above. Currently, the project is not intended to be deployed per se: rather, the focus is currently on implementing unit tests for the project that reflects and adheres to the project's goals, issue management plan, and risk management plan.

The project's unit tests can be run with pulp test after installing the project's dependencies.

That said, the application tier of the 3-tier application can currently be launched with pulp run or node (after running pulp build, e.g. pulp build --to tier2.js && node tier2.js).

• The application tier is intended to be deployable in RELEASE-01/TIER-02, the next scheduled milestone & deliverable for the project, which will be accompanied with a tagged commit according to project policies & procedures.

By RELEASE-01, self-contained binary executable(s) that can be used to deploy the 3-tier application for a variety of platforms & architectures should accompany the first scheduled release & set of deliverables for the project.

• In the future, my thought is that the data-access tier should be deployable as a web application in and of itself rather than serving as a logical abstraction over existing DBMS backends (i.e. SQLite3, etc.) & connections spawned within a running instance of e.g. the application tier.

June 5th, 2020: closed, superseded by #29.

June 4th, 2020: closed for future revision and review.

Lessons Learned Assessments

Proposed Policies

• Monthly lessons learned assessments?
• Monthly lessons learned audit reports ("post-mortem" analyses)?

Lessons Learned Mitigations

• ...

Lessons Learned Repository

• ...

Referenced in Issue Management

• Use e.g. purescript-heterogeneous to automatically convert heterogeneous record types to a homogenous record type with Foreign values, and hence a Foreign with an coercion, which can e.g. be applied to JSON.stringify for serialization.

Steps to Resolution

• Implement a test suite that tests the parsing/validation of randomly-generated flow record events passed as input in JSON format.

Follow-Up Issues

• Review the project's current approach to parsing/validation of foreign JSON objects and their properties in general (see: //issues/21).
• Review the project's current approach to marshalling/serialization of PureScript record types to foreign JSON objects in general (see: //issues/27). For example, research and evaluate the use of purescript-heterogenous for a more ideal general approach to marshalling/serialization of JSON objects.
• Define an entry point, Main.Control.{TCP,UDP}.NetFlowv9, that spawns a self-contained flow collector listening on a port passed as a process argument, which parses/validates incoming NetFlowv9 packets as flow records in Data.Flow record event format, storing forwarded events in e.g. a SQLite3 database file passed as an argument to the process.

Steps to Resolution

• This issue expired on May 31st, 2020 following the current issue management policies & procedures, and lessons learned, for the project.
• It will be superseded by a follow-up issue.

Summary

• Prior to closing this issue, I implemented unit tests for the parsing/validation of randomly-generated processed event data that can be forwarded to Tier 2, the current application tier for the 3-tier application (in a follow-up issue, I propose that the functionality of Tier 2 should be moved to Tier 3, and that Tier 2 should allow clients forward raw security event data in JSON format instead).

Contents

• Example
• Text.Parsing.Forward
• Data.Forward
• Data.Event
  • Proposed Changes

Example

Currently, clients can forward processed event data to the forwarding route(s) served by the application tier, employing a common JSON format for each type of forward-able event.

An instance of the application tier can launched with pulp run after building the project, with issues pending. Event forwarding requests can be executed against the application e.g. as follows.

POST /forward/traffic?{
  "eventCategory" : "IN",
  "eventType"     : "SUCCESS",
  "eventID"       : "16741",
  "sourceID"      : "9ad08560-ad6c-4b90-940e-e0c7d5908bc0",
  "sessionID"     : "1addae90-8497-4956-9f48-39686291e9b5",
  "destinationID" : "8de8ff2a-1c1e-45cd-8bad-52a1f2680068",
  "logID"         : "b814ae2a-f5e6-4f90-b75f-e2aa29778b96",
  "schemaID"      : "c26cbb30-6690-41f0-aebc-c22baef04047",
  "featureID"     : "3712ca2f-40b3-468b-a96d-043c8255baff",
  "instanceID"    : "e98568f2-b401-4ce1-a043-4acece96ffdc",
  "startTime"     : "1970-01-01T00:00:00.000Z",
  "duration"      : "441763200000",
  "endTime"       : "1984-01-01T00:00:00.000Z"
}

curl -G -k "https://localhost:3000/forward/traffic" --data-urlencode '{...}'

• In Control.Tier2, clients should be required to pass forwarded event data in the body ofPOST requests to the forwarding route(s) served by the application tier, i.e. instead of requiring clients to pass forwarded event data in the URL query string.
• Conduct a SQLi vulnerability scan (using sqlmap) against the forwarding route(s) currently served by the application tier, before & after resolving the previous proposed action. Produce a summary report or each type of audited application-layer event (events by source, duration by feature, etc.).
• Include the results of this vulnerability scan as part of the scheduled risk assessment for RELEASE-01/TIER-02.
• Parsing/validation should be applied to the HTTP request headers of incoming requests to forward events (e.g. verifying that the request method is POST, Content-Type is application/json, Content-Length is correct w.r.t. the body of the request, and so on).

Text.Parsing.Forward

Unit tests are currently implemented for the parsing/validation of each type of forwarded event: an appropriate sample of randomly-generated forward-able events in URL format are currently parsed/validated when running pulp test.

• The parsing/validation of forward-able events should be modified to accommodate the proposed changes to the event forwarding request format (or e.g. the body of the POST request should be appended as a query to the URL of the request, as a result of the parsing/validation of the raw HTTP request headers & body, without revising e.g.Text.Parsing.Forward{.event}.
• Should clients be able to pass event data in both the URL query string or the body of the POST request?

• Implement an additional test suite, where a sample of randomly-generated forward-able events are moreover are forwarded to a self-contained instance of the back-end web application as part of the test.

Currently, Data.Forward.Event is a data type modelling each type of forward-able event.

• If the parsing of each type of forward-able event is unambiguous, can the event forwarding API of the application be simplified to e.g. serve only a single forwarding route for all types of events?
  e.g. POST /forward (or POST/forward?{...}) instead of POST /forward/traffic, POST/forward/linux, ...?

Data.Forward

Currently, Data.Event is used to model the proposed common structure of forward-able events.

Data.Event

• Does the common Event data type require further revision?

Proposed Changes

type Feature a b = (EventCategory a => EventID b =>
  { eventCategory :: a
  , eventType     :: EventType
  , eventID       :: b
  })

Description: The feature extracted from an audited security event (e.g. flow record, Linux Auditing System, Windows Security, ...).

Data.Event.Identifier

type Identifier =
  { sourceID      :: SourceID
  , sessionID     :: SessionID
  , destinationID :: DestinationID
  , logID         :: LogID
  , schemaID      :: SchemaID
  }

Description: A 5-tuple uniquely identifying the occurrence of the event, analogous to / inspired by the 5-tuple (sIP, sPort, dIP, dPort, protocol) identifying a flow record event.

sourceID: A unique identifier assigned to e.g. the remote hosts of forwarding agents for forward-able events. Analogous to sIP.

• Should sourceID be derived using a unique identifier for the application instance as a UUIDv5 namespace UUID (e.g. analogous to a SilK sensor name)?
• Should an applicationID (or sensorID) in configurations settings passed in the command line arguments of the entry point defined to launch the application tier?

sessionID: A unique identifier assigned to e.g. forwarding agents on remote hosts. Analogous to sPort.

• Should sessionID be derived using sourceID as a UUIDv5 namespace UUID?

destinationID: A unique identifier assigned to the location that the forwarded event will be stored. Analogous to dIP.

• Should destinationID be signed / verifiable from the e.g. sourceID and sessionID for the event (and moreover the applicationID from which they were derived)?

logID: A unique identifier for the resource, e.g. a DBMS instance / namely SQLite3 database file(s), that will store the forwarded event. Analogous to dPort.

schemaID: A unique identifier used to determine e.g. the table in a SQLite3 database file where the forwarded event will be stored (and the schema for this table). Analogous to protocol.

• Should schemaID be signed / verifiable from an sourceID, sessionID, destinationID, logID (and applicationID) for the forwarded event?

Data.Event.Checksum

type Checksum =
  { featureID  :: FeatureID
  , instanceID :: InstanceID
  }

featureID: a unique identifier for the feature (3-tuple) extracted from the event.

instanceID: a unique identifier for raw event from which the processed event was derived.

• Should raw event data be included as type Content = ... (in a follow-up release)?

Data.Event.Period

type Period = 
  { startTime :: Date
  , duration  :: Int
  , endTime   :: Date
  }

Data.Event.EventType

** Extensibility **

ea0ac96

• Separate module for common Data.Event.EventType (success or failure).

0288da6

• Explicit type class in Data.Event.Class analogous to EventCategory and EventID.

• Explicit type parameter with EventType type class constraint.

Under Construction

June 7th, 2020: closed, post-mortem will be applied to current lack of data-access-tier deployment guide for lessons learned for future project(s).
Referenced in Building & Installation

Steps to resolution

• Add purescript-parsing-common
  • Add purescript-parsing-repetition
  • Add purescript-parsing-alphanumeric
  • Add purescript-parsing-validation
  • Add purescript-parsing-arguments
  • Add purescript-parsing-json
  • Add purescript-parsing-foreign
  • Add purescript-parsing-hexadecimal
  • Add purescript-parsing-ip
  • Add purescript-parsing-integer
  • Add purescript-parsing-number
  • Add purescript-parsing-uuid
  • Add purescript-parsing-date
  • Add purescript-parsing-xml
  • Add purescript-parsing-enum
  • Add purescript-parsing-sqlite3

Follow-Up Issues

• Add purescript-parsing-linux-audit

• Add purescript-parsing-flow-record

• Add purescript-parsing-netflowv9

• Add purescript-parsing-silk

• Add purescript-parsing-windows-security

• Add purescript-event-collector-linux

• Add purescript-event-collector-silk

• Add purescript-event-collector-windows

• Add purescript-event-collector-netflowv9

• Add purescript-event-collector-json

• Add purescript-event-collector-xml

• Add purescript-event-collector-csv

• Add purescript-ffi-date

• Add purescript-ffi-uuid

• Add purescript-ffi-https

• Add purescript-ffi-rsa

• Add purescript-ffi-sqlite3

• Add purescript-printing-sqlite3

• Add purescript-linq-{xml,json,sqlite3}

• Add purescript-random-{range,array,enum,record(?)}

• purescript-parsing-s-expression

• purescript-parsing-rson (Rust Object Notation)

Under Construction

Referenced in RELEASE-01/TIER-03

Under Construction

Referenced in RELEASE-01/TIER-03

June 5th, 2020: closed, superseded by #29.

June 5th, 2020: closed, superseded by #29.

June 7th, 2020: closed, post-mortem will be applied to current management of project risks for lessons learned for future project(s).

Referenced in Building & Installation

Steps to Resolution

• Tier 2, the application tier of the 3-tier web application, should allow clients to forward "raw" event data in JSON format, as opposed to the current functionality of allowing clients to forward processed event data in JSON format.
• Tier 3, the data-access tier of the 3-tier web application, should allow clients to forward processed event data in JSON format, i.e. adopting the current functionality of Tier 2 instead. I propose that Tier 3 should be deployable as an application service in and of itself.

Expired Sub-Issues

• Event Forwarding

Current Sub-Issues

• Linux Auditing System Events
• ...

June 7th, 2020: closed, post-mortem will be applied to current management of project goals for lessons learned for future project(s).

Referenced in Building & Installation

June 7th, 2020: closed, generic replication/failover when executing queries against DBMS backend(s) will be implemented in follow-up future project(s).

Referenced in RELEASE-01/TIER-03

Under Construction

Referenced in RELEASE-01/TIER-02