NIAS: NSIP Integration As A Service.
This is a Golang version of the NIAS open-source components. This version incorporates NAPLAN validation and post-processing functionality, as well as the generic functionality of NIAS.
NIAS is a suite of open-source components designed to enable as many different users as possible to quickly and easily solve issues of system integration using the Australian SIF Data Model for school education.
The product was developed by harnessing existing open source middleware components, including:
Over these components, two main modules have been built:
- The SIF Store & Forward (SSF) is an opinionated message queueing system, which ingests very large quantities of data and stores them for delivery to clients. XML messages on the system are assumed by default to be in SIF. The SSF service builds an education-standards aware REST interface on top of the NATS message queues, and provides a number of utility services to ease SIF-based integrations.
- The SIF Memory Store (SMS) is a database that builds its internal structures from the data it receives, using RefIds both as keys to access stored messages, and to map out a network graph for SIF objects.
The software also uses these components as architecture to support Test Administration Authorities' interaction with NAPLAN Online:
- napval validates NAPLAN registration records, in either SIF/XML or CSV format.
- naprr post-processes the NAPLAN results & reporting dataset, including generating local reports, and aligning Year 3 Writing results to the codeframe.
The code is open source, and is released through the NSIP Github repository.
This product delivers the following high level functions:
- Support for persistent and ordered queues of SIF messages, which can be reread multiple times.
- Support for asynchronous queues in both clients and servers.
- Support for format-agnostic messaging infrastructure.
- Support for data exchange through an event/subscribe model (in brokered environments)
5. Support for message validation. - Support for extracting arbitrary relations between object types within SIF (bypassing need to configure service path queries, and simplifying the query API for objects).
7. Support for extracting arbitrary relations between object types from different standards (allowing multiple data standards to coexist in an integration, referring to the same entities). - Support for privacy filtering in middleware (which releases object providers from having to do privacy filtering internally).
9. Support for simple and extensible interactive analytics.10. Support for the ODDESSA data dictionary as a service. - Support for data format conversions, including CSV to SIF, and SIF 2 to SIF 3.
This product only acts as middleware. It does not provide integration with the back ends of products (although this can be provided by combining NIAS with the SIF Framework). It is not intended to deliver business value to end consumers, or to compete with existing market offerings.
The product delivers only exemplar analytics, and the SIF team is not committing to developing analytics and queries for all product users. Users that do develop their own analytics and queries are encouraged to contribute these back as open source.
The product delivers only exemplar integrations between multiple standards (SIF/XML and IMS OneRoster/CSV), and the SIF team is not committing to developing standards integrations for all product users. Users that do develop their own standards integrations are encouraged to contribute these back as open source.
The product does not incorporate authentication or authorisation.
- The product is released with the SIF-AU 3.4 XSD schema, and validates against it. Other schemas can be used, but may require re-coding of some modules.
- go version 1.8+
go get github.com/pwaller/goupx
Manually unzip file directory in the zip go-nias
and put it in c:
Then run gonias.bat
file from the nias
subdirectory
Install golang. Making sure you have a working
$GOPATH
etc (common mistake is to skip the src/
directory after $GOPATH
)
In $GOPATH/src/github.com/nsip
do:
git clone https://github.com/nsip/nias2
./build.sh
Separate executables are run to process NAPLAN data; see NAPVAL readme, NAPRR readme
See also NAPRR readme, NAPLVAL readme
NIAS2 relies on the following infrastructure:
unit_test_files/
- Contains files used in unit/integration testing of the code. Currently restricted to CSV files input into the validation module.
build.sh
, build/
, release.sh
build.sh
is the script to build all NIAS2 executables for the various supported platforms.build_sms.sh
is the script to build the executables specific to the SSF and SMS. The builds for each platform are built inbuild/{PLATFORM}/nias/
.build_napval.sh
is the script to build NIAS2 executables speficic to NAPLAN registration for the various supported platforms. The builds for each platform are built inbuild/{PLATFORM}/napval/
.build_naprr.sh
is the script to build NIAS2 executables specific to NAPLAN results & reporting for the various supported platforms. The builds for each platform are built inbuild/{PLATFORM}/naprr/
.- The
release_naprr.sh
andrelease_napval.sh
script zips the builds at the top level of thebuild
folder, and creates a new release of the NIAS2 code on github, including the binary files specific to NAPLAN results & reporting and NAPLAN registration, respectively. - The
release_napval.sh
script zips the builds at the top level of thebuild
folder, and creates a new release of the NIAS2 code on github, including the binary files specific to both NAPLAN results & reporting and NAPLAN registration. - The supported platforms are:
- Mac OSX
- Windows 32 bit
- Windows 64 bit
- Linux 32 bit
- Linux 64 bit
bin/
Contains the scripts and batch files to start and stop running NIAS. These are copied into the builds for each platform:
gonias.sh
: launch NIAS (OSX, Linux)gonias.bat
: launch NIAS (Windows)stopnias.sh
: stop NIAS (OSX, Linux)stopnias.bat
: stop NIAS (Windows)
tools/
Contains utilities for managing NIAS2
release.go
creates a new release of the NIAS2 code on github.
app/
Contains the code to run executables within NIAS as single pieces of software, along with necessary configuration files, and test executables. The configuration fields are copied into the binary distributions of NIAS.
sms/
: the SIF Memory Store functionality of NIAS (ported from Ruby)napval/
: NAPLAN Registration records validationnaprr/
: NAPLAN Results and Reporting post-processingnapcomp/
: Comparison of students between NAPLAN Registration and NAPLAN Results and Reporting (included innaprr
distribution)
napval/
Code for NAPLAN Registration records validation
naprr/
Code for NAPLAN Results and Reporting post-processing
sms/
SIF Memory Store functionality of NIAS
xml/
Golang structs corresponding to SIF XML objects relevant to executables. Currently limited to NAPLAN-specific objects.
lib/
Library code shared between all executables:
config.go
: read configuration files (in toml format)encoding.go
: encode NIAS messagesnats.go
: create NATS connections and process chainsniasmessage.go
: message wrapper typesserver_connections.go
: standardised NATS server access moduleservice.go
: service interface to handle message requeststransactiontracker.go
: transaction status reporting structure