Giter Club home page Giter Club logo

lrs's Introduction

com.yetanalytics/lrs

CI Clojars Project

Yet Analytics Core LRS Library. Includes protocols and functions used to implement a conformant xAPI Learning Record Store.

Usage

This project is set up to use the clj/clojure executable and deps.edn, with a handy Makefile.

To use the dev profile, which contains all dev/repl/test stuff, use the :dev alias: -A:dev.

Makefile targets of note:

  • clean - Clean all build/ephemeral files. Note that this includes node_modules and package.json but not package-lock.json.
  • node_modules - As a convenience, have cljs.main figure out the node deps in src/main/deps.cljs and upstream, and pull them. You'll want this before using something like CIDER.
  • repl - Run a Clojure repl with the :dev alias.
  • repl-cljs - Run a Clojurescript node repl.
  • run-dev - Start an in-memory implementation of the LRS in Clojure.
  • run-dev-cljs - Start an in-memory implementation of the LRS in ClojureScript.
  • test-lib-clj - Run the tests in Clojure.
  • test-lib-cljs - Run the tests in ClojureScript.
  • test-lib - Run the tests on both Clojure and ClojureScript.
  • test-conformance-clj-sync - Run ADL LRS Conformance Test Suite on a synchronous Clojure LRS on the JVM.
  • test-conformance-clj-async - Run ADL LRS Conformance Test Suite on an asynchronous Clojure LRS on the JVM.
  • test-conformance-cljs - Run ADL LRS Conformance Test Suite on an (always asynchronous) ClojureScript LRS on node.
  • test-conformance - Run all conformance tests.
  • test-all - Run all lib and conformance tests.

Implementation Notes

Async GET /statements Error Handling

LRS Applications implementing the com.yetanalytics.lrs.protocol/-get-statements-async method return a channel containing results. Since the body is streamed it is not possible to change the status of the request if an error is encountered. Applications can immediately terminate the stream (resulting in a malformed body) by passing :com.yetanalytics.lrs.protocol/async-error to the channel. This is preferable to returning a structurally valid response that is missing data. See this PR for more information.

Statement Attachment Handling & Normalization

PUT/POST /statements

xAPI clients can send statement data with arbitrary file attachments using the multipart/mixed Content-Type per the spec.

Format

Request bodies should be formatted as multipart/mixed per RFC 1341 with the following exceptions:

  • Preamble text is not supported, with the exception that a single leading CRLF before the first boundary is permitted for compatibility.
  • Epilogue text is not supported, but any number of empty trailing lines is permitted for compatibility.
Normalization

An attachment referenced in an xAPI statement (or substatement) that does not have a fileUrl property must be included at least once in the request body as a part identified by hash. In addition, multiple attachment objects, either in a single statement or across multiple statements in the same request, can refer to the same attachment, as recommended by the spec.

In practice this means that duplicate attachments may be present on the request. lrs will normalize/deduplicate attachments before sending them to implementation code such that every attachment in the list has a distinct hash. See this PR for more information.

Note that requests containing attachments not referenced in statement data will fail with a 400 status.

GET /statements

LRS implementations should return attachments in the same normalized form mentioned above, though this is not checked or enforced by lrs.

Bench Testing

Facilities to bench test any LRS with DATASIM are available. For instance, to bench the in-memory LRS:

$ make run-dev

And in another terminal:

$ clojure -Abench -m com.yetanalytics.lrs.bench http://localhost:8080/xapi -s 1000 -b 100

This will bench the LRS with 1000 statements in POST batches of 100.

License

Copyright © 2018-2021 Yet Analytics Inc.

Licensed under the Apache License, Version 2.0.

lrs's People

Contributors

cliffcaseyyet avatar emmaganyet avatar kelvinqian00 avatar milt avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

lrs's Issues

No get-activity-params spec

In lrs.protocol, there is no ::get-activity-params spec, even though there are ::get-statements-params and ::get-person-params specs for statement and agent queries, respectively.

Spec failure on `format-statement-ids`

If instrumentation is turned on, format-statement-ids can throw a spec failure. The reason for this is that if the Statement being formatted has a SubStatement, that SubStatement is recursively passed in; since SubStatements do not have IDs, they fail the ::xs/lrs-statement spec.

LRS ignores attachment multipart parts with duplicate SHAs

Multiparts are made into a map, as can be seen here: https://github.com/yetanalytics/lrs/blob/master/src/main/com/yetanalytics/lrs/pedestal/interceptor/xapi/statements/attachment.cljc#L164-L172

(defn multipart-map
  "Given a list of multiparts, make a map of them by xapi hash"
  [multiparts]
  (reduce (fn [m {:keys [headers] :as multipart}]
            (assoc m
                   (get headers "X-Experience-API-Hash")
                   multipart))
          {}
          multiparts))

Meaning dups are ignored. Instead, make it a map with vector vals a la group-by and reduce over the statements + attachments, pairing as we go.

Incorrect parameters spec for `-get-person`

The following is the current params spec for get-person:

(s/def ::get-person-params 
  (sc/with-conform-gen :xapi.agents.GET.request/params))

However, this is incorrect, since this is simply the spec for statement resource queries, not agent resource queries. In the former, agent is an optional parameter, but in the latter, agent is required. Therefore, during generative testing, -get-person would receive missing agent values, causing unnecessary errors.

EDIT: I likely used these specs wrong in lrsql and incorrectly determined the source of the issue. The followup comment still stands, however.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.