Giter Club home page Giter Club logo

congame's Introduction

congame

congame is the best experimental economics software written in Racket. zTree and oTree are two common experimental economics platforms. The reason for building congame is the usual: I used oTree for some of my experiments, but found that it lacked some specific features I needed in my experiments:

  1. It is tedious and errorprone to have multi-part studies across several days or weeks
  2. It is tedious and errorprone to reuse and compose substudies into larger studies
  3. Specification of the experimental study are interweaved with code that deals with web specific concerns

congame addresses the first concern fully, and eases the other two -- at least for the kinds of studies I run. Unsurprisingly, congame lacks many features of oTree (and other behavioral research software):

  • There is no easy way for studies requiring multiple participants to interact with each other or respond to each other
  • There are almost no templates for many existing tasks and designs

Furthermore, congame has a few features that many will consider bugs:

  • It is written in Racket
  • It is build on top of the koyo web development kit developed by the indefatiguable (except for lack of coffee) Bogdan Popa
  • There is precisely zero documentation (beyond the code itself)

koyo is much less battle tested than Django, the Python framework on which oTree is built, and Racket is a very uncommon choice compared to Python. This means that at this stage, it is fairly unlikely that you will want to use congame.

Future Vision

The vision is to develop congame further, in particular as follows:

  • Provide easy ways to implement strategic interaction between participants, such as prisoner's dilemma etc
  • Ensure easy ways to change and test treatments and randomization in the study in a way that is entirely independent from the web aspects of the studies
  • Allow for bots, in addition to users, to participate. This will benefit with simulations, debugging, etc - similar to mTree

The features that will get implemented will however be determined by the priorities of the studies that I am actually running -- hence strategic interactions may never be implemented, since my studies don't require that.

Setup

Requirements

  • You need Racket since this is a Racket application.
  • You need node and nvm to build the assets.
  • You need access to a couple local Postgres databases. One named congame and the other congame_tests. The latter is exercised by unit tests.

First-time Setup

$ nvm use && npm install && npm run build
$ raco pkg install chief                 # install and build chief to launch web server
$ raco pkg install congame-core/         # install and build the core library and its deps
$ raco pkg install congame-identity/     # install and build the application and its deps
$ raco pkg install congame-smtp-proxy    # install and build the smtp-proxy dispatching traffic 
                                         # to the right identity server (staging or production)
$ raco pkg install conscript/            # install and build #lang conscript and dialects
$ raco pkg install congame-web/          # install and build the application and its deps
$ raco pkg install studies/              # install studies used in congame-example-study
$ raco pkg install congame-example-study # install example studies; needed for congame-doc
$ raco pkg install congame-doc/          # install and build the docs and their deps
# To install a package or study that tests depend on (e.g. congame-pjb-studies),
# uncomment the following line and replace by appropriate package.
# raco pkg install congame-price-lists/  # needed by congame-pjb-studies
# raco pkg install congame-pjb-studies/
$ raco pkg install congame-tests/  # install and build the tests and their deps

Development environment

Copy .env.default to .env. chief will automatically load the variables defined in this file into the environment of the subprocesses defined in the Procfile whenever it is run.

The app expects to be run behind an SSL terminated connection (for example, behind an nginx instance using a self-signed cert), even for local development . You can disable this requirement by setting current-continuation-key-cookie-secure? parameter to #f before the application is started.

Installing study packages

  • The package's info.rkt needs to (define congame-studies) like so:
(define congame-studies
  ;; Path to package, followed by the study provided by the package that you want to use
  '((congame-example-study/example consent-study)
    (congame-example-study/example simple-study)))
$ raco pkg install congame-example-study/  # install and build the study package and its deps

To get congame to pick up the new package, run

rm -r congame-web/studies/compiled/

then

touch congame-web/studies/all.rkt

This will cause the "all.rkt" module to be re-compiled, and that will in turn make the new study available in the web app.

Updating study caches

Run ./bin/bust-study-caches.sh after adding studies or bots to info.rkt files within study packages.

Running the app locally

$ nvm use
$ raco chief start

Re-building the documentation

$ raco setup --tidy --check-pkg-deps --unused-pkg-deps congame

Deployment Recommendation

See ci/README.md.

Acknowledgement

The creation of congame was supported by Közép-Európai Egyetem and Central European University Private University.

License

congame is licensed under the 3-Clause BSD license.

See LICENSE.

congame's People

Contributors

bogdanp avatar marckaufmann avatar lfloradrucker avatar dependabot[bot] avatar

Stargazers

Shi Xin avatar  avatar Nikolaus Schlemm avatar Sebastian Schmittner avatar Moritz Schwarz avatar  avatar Saad Azghour avatar Niklas Witzig avatar Jörgen Brandt avatar  avatar David.Gao avatar Dyllon Gagnier avatar Jake Chvatal avatar Clemens avatar Ezequiel Birman avatar Vic avatar Garrett van Wageningen avatar  avatar

Watchers

Abhik Khanra avatar  avatar  avatar James Cloos avatar

congame's Issues

How to call scripts that initialize some part of server (e.g. generate 200 matrices)?

If no csv mapping names to matrix-images (presumed to exist) is found, the server generates both new images and a csv file. See here.

When I deleted the matrix.csv file and ran it, it works, but logs this:

12:50:28 web.1 | [2021-02-26 12:50:28] [ 31525] [ info] runner: starting application process
12:50:38 web.1 | [2021-02-26 12:50:38] [ 31525] [warning] runner: timed out while waiting for 'listening' output
12:50:38 web.1 | [2021-02-26 12:50:38] [ 31525] [ info] runner: application process started with pid 31562

The timeout is almost surely from waiting for the 200 matrices to be finished.

My question is: is this the correct approach in general if I want something to be setup correctly before everything gets launched? If not, what is? If it is, how should it be improved so that the pipeline knows to wait for it?

I feel that the right time for the matrices is the time when the study that requires the matrices (here: pjb-pilot-study) is registered, so study registration seems the natural time for setup that is required by all study instances.

Required value is not `put` in `make-step/study`

I got an error where a value can't be getted, because it wasn't putted. It should however have been put. This is on the local dev server, maybe something to do with this, but this might be one of those weird errors that occur only once in a while and are tricky to pinpoint. Is there a possible problem with a value not yet having been written to the database by the time the step is run that expects it?

It did not happen again after clearing the participant and rerunning the study.

17:42:22 web.1 | [2021-02-16 17:42:22] [ 9765] [ debug] study: running step: #(struct:step/study the-study #procedure:...onents/study.rkt:320:3 #procedure:void #procedure:transition #(struct:study (participation-fee practice-tasks required-tasks price-lists) (task-treatment rest-treatment) (#(struct:step explain-study #<pr...
17:42:22 web.1 | [2021-02-16 17:42:22] [ 9765] [ debug] study: running step: #(struct:step/study required-tasks #procedure:...onents/study.rkt:320:3 #procedure:void #procedure:temp96 #(struct:study (n title max-wrong-tasks hide-description?) (success? correct-answers wrong-answers) (#(struct:step start-tasks #<procedure:in...
17:42:22 web.1 | [2021-02-16 17:42:22] [ 9765] [ debug] study: GET
17:42:22 web.1 | stack: (root the-study)
17:42:22 web.1 | key: required-tasks
17:42:22 web.1 | [2021-02-16 17:42:22] [ 9765] [ debug] study: GET
17:42:22 web.1 | stack: (root the-study)
17:42:22 web.1 | key: required-tasks
17:42:22 web.1 | [2021-02-16 17:42:22] [ 9765] [ debug] study: running step: #(struct:step start-tasks #procedure:initialize-tasks #procedure:continuer #procedure:task-completion)
17:42:22 web.1 | [2021-02-16 17:42:22] [ 9765] [ debug] study: GET
17:42:22 web.1 | stack: (root the-study required-tasks)
17:42:22 web.1 | key: n
17:42:22 web.1 | Servlet (@ /study/pjb-pilot) exception:
17:42:22 web.1 | get: value not found for key n
17:42:22 web.1 |
17:42:22 web.1 | context...:

Don't send emails to bots

We should ensure we send no emails to bots. Either we pass in an extra parameter to the send-email function/do an extra check before every send email. Alternatively, we could change the mail-adapter-send-email-with-template to check in the database whether the email belongs to a bot, in which case it isn't sent. The benefit of the latter approach is that it doesn't require micromanaging every email-send action, so that forgetting to check for bots doesn't lead to email attempts by mistake. @Bogdanp, what do you think?

Allow prolific users to sign up with only their prolific ID/email without verification

Prolific users have an ID that also corresponds directly to their email, such as [email protected]. If they have to sign up with their prolific ID, there is more scope for mistyping and if they need to verify the email, it takes longer. For some studies this is unnecessary overhead.

How can we achieve this given that the studies assume that the person is verified and logged in? Can we generate a password for the participant (or a new one, if they already did a study in the past) and log them in if they arrive through the correct url to a study? Or is this going to be a lot of work, in which case we might shelve this feature for a while.

How to identify OS-level libraries needed by Racket packages?

For 2htdp/image, there were several missing (Linux) libraries missing in the docker image, which I only realized upon each new deploy. I added one library at a time, with each new error message on the staging server, but this cycle takes ~5-15 minutes and eats up Github minutes.

One question and one feature:

  • @Bogdanp : Is there a way of easily checking in the docs what libraries something relies on?
  • We should implement a way to test the docker image on the development machine, the main issue is creating the right environments for it. @Bogdanp : should this piggy-back on our existing deploy.sh script, with a custom development.env file, or should it have its own separate setup that is simpler and interferes less with staging/production? Any best practices (or other ways to achieve the same)?

@Bogdanp :

Add optional `#:pre` and `#:post` hooks to study-steps

Randomization and initialization of variables is most naturally done at the beginning of a study, but there is currently no way of running them before displaying the first page. I suggest that we add #:pre and #:post (or before/after or setup/teardown) hooks to make-step/study. Currently this can be solved in two ways, both of which are not great.

First, we can pass values in as #:require-bindings. But this does not work for the top-level study; assumes that the parent study knows how to set certain values in a way that only the child study knows; and requires adding an extra step before make-step/study is called, even if it is naturally the first step.

Second, we can add the code before the rendering happens in the first step of the child study. But since this code can be run repeatedly by refreshing the page, it shouldn't be mutating (or at least it should be idempotent), which is precisely not what initialization usually is.

Move to within-subjects design in pilot for additional statistical power

The current pilot identifies differences in WTW between subjects, which has lower power. Alternatively, I could elicit WTW repeatedly for each individual and repeat the same experiment multiple times in succession, so that I get a better sense of individual-level effects and see changes in WTW better. However, if the main effect later on is estimated via between-subjects, then I still need that too, but the within subject might allow me to identify better which treatments cause the largest changes in WTW.

The first run might be pure between subjects (ask only once), but the later runs ask twice or more.

Matrices are too hard/take too much time

Participants really dislike doing extra matrices. Of the three participants, one dropped out in the middle of the required tasks, one didn't want to do any extra tasks for any amount until $2.80 and one was willing to do 5 extra matrices for $2.00, which is fairly pricey. So:

  1. Make them smaller
  2. Allow getting the number wrong by +1/-1

Think about using other tasks.

Can't see bot data for bot run from command line

We now show only bots in bot sets, hence I can't see what the progress is on the bot run from the command line without going into the DB.

The main reason I wanted this is to run the bot in non-headless (headful? headed?) mode to see what's going on, but it chokes on the first page. I haven't looked into it, as I am trying to fix something else.

run-study skips the first step in a study

Specifically, after the consent page, it runs task-study, but somehow skips the first step, 'initialize-tasks and thereupon fails to set the necessary variables. No amount of refreshing helps. I will add the commit of the code I was on at the time, but nothing to see really. I'll update if it replicates. I wonder if it is related to the other problems we are having with ending a step/study and starting a step/study.

Logs:

12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: step #(struct:step consent #procedure:consent #<procedure:consent/bo
t> #procedure:temp104) returned continue
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: GET
12:37:55 web.1 | stack: (root the-study)
12:37:55 web.1 | key: consent?
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: PUT
12:37:55 web.1 | stack: (root the-study)
12:37:55 web.1 | key: rest-treatment
12:37:55 web.1 | value: elicit-then-get-rest
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: running step: #(struct:step/study required-tasks #<procedure:...onents
/study.rkt:320:3> #procedure:void #procedure:temp108 #(struct:study (n title max-wrong-tasks hide-description?) (success? correct-answer
s wrong-answers) (#(struct:step start-tasks #<procedure:i...
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: GET
12:37:55 web.1 | stack: (root the-study)
12:37:55 web.1 | key: required-tasks
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: GET
12:37:55 web.1 | stack: (root the-study)
12:37:55 web.1 | key: required-tasks
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: running step: #(struct:step task #procedure:task #<procedure:task/bo
t> #procedure:task-completion)
12:37:55 web.1 | [2021-02-17 12:37:55] [ 19650] [ debug] study: GET
12:37:55 web.1 | stack: (root the-study required-tasks)
12:37:55 web.1 | key: current-matrix
12:37:55 web.1 | Servlet (@ /study/pjb-pilot;((%22k%22%20.%20%22(1%2015%2017938064)%22))) exception:
12:37:55 web.1 | get: value not found for key current-matrix
12:37:55 web.1 |
12:37:55 web.1 | context...:
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-pjb-studies/tasks.rkt:98:0: task
12:37:55 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
12:37:55 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
12:37:55 web.1 | [repeats 1 more time]
12:37:55 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
12:37:55 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:117:0: send/suspend/dispatch
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:444:0: run-step
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
12:37:55 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
12:37:55 web.1 | [repeats 1 more time]
12:37:55 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
12:37:55 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:117:0: send/suspend/dispatch
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:444:0: run-step
12:37:55 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
12:37:55 web.1 | ...

Fix dependency of pjb on congame-web/config

Re the build failure, this is one place where the circular dependency is biting us and we should fix it by either coming up with a different mechanism for studies to access configuration or with a different mechanism for studies to be able to define jobs (or both).

Bot doesn't run on staging

I tried to run a bot from a bot-set on staging server, but it failed with the following error that I didn't investigate at all due to looking into #39:

error exn:fail:contract
process*: contract violation expected: path-string? given: #f
ID: a15> error exn:fail:contract
process*: contract violation expected: path-string? given: #f
ID: a15a3a07526c4141ad14bb3b99a2d2ef
March 12, 2021, 2:18:50 p.m. UTC

Exception

exn:fail:contract: process*: contract violation
expected: path-string?
given: #f
File ".../racket/system.rkt", line 100, in unknown
File ".../racket/system.rkt", line 98, in do-process*/ports
File ".../marionette/main.rkt", line 46, in start-marionette!
File ".../marionette/main.rkt", line 121, in unknown
File ".../pages/admin.rkt", line 533, in unknown
...
(26 additional frame(s) were not displayed)
a3a07526c4141ad14bb3b99a2d2ef
March 12, 2021, 2:18:50 p.m. UTC

Exception

exn:fail:contract: process*: contract violation
expected: path-string?
given: #f
File ".../racket/system.rkt", line 100, in unknown
File ".../racket/system.rkt", line 98, in do-process*/ports
File ".../marionette/main.rkt", line 46, in start-marionette!
File ".../marionette/main.rkt", line 121, in unknown
File ".../pages/admin.rkt", line 533, in unknown
...
(26 additional frame(s) were not displayed)

Create additional bot runner that runs study without going through browser/marionette

Our current bot runner works by running a browser, requesting the pages, and sending responses like an actual user would. This is fantastic for testing purposes, and even works for data generation. It is however fairly slow, since even a single bot run takes several seconds to a minute on the current pilot study, and this will only get worse. This won't work for generating hundreds of data points.

Instead, we probably should write a different runner that walks the study, but does so without having the bots send requests and the server sending responses, instead it uses a direct response/request loop. This will require that we replace handlers by something that a bot can understand and respond to.

An alternative approach would be that a single bot walks through the study, but on each page it doesn't generate a single answer, but 100 (say). This approach requires that we change the storage layer though, so that a single user can submit not merely one answer, but a list or hash of answers. This would work because generating 100 answers isn't the bottleneck, but the response/request waiting is. I have no sense of how hard it is to add this option to the DB layer, or to somehow replace it for bots. Maybe one can let the bot-side save everything on its side, so that it doesn't interfere with the DB - otherwise we would add features to the normal-user side that are not necessary, which will lead to confusing and difficult-to-understand bugs.

Edit: The alternative is stupid. It cannot respond to different answers by different bots. We also shouldn't change behavior specifically for bots.

Refreshing a page leads to error

When I refreshed the page determine-extra-tasks, I get an error because car is passed () when it expects a pair. I thought the failing call is (car resume-stack) here, but this is inside an else-clause where we check for (null? resume-stack) in the first clause. It went away with several refreshs.

18:08:23 web.1 | [2021-02-16 18:08:23] [ 10678] [ debug] study: PUT
18:08:23 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:23 web.1 | key: remaining-price-lists
18:08:23 web.1 | value: ()
18:08:23 web.1 | [2021-02-16 18:08:23] [ 10678] [ debug] study: running step: #(struct:step determine-extra-tasks #<procedure:determin
e-extra-tasks> #procedure:handler/bot #procedure:default-transition)
18:08:26 web.1 | [2021-02-16 18:08:26] [ 10678] [ debug] study: GET
18:08:26 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:26 web.1 | key: remaining-price-lists
18:08:26 web.1 | [2021-02-16 18:08:26] [ 10678] [ debug] study: GET
18:08:26 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:26 web.1 | key: answered-price-lists
18:08:26 web.1 | Servlet (@ /study/pjb-pilot;((%22k%22%20.%20%22(14%2011%2094073931)%22))) exception:
18:08:26 web.1 | car: contract violation
18:08:26 web.1 | expected: pair?
18:08:26 web.1 | given: '()
18:08:26 web.1 |
18:08:26 web.1 | context...:
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-pjb-studies/pjb-pilot.rkt:361:15: temp57
18:08:26 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:379:33
18:08:26 web.1 | f807
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:408:0: run-study
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
18:08:26 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
18:08:26 web.1 | [repeats 1 more time]
18:08:26 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
18:08:26 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:117:0: send/suspend/dispatch
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:444:0: run-step
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
18:08:26 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
18:08:26 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
18:08:26 web.1 | [repeats 1 more time]
18:08:26 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
18:08:26 web.1 | ...
18:08:27 web.1 | [2021-02-16 18:08:27] [ 10678] [ debug] study: GET
18:08:27 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:27 web.1 | key: remaining-price-lists
18:08:27 web.1 | [2021-02-16 18:08:27] [ 10678] [ debug] study: GET
18:08:27 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:27 web.1 | key: answered-price-lists
18:08:27 web.1 | Servlet (@ /study/pjb-pilot;((%22k%22%20.%20%22(14%2011%2094073931)%22))) exception:
18:08:27 web.1 | car: contract violation
18:08:27 web.1 | expected: pair?
18:08:27 web.1 | given: '()
18:08:27 web.1 |
18:08:27 web.1 | context...:
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-pjb-studies/pjb-pilot.rkt:361:15: temp57
18:08:27 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:379:33
18:08:27 web.1 | f807
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:408:0: run-study
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
18:08:27 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
18:08:27 web.1 | [repeats 1 more time]
18:08:27 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
18:08:27 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:117:0: send/suspend/dispatch
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:444:0: run-step
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
18:08:27 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
18:08:27 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
18:08:27 web.1 | [repeats 1 more time]
18:08:27 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
18:08:27 web.1 | ...
18:08:27 web.1 | ...
18:08:28 web.1 | [2021-02-16 18:08:28] [ 10678] [ debug] study: GET
18:08:28 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:28 web.1 | key: remaining-price-lists
18:08:28 web.1 | [2021-02-16 18:08:28] [ 10678] [ debug] study: GET
18:08:28 web.1 | stack: (root the-study elicit-WTW-and-work)
18:08:28 web.1 | key: answered-price-lists
18:08:28 web.1 | Servlet (@ /study/pjb-pilot;((%22k%22%20.%20%22(14%2011%2094073931)%22))) exception:
18:08:28 web.1 | car: contract violation
18:08:28 web.1 | expected: pair?
18:08:28 web.1 | given: '()
18:08:28 web.1 |
18:08:28 web.1 | context...:
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-pjb-studies/pjb-pilot.rkt:361:15: temp57
18:08:28 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:379:33
18:08:28 web.1 | f807
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:408:0: run-study
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
18:08:28 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
18:08:28 web.1 | [repeats 1 more time]
18:08:28 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
18:08:28 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:117:0: send/suspend/dispatch
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:444:0: run-step
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:320:3
18:08:28 web.1 | /home/marc/Git/kutat/congame/congame-core/components/study.rkt:371:0: response/step
18:08:28 web.1 | /usr/share/racket/collects/racket/contract/private/arrow-higher-order.rkt:375:33
18:08:28 web.1 | [repeats 1 more time]
18:08:28 web.1 | /usr/share/racket/pkgs/web-server-lib/web-server/servlet/web.rkt:123:10
18:08:28 web.1 | ...
18:08:33 web.1 | [2021-02-16 18:08:33] [ 10678] [ debug] study: running step: #(struct:step/study the-study #<procedure:...onents/stud
y.rkt:320:3> #procedure:void #procedure:transition #(struct:study (participation-fee practice-tasks required-tasks price-lists) (task-tr
eatment rest-treatment) (#(struct:step explain-study #<pr...
18:08:33 web.1 | [2021-02-16 18:08:33] [ 10678] [ debug] study: running step: #(struct:step/study elicit-WTW-and-work #<procedure:...o
nents/study.rkt:320:3> #procedure:void #procedure:temp101 #(struct:study (price-lists) (WTWs) (#(struct:step introduce-WTW #<procedure:i
ntroduce-WTW> #procedure:handler/bot #<procedure:defaul...
18:08:33 web.1 | [2021-02-16 18:08:33] [ 10678] [ debug] study: GET
18:08:33 web.1 | stack: (root the-study)
18:08:33 web.1 | key: price-lists

Failing should lead to study completion with payment info

It is possible for participants to do some of the work and then fail at a later stage. In this case, they often can receive partial payment. This requires however that we add a way to mark the study as 'completed in the database.

I suggest that we have a way to mark a substudy as failed, which is like done but leads parent studies to throw failed until there is a level at which it is caught. On the top-level, I will usually want to catch it with a final goodbye screen showing partial payments, as well as the reason for failing. Alternatively, we could simply have a default study that we jump to upon failed, so that one can only fail from a substudy to the overall study, but nowhere in between.

One issue to deal with is that parent studies expect that a substudy provides its #:provides -- but this requirement should be overridden on failed.

local dev install doesn't work due to migration error

I get the following:

20:56:47 web.1 | [2020-10-15 20:56:47] [ 24194] [ debug] north-adapter: applying revision 4b823e9c3a3c15057acd77d39a6fe1af
20:56:47 web.1 | migrate!: failed to apply revision 4b823e9c3a3c15057acd77d39a6fe1af
20:56:47 web.1 | cause: query-exec: column "status" does not exist
20:56:47 web.1 | SQLSTATE: 42703
20:56:47 web.1 | context...:
20:56:47 web.1 | do-error
20:56:47 web.1 | .../private/more-scheme.rkt:168:6: loop
20:56:47 web.1 | .../database/migrator.rkt:20:3: component-start
20:56:47 web.1 | .../private/arrow-val-first.rkt:486:18
20:56:47 web.1 | .../private/system.rkt:80:4: for-loop
20:56:47 web.1 | .../private/system.rkt:77:0: system-start
20:56:47 web.1 | winder-dummy
20:56:47 web.1 | .../private/arrow-val-first.rkt:486:18
20:56:47 web.1 | .../congame/dynamic.rkt:66:0: start
20:56:47 web.1 | .../private/arrow-higher-order.rkt:379:33
20:56:47 web.1 | proc
20:56:47 web.1 | call-in-empty-metacontinuation-frame
20:56:47 web.1 | call-with-module-prompt
20:56:47 web.1 | body of (submod "/home/marc/Git/kutat/congame/congame/dynamic.rkt" main)
20:56:47 web.1 | temp35_0
20:56:47 web.1 | for-loop

I can't tell why this is happening, since that revision is supposed to create the colum status. However that revision does two things: create the column, and create a constraint. I am wondering now if one has to split both? On the other hand it works fine on the server, so my guess is that something weirder has happened.

Extract prolific url parameters automatically

Prolific generates a URL with the following query parameters that we should extract:

?PROLIFIC_PID={{%PROLIFIC_PID%}}&STUDY_ID={{%STUDY_ID%}}&SESSION_ID={{%SESSION_ID%}}

Ideally this should be integrated with #44.

Wrong default value for `bot-set-id` field in users table

The CI/CD fails with the new way of adding bots without real signup email and proper verification url, which is unsurprising:

raco test: "congame-tests/info.rkt"
1 congame-tests/congame/components/hash.rkt
0 congame-tests/congame/common.rkt
0 congame-tests/info.rkt
1 1 congame-tests/congame/components/auth.rkt
1 1 congame-tests/congame/components/mail.rkt
1 1 congame-tests/congame/components/user.rkt
3/4 test failures

Error message:

raco test: (submod "congame-tests/congame/components/auth.rkt" test)
query-value: cannot convert given value to SQL type
given: #f
type: int4
expected: int32?
dialect: PostgreSQL
context...:

Edit: I thought it was about email sending, but it was more mundane.

Extract payment information in simple csv

Check how participants get paid on prolific and how to make this easy to handle by either directly hooking congame up with prolific, or generating a csv with user, participant, and payment information.

Simplify form building

Currently, even making a one-question form is a boiler-plate fest that rivals Java. Other software shines for making surveys (Qualtrics in particular), primarily thanks to GUIs. While I don't expect us to compete with Qualtrics, I would like to stop competing with Java.

  1. @Bogdanp : Quick discussion of what should be possible in the short-term
  2. @MarcKaufmann : build a bunch of surveys (plan for upcoming week) and identify pain points and common defaults that form-building tools should use
  3. build the tools

How to correctly load JS

For whatever reason, the js/audio-player.js doesn't work, even though it clearly loads since it correctly sets the objects playerpause and I see it getting loaded. The code is here and for ease copied:

var audioContainer = document.getElementById('audioContainer');
var audio = document.getElementById('audio');
var audioControls = document.getElementById('audio-controls');


var playpause = document.getElementById('playpause');
var volinc = document.getElementById('volume-up');
var voldec = document.getElementById('volume-down');

playpause.addEventListener('click', function(e) {
    if (audio.paused || audio.ended) audio.play();
    else audio.pause();
});

When clicking the button, all I get is:

Uncaught TypeError: up.syntax is undefined
nativeCallback http://127.0.0.1:5100/static/vendor/unpoly.min.js?rev=dev:1
n http://127.0.0.1:5100/static/vendor/unpoly.min.js?rev=dev:1

and once a complaint (might have been upon loading the page):

Cannot play media. No decoders for requested formats: application/ogg

Yet, when I directly type into the console audio.play(), it works and the music starts. Similarly, if I copy-paste the above code snippet into the console, then clicking on the button also works.

What gives?

Continuing can fail with a resume error

This happened to both Marc and I and we have been unable to reproduce it. Might be related to failing a form and then continuing. The error was related to "resuming". Might have been caused by a form failure followed by a jump outside of a substudy.

Media files don't work on staging server

For some reason, the test-audio.mp3 doesn't load or exist or is not of the right type on the staging server:

Media resource https://staging.totalinsightmanagement.com/resource/6c5a0c28d6b45c19dfa9f61a70b47d0c/test-audio.mp3 could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_METADATA_ERR (0x806e0006)
Uncaught (in promise) DOMException: The media resource indicated by the src attribute or assigned media provider object was not suitable.

When I follow the link to the resource directly, I get a message that "No video with supported format and MIME type found."

My local dist created via raco koyo dist does have the test-audio.mp3 file in it, and it runs fine. Nginx is not configured to deal with mp3's in some special way, at least not in the /etc/nginx/sites-available config. In Chromium-browser (Ubuntu), the hidden button is instead not hidden, and the music also doesn't play. It does not throw any errors that I can see.

I found this, which suggests that the content-type is not properly set. The decoding issue, although then why does it work in development, and even when I use the raco koyo dist version? Something something docker/https?

Forms: form-actions don't close over variables in original form-handler call

We use (form a-form an-action a-renderer) to create forms. The action should be taken upon a valid form submission. Suppose we have the following:

(define (handler some-argument)
  (define n (random 10))
  (define a-form ...) ; asks to guess the number
  (define (an-action guessed-number)
    (if (= n guessed-number) (put 'success? #true) (put 'success? #false)))
  (define (a-renderer rw) ...)
  (form a-form an-action a-renderer))

Then after submission of the form with a guessed-number, the handler is called again, which generates a new n. If a valid guess was provided, it then compares the guessed-number against this new value of n, not the original one -- which is of course not the intended behavior.

How can we fix this?

Integrate bots into testing

Implement a way to have bot-sets specifically for various testing purposes that are run upon deployment, or can be run easily from the admin interface. Unlike other bot sets:

  1. they should come pre-configured, rather than require manual adding of bot-sets.
  2. we should be able to trigger all test bots across all studies if we want to
  3. it should be possible to trigger them at the study-specific level

Integrate user login/emailing with prolific

Since prolific provides prolific-specific emails, we can ask participants to use those to sign up, and to email them updates/notification about study participation if studies go over multiple days.

Figure out if there is anything at all we need to do on the congame side of things to implement this.

Price lists should not clear answers when an incomplete form is submitted

When a form is submitted with only some of the choices made, it doesn't pass validation and is reset to an empty form. Instead it should keep the answers that were provided and only require the remaining to be filled in.

Is there a way to switch this behavior on for forms in general, or does it have to be done on a per-form basis?

Break out study code into separate repositories

The code for specific experiments should live in its own repositories and be easy to integrate with congame-core.

  1. Until congame is a fully-fledged package, what is the best way to split it out in a way that makes deployment still easy?
  2. And why is it again that we moved from an executable to docker -- it had something to do with being unable to raco koyo dist. Was it the serializable structs, or the define-resource?

Rounds & groups

Need new operators:

  • rounds: get current round, set new round per participant
  • groups: assign participants in a pool to some group, but also to re-assign
  • group data: get/group, put/group

study_data should have round & group in the primary key.

How to integrate local pkg deps are not found?

The package 'congame-pjb-studies' depends on 'congame-price-lists' and 'study-tools' packages, but when installing via

$ raco pkg install congame-pjb-studies/

if cannot find these packages which exist somewhere on the local machine after all. Am I right to assume that the approach is to install each of the required dependencies manually first for now, and later to host all the packages so they are found?

Ensure new bots are registered despite cache

The following is the info.rkt file for congame-pjb-studies:

#lang info

(define collection "congame-pjb-studies")
(define deps '("base"
               "congame-core"
               "congame-price-lists"))
(define build-deps '())
(define congame-studies
  '((congame-pjb-studies/pjb-pilot pjb-pilot-study)))
(define congame-bots
  '((congame-pjb-studies/pjb-pilot-bot pjb-pilot-bot #:for pjb-pilot-study #:models (pjb-pilot-bot-model))))

Despite this, the call in congame/studies/all.rkt to `find-relevant-directories doesn't find this bot:

all.rkt> (find-relevant-directories '(congame-studies))
'(#path:/home/marc/Git/kutat/congame/congame-pjb-studies
#path:/home/marc/Git/kutat/congame/congame-price-lists)
all.rkt> (find-relevant-directories '(congame-bots))
'(#path:/home/marc/Git/kutat/congame/congame-price-lists)

I also directly print the *bot-registry* after registering all the bots, and it shows the same, so this is not a REPL issue. I am fairly sure this is a cache issue, but can't seem to fix it. I tried:

  • raco pkg update for congame/, congame-pjb-studies/, can congame-core/
  • raco setup --tidy ... as well as raco setup -c ...
  • deleted the compiled/ folders in the top-level of each of the above.

Writing this, I decided to delete all compiled/ folders and update packages. And zing! It works.

So it would be good to make sure this is handled more automatically, so that if someone update info.rkt to include bots, that we automatically remove and update the cache.

Cache git lfs objects?

It seems that git lfs can eat lots of bandwidth without caching, although I don't get why it would need more than without git lfs. Is it just a case that git lfs downloads all files in the pipeline, whereas in a normal pipeline files can be cached more easily -- or is the concern simply that those who use git lfs usually have large files, hence caching becomes important?

If it is worse than the prior solution then (i) why did I start using it and (ii) let's cache it.

Identify what leads to most significant changes in WTW

  • How many and which songs/activities to have after the tasks
  • Which task to have: with/without cognitive, with time pressure, with piece-rate
  • Number of tasks, length of breaks
  • Have them come back later after doing more work on prolific to have increased WTW, proving that they did more tasks in the meantime
  • Task is to predict additional behaviors in studies (e.g. own or others WTW or similar in hypothetical or actual situations); vignettes
  • Elicit before and after extra tasks: under broad bracketing, should be no change -- except under BB of PB type behavior, so that the later behavior is extra immediate work, while the earlier work is realized to be about extra later work (one would have to work anyway). Under NB, can identify change in WTW from extra work and if it is big, try to ride on that.
  • Tasks could be about learning something, having more and more such tasks over time and payment for it (long term)

Start at the low end of the spectrum (i.e. short runs) where little time is needed; only later scale up to larger amounts of tasks.

In production, email admin rather than erroring out if a double payment

Due to an error (see #27 (comment)), the transition for 'required-tasks got called twice hence it attempted to put a payment twice. This led to an error that can't be solved as either normal or admin user.

We probably should disallow the double-calling of a transition, but errors can creep in that break a step and then it almost surely can happen -- or is there a way to avoid this for errors in the handlers?

Nonetheless, should the impossible happen in production, it seems better to log to sentry (with payment details), while leaving the DB alone.

Email everybody

I pushed a broken commit as I am trying to implement email again. The issue, if you remember, is that the job broker needs to have access to the mailer in order to be able to set up jobs for email sending anywhere in studies. In the earlier version, we did this via system-ref -- which is done via ... system, I guess? I can't figure out where define-system is defined, so currently I get the error that system-ref used elsewhere is undefined.

The questions are:

  1. How to get (system-ref 'mailer) to work in congame-pjb-studies/...? (I don't know, found a define-system in koyo/console, but didn't see that imported previously.)
  2. Is the below still the way to set up emails? (I believe yes)
  3. The with-sentry is there to ensure I get notified if the job trips up (I am 90% certain of this, but checking)
  (define-job (send-study-completion-email u payment)
  (with-sentry
    (define mailer (system-ref 'mailer))
    (mailer-send-study-completed-email mailer u payment)))

(define (send-completion-email)
  (schedule-at
   (get-moment)
   (send-completion-email
    (user-username (current-user))
    (get-payment (current-participant-id)))))

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.