Comments (15)
Thanks @jessesnyder, good idea.!
Regarding 2, your proposal makes sense but it limits us in that we can't give participants additional bonuses after worker_complete is called (the bonus is finalised at that stage). Instead, I think that each row in the Bonus table should correspond to a moment when the recruiter actually gave a bonus to the participant (by calling e.g. the MTurk API). By default, this only happens once, at worker_complete, where the participant is given the default bonus as determined by experiment.bonus. However, we can add other bonuses if we want at other points.
I updated your list correspondingly, including some clarification about 1 too:
- Approve an assignment via the recruiter and pay out base_pay without other side effects, and in a way so that a later call to
worker_complete
will not cause an error, but instead just perform the remaining tasks that were normally performed byworker_complete
but have not not been performed yet (e.g. updateparticipant.end_time
, setparticipant.status
, recruit the next participant). - Whenever
recruiter.reward_bonus
is called, record the corresponding bonus in theBonus
table (inc. the amount of bonus as well as the reason) and updateparticipant.bonus
to the sum of the correspondingBonus
objects. By defaultrecruiter.reward_bonus
is only called once, duringworker_complete
where it gets the bonus amount fromexperiment.bonus
, but we're allowing experimenters to call it more often if they want.
I hope this is clearer?
from dallinger.
Perfect; thanks!
I realized there's a very crucial fact not represented in my diagram, which is that when either the MTurkRecruiter or ProlificRecruiter are in use for the participant, the worker_function
isn't called at this point, because their submitted_event()
methods return None
. This used to be intimately familiar to me, but time away allows moss to accumulate.
For both of these recruiters, the worker_function
is called with AssignmentSubmitted
as an argument, so all the same stuff happens, but the trigger happens later (for MTurk, an SNS notification is sent when the HIT is submitted there, and for Prolific, when our Prolific exit form is submitted).
I haven't thought through the implications for this project yet.
I think this version of the diagram accurately describes the MTurk/Prolific case.
from dallinger.
It was helpful for me to trace the dallinger2.js submitAssignment
to /worker_complete
all the way through in a sequence diagram. Attaching in case it's helpful to anyone else (apologies for my very poor handwriting. 😬 )
and transcribed to legible format
from dallinger.
Diagram updated... had missed call to Recruiter.assign_experiment_qualifications()
, which is an important one!
from dallinger.
We need to be able to approve the participant partway through the experiment
What's included in this? Just Recruiter.assign_experiment_qualifications()
(@pmcharrison Let's talk tomorrow)
from dallinger.
(Possibly) tangential question: the implementation of Experiment.recruit()
seems very different from what its docstring suggests:
https://github.com/Dallinger/Dallinger/blob/master/dallinger/experiment.py#L594
from dallinger.
Nice job with the diagram @jessesnyder!
We need to be able to approve the participant partway through the experiment
What's included in this? Just
Recruiter.assign_experiment_qualifications()
(@pmcharrison Let's talk tomorrow)
I think we should include that, yes, but we also need to include AssignmentSubmitted.approve_assignment
, which calls approve_hit
and sets base_pay
.
def approve_assignment(self):
self.participant.recruiter.approve_hit(self.assignment_id)
self.participant.base_pay = self.config.get("base_payment")
The problem currently is that the place where AssignmentSubmitted.approve_assignment
is called, i.e. AssignmentSubmitted.__call_
, also sets a participant end time, allocates a bonus, and recruits a participant, which we might not want.
Under the current system, we can already approve a participant 'early' by manually calling AssignmentSubmitted.approve_assignment
on that participant. Suppose the participant hits worker-complete
a few minutes later, there is a possibility (I suppose) that the recruiter encounters an error if it tries to tell the recruiter to complete the HIT but the HIT has already been completed. So, maybe it is a good idea to add a boolean participant attribute called hit_approved
that is set to True
when approve_assignment
has been called, so that we know when to skip this call?
The other thing that I guess we could do is improve the API for calling approve_assignment
. Maybe something like this:
class Participant:
def approve_assignment(self):
event_type = participant.recruiter.submitted_event()
worker_function(
event_type=event_type,
assignment_id=self.assignment_id,
participant_id=self.id,
)
That way if someone wants to approve a participant early, they can just call participant.approve_assignment()
.
Under this approach, the participant will not receive any bonus on participant.approve_assignment()
; they'll have to wait until worker-complete
, which will calculate the bonus according to experiment.bonus
and record it in participant.bonus
. I think this behaviour is good.
To support more advanced implementations, I do think it would be nice to implement this extra Bonus
table, to which a Bonus
row is added every time we call recruiter.reward_bonus
. That will be really helpful for people who want to award bonuses outside the worker-complete
function.
Once we do that, I think it would be good to update participant.bonus
each time recruiter.reward_bonus
is called, updating it to include the sum total of all bonuses that have been allocated to that participant. That way it'll account for multiple bonus rewards.
How should people allocated those manual additional bonuses? I guess it would be good to implement participant.reward_bonus
which under the hood calls recruiter.reward_bonus
.
from dallinger.
Thanks for these ideas, @pmcharrison. I realize I'm not actually clear on the specifics of the new use cases we're trying to support. Can you define these?
An attempt:
- Approve an assignment via the recruiter and pay out base_pay without other side effects, and in a way so that a later attempt to approve the assign will either not happen, or not cause an error
- Record multiple bonus amount + reason events during participation, which will be combined into a single bonus payment when the participant finishes the experiment
- etc.
from dallinger.
@pmcharrison Possible complication, but maybe I'm missing something (🤞🏼): neither MTurk nor Prolific support approving a HIT/Assigment/Study before the participant submits them on those platforms. Maybe I'm wrong about Prolific, since my impression is that this is exactly the situation you're urgently trying to address here (and maybe already have a workaround)(?)
The status check and retries implemented for Prolific worry me, though:
Dallinger/dallinger/prolific.py
Line 49 in 0aae002
from dallinger.
Sorry, I think I have been conflating these two terms. The plan is for us to submit and approve the assignment early, not to approve and then submit later.
from dallinger.
Maybe I don't understand what "early" means, if it must also mean "sometime after the participant declares themself done on MTurk or Prolific".
from dallinger.
I suppose this is a tricky issue, isn't it? The way the participant 'declares themselves done' is by being sent from the recruiter-exit page to an 'external submit' URL hosted by MTurk/Prolific. This normally happens at the end of the experiment, so it's natural to transition to this external URL. However, in our proposal we want the participant to keep on going after hitting this external URL, so we need to work out how we'd hit the URL without disrupting the experiment.
from dallinger.
@pmcharrison For the enhanced bonus scenario, I feel like Dallinger already supports this through letting you record bonus-worthy things that happen during the experiment, storing them in one of the Participant.property[1-5]
fields, and then calculating the total bonus at the end? I remembered doing something in this vein here. Is a new Bonus table necessary?
from dallinger.
I think the bonus table becomes relevant once you relax the constraint that bonus calculation/payment only happens once (on worker complete) and allow it to happen at arbitrary other times too. I agree it is only helpful and not necessary though as there are other ways that the user can keep track of this.
from dallinger.
Note: This has been paused for now. Jesse has made good progress on relevant refactoring. We think we don't need to progress this further in the imminent future.
from dallinger.
Related Issues (20)
- Link to PsyNet issue #715: Check/implement logic for customising recruiter classes in PsyNet and in custom experiments
- Create a CI job for psynet to run tests with logging maxed out on postgres and publish pgbadger analysis
- Flask skeleton for Dallinger Control Center
- Automatically install a munin master and provide a secure, RESTful endpoint returning server usage stats HOT 1
- Standardize recruiter classes HOT 2
- Restructure recruiter files HOT 1
- More generic recruiter vocabulary HOT 1
- Refactor the recruiter commandline interface
- Refactoring of recruiters -- EPIC
- Log recruiter costs in app database
- Drop support for Python 3.9 HOT 1
- Fix Chrome and ChromeDriver download link in Dockerfile
- Remove redundant app name from Docker redis container name
- Testing the Prolific API in the CI
- Worker processes should take lowest priority in Docker compose: 3pts HOT 2
- When deploying to Prolific, allow user to choose the workspace and the project to deploy to: 2pts HOT 1
- Show database reset button in dashboard only in debug mode
- Make sure app name only contains valid characters
- Customizable worker log level in Dallinger
- Fix SyntaxWarning
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dallinger.