Discussion/Epic: Decouple worker_complete, paying base payment, and paying bonus: 8pts about dallinger HOT 15 OPEN

Thanks @jessesnyder, good idea.!

Regarding 2, your proposal makes sense but it limits us in that we can't give participants additional bonuses after worker_complete is called (the bonus is finalised at that stage). Instead, I think that each row in the Bonus table should correspond to a moment when the recruiter actually gave a bonus to the participant (by calling e.g. the MTurk API). By default, this only happens once, at worker_complete, where the participant is given the default bonus as determined by experiment.bonus. However, we can add other bonuses if we want at other points.

I updated your list correspondingly, including some clarification about 1 too:

1. Approve an assignment via the recruiter and pay out base_pay without other side effects, and in a way so that a later call to worker_complete will not cause an error, but instead just perform the remaining tasks that were normally performed by worker_complete but have not not been performed yet (e.g. update participant.end_time, set participant.status, recruit the next participant).
2. Whenever recruiter.reward_bonus is called, record the corresponding bonus in the Bonus table (inc. the amount of bonus as well as the reason) and update participant.bonus to the sum of the corresponding Bonus objects. By default recruiter.reward_bonus is only called once, during worker_complete where it gets the bonus amount from experiment.bonus, but we're allowing experimenters to call it more often if they want.

I hope this is clearer?

from dallinger.

Perfect; thanks!

I realized there's a very crucial fact not represented in my diagram, which is that when either the MTurkRecruiter or ProlificRecruiter are in use for the participant, the worker_function isn't called at this point, because their submitted_event() methods return None. This used to be intimately familiar to me, but time away allows moss to accumulate.

For both of these recruiters, the worker_function is called with AssignmentSubmitted as an argument, so all the same stuff happens, but the trigger happens later (for MTurk, an SNS notification is sent when the HIT is submitted there, and for Prolific, when our Prolific exit form is submitted).

I haven't thought through the implications for this project yet.

I think this version of the diagram accurately describes the MTurk/Prolific case.

from dallinger.

It was helpful for me to trace the dallinger2.js submitAssignment to /worker_complete all the way through in a sequence diagram. Attaching in case it's helpful to anyone else (apologies for my very poor handwriting. 😬 )

worker_complete.pdf

and transcribed to legible format

from dallinger.

Diagram updated... had missed call to Recruiter.assign_experiment_qualifications(), which is an important one!

from dallinger.

We need to be able to approve the participant partway through the experiment

What's included in this? Just Recruiter.assign_experiment_qualifications()

(@pmcharrison Let's talk tomorrow)

from dallinger.

(Possibly) tangential question: the implementation of Experiment.recruit() seems very different from what its docstring suggests:
https://github.com/Dallinger/Dallinger/blob/master/dallinger/experiment.py#L594

from dallinger.

Nice job with the diagram @jessesnyder!

We need to be able to approve the participant partway through the experiment

What's included in this? Just Recruiter.assign_experiment_qualifications()

(@pmcharrison Let's talk tomorrow)

I think we should include that, yes, but we also need to include AssignmentSubmitted.approve_assignment, which calls approve_hit and sets base_pay.

    def approve_assignment(self):
        self.participant.recruiter.approve_hit(self.assignment_id)
        self.participant.base_pay = self.config.get("base_payment")

The problem currently is that the place where AssignmentSubmitted.approve_assignment is called, i.e. AssignmentSubmitted.__call_, also sets a participant end time, allocates a bonus, and recruits a participant, which we might not want.

Under the current system, we can already approve a participant 'early' by manually calling AssignmentSubmitted.approve_assignment on that participant. Suppose the participant hits worker-complete a few minutes later, there is a possibility (I suppose) that the recruiter encounters an error if it tries to tell the recruiter to complete the HIT but the HIT has already been completed. So, maybe it is a good idea to add a boolean participant attribute called hit_approved that is set to True when approve_assignment has been called, so that we know when to skip this call?

The other thing that I guess we could do is improve the API for calling approve_assignment. Maybe something like this:

class Participant:
    def approve_assignment(self):
        event_type = participant.recruiter.submitted_event()

        worker_function(
            event_type=event_type,
            assignment_id=self.assignment_id,
            participant_id=self.id,
        )

That way if someone wants to approve a participant early, they can just call participant.approve_assignment().

Under this approach, the participant will not receive any bonus on participant.approve_assignment(); they'll have to wait until worker-complete, which will calculate the bonus according to experiment.bonus and record it in participant.bonus. I think this behaviour is good.

To support more advanced implementations, I do think it would be nice to implement this extra Bonus table, to which a Bonus row is added every time we call recruiter.reward_bonus. That will be really helpful for people who want to award bonuses outside the worker-complete function.

Once we do that, I think it would be good to update participant.bonus each time recruiter.reward_bonus is called, updating it to include the sum total of all bonuses that have been allocated to that participant. That way it'll account for multiple bonus rewards.

How should people allocated those manual additional bonuses? I guess it would be good to implement participant.reward_bonus which under the hood calls recruiter.reward_bonus.

from dallinger.

Thanks for these ideas, @pmcharrison. I realize I'm not actually clear on the specifics of the new use cases we're trying to support. Can you define these?

An attempt:

1. Approve an assignment via the recruiter and pay out base_pay without other side effects, and in a way so that a later attempt to approve the assign will either not happen, or not cause an error
2. Record multiple bonus amount + reason events during participation, which will be combined into a single bonus payment when the participant finishes the experiment
3. etc.

from dallinger.

@pmcharrison Possible complication, but maybe I'm missing something (🤞🏼): neither MTurk nor Prolific support approving a HIT/Assigment/Study before the participant submits them on those platforms. Maybe I'm wrong about Prolific, since my impression is that this is exactly the situation you're urgently trying to address here (and maybe already have a workaround)(?)

The status check and retries implemented for Prolific worry me, though:

from dallinger.

Sorry, I think I have been conflating these two terms. The plan is for us to submit and approve the assignment early, not to approve and then submit later.

from dallinger.

Maybe I don't understand what "early" means, if it must also mean "sometime after the participant declares themself done on MTurk or Prolific".

from dallinger.

I suppose this is a tricky issue, isn't it? The way the participant 'declares themselves done' is by being sent from the recruiter-exit page to an 'external submit' URL hosted by MTurk/Prolific. This normally happens at the end of the experiment, so it's natural to transition to this external URL. However, in our proposal we want the participant to keep on going after hitting this external URL, so we need to work out how we'd hit the URL without disrupting the experiment.

from dallinger.

@pmcharrison For the enhanced bonus scenario, I feel like Dallinger already supports this through letting you record bonus-worthy things that happen during the experiment, storing them in one of the Participant.property[1-5] fields, and then calculating the total bonus at the end? I remembered doing something in this vein here. Is a new Bonus table necessary?

from dallinger.

I think the bonus table becomes relevant once you relax the constraint that bonus calculation/payment only happens once (on worker complete) and allow it to happen at arbitrary other times too. I agree it is only helpful and not necessary though as there are other ways that the user can keep track of this.

from dallinger.

Note: This has been paused for now. Jesse has made good progress on relevant refactoring. We think we don't need to progress this further in the imminent future.

from dallinger.