False negatives: repobee reviews check about repobee HOT 6 CLOSED

Hi @connorferster,

Thanks for the kind words!

So, I can't really say off the top of my head what is the matter. Let's try to get to the bottom of it, I have a few questions (but before you answer them read the rest of this message, you may not have to).

1. Which VCS platform are you on (GitHub, GitLab, Gitea)?
2. Can you provde the exact commands that you ran (with options, of course excluding anything sensitive)?
3. When RepoBee assigns reviewers, it creates review teams (groups if on GitLab). Can you verify that your TA is a member of the review team of the repo he is writing up a review for?

Now, we might be able to get to the bottom of why the reviews aren't working for you, but there's an inconvenient fact in that the initial implementation I made of peer reviews is... well, not great. I really wanted to put all of the state on the VCS platform, but in doing so it all became incredibly convoluted and error prone. It turned out to be one of those "seemed like a good idea at the time" kind of things.

There is another option: the RepoBee 4 version of peer review commands are available as a preview feature. They function similarly to the current commands, but the assign command spits out a JSON file that contains the allocations, and all other commands take that as input. It makes for a way more robust review experience. See #868 for details on how to use it. It might even be worthwhile to try that out first, before we spend any significant amount of time trying to debug your current problem.

As the preview commands aren't officially released, any feedback you could provide on them could directly impact how I tweak them for the eventual release. RepoBee 4 was supposed to be released this semester (with a full GUI, as well), but life got in the way, and I don't want to make a major release when semesters have already started.

Sorry for the essay, I just get very excited when people use my stuff :)

from repobee.

Oh, and just to be clear, the reason I recommend trying out the preview commands is that even if they also have some problem, they're a whole lot easier to debug as the review allocations are in a JSON file.

from repobee.

@slarse

Sorry for the essay, I just get very excited when people use my stuff :)

Lol, I can relate! :)))

Quick answers:

1. GitHub

repobee repos setup -a Workbook_01 --sf test_students.txt
repobee reviews assign -a Workbook_01 --sf test_students.txt --num-reviews 1
repobee reviews check -a Workbook_01 --sf test_students.txt --num-reviews 1 --title-regex '\APeer review\Z'

3. Yes, I can confirm that. We were the only two students in the test_students.txt file.

The JSON approach makes a lot of sense. I did the same thing when I created an adhoc review system that ran out of submissions of assignments to my email inbox. Submissions were auto-filed by Outlook into a sub-folder, running the script would make random pairs of students out of the submitted assignments sub-folder, and then the script would email the reviewing parties the submissions of the other.

Repobee 4, you say? That sounds great. Two questions:

1. So, when I installed repobee this year, it already came with this prototype functionality?
2. How do I set that environment variable?

Looking forward to the next major release!

from repobee.

@connorferster Regarding your answers, everything looks in order ... Can't really say based on that what might be wrong. Let's try the preview commands before we delve any further into debugging.

So, when I installed repobee this year, it already came with this prototype functionality?

Yes, it's been in preview for quite a while, hehe. If you want to ensure that you have the latest version, run repobee manage upgrade, but the preview commands were released in August 2021 so if you installed at any point after that you'll have them.

How do I set that environment variable?

It depends on your enviornment. In bash or any similar shell you'd run export REPOBEE_4_REVIEW_COMMANDS="true". You can verify that they are active by running any RepoBee command and you'll get a warning about using an unstable feature:

[~] $ repobee --help
[WARNING] Activating preview feature REPOBEE_4_REVIEW_COMMANDS

from repobee.

Thanks very much, @slarse!

I will set the environment variable on the next assignment and see how it goes.

from repobee.

Cool! I'll be very interesting in hearing how it goes.

from repobee.