cal-cs-61a-staff / cs61a-apps Goto Github PK

(I will resolve this after Spring 2021, opening an issue to keep track of the TODOs)

Currently, various app names use hyphens instead of underscores, which makes them harder to document and also violates Python guides. Since these hyphens are replaced for database and RPC purposes anyway (and symlinks need to be created to special-case them for documentation), they should be replaced globally (and domains should be updated for the 4/12 apps that use hyphens in their domain).

This is not an atomic change, and the following needs to happen (in this order):

1. Update buildserver to replace underscores with hyphens for default domains
2. Update the domains tool to handle the above
3. Update existing links (likely need to redeploy the affected apps)
4. Update the PR proxy in hosted to handle the domains
5. Verify that Cloud Storage buckets are unaffected
6. Verify that database users are unaffected
7. Rename apps
8. Remove old Cloud Run deploys once affected apps are redeployed
9. Redeploy statically served apps, since static-server is among the affected apps
10. Make sure the CS 61A, CS 162, and CS 170 websites stay live

Affected apps: ag-master, ag-worker (domain update needed), cs162-website, cs170-website, exam-admin, exam-server, exam-write, hog-calc, hog-contest (domain update needed), ok-help (domain update needed), static-server (domain update needed), website-base. grade-display is also affected, but will be deleted before this happens anyway.

Since this affects course websites and examtool, I will start this after grades are due for SP21, and finish it before any courses need to use examtool for SU21 (or before 61A starts for SU21, whichever comes first). Shouldn't take that long, though.

In SQL, 3/2 and 3.0/2 are treated differently, but since the code.cs61a.org SQL interpreter is written in JS, this functionality isn't properly reflected in the step-by-step visualizer. The fix here is to wrap all values in a JS object that includes their type, and to write logic around this type. This fix is a bit too involved to be implemented this close to the SU21 final, as things might break if something goes wrong, but it should be fixed soon after.

When using CONFIG PICK, a student is presented with a randomly selected subset of questions.

examtool substitutions should list which questions were selected; or there should be some other command-line option to list that. This helps know what a student saw, when investigating possible academic misconduct.

Suppose I answer a multiple choice question. Then I try to scroll down using the down-arrow on my keyboard. This doesn't scroll the page; it changes my multiple-choice selection.

I discovered this had invisibly changed my selection on a question. I answered a multiple-choice question, scrolled down with the "page-down" key (moving the question I answered off the screen), then pressed the "down-arrow" key to scroll down a tiny bit more, and the screen scrolled down a tiny bit more. What I didn't realize is that my answer to the previous question had also been changed by the down-arrow -- in addition to scrolling the screen, it also invisibly changed my answer -- and I could not see the change, since it happened off-screen, so I had no way to detect it.

This is bad, because it can cause a student to lose points on a question they answered correctly, without realizing it.

I've been told that examtool cheaters only reports the first instance of cheating on each exam. The problem is that, if this first instance happens to be a false positive, then we fail to detect any other cheating on the exam. Depending on exam design, false positives might not be unusual.

It would be helpful to report all instances of cheating.

When using # DEFINE <target> ..., it appears the target is case-insensitive (e.g., target VAR is the same as var and Var). This should be documented in exam-write/Readme.md.

Copied from Cal-CS-61A-Staff/examtool-web#16:

For some cheating investigations, it would be useful to know whether the student has pasted anything into the page. This could plausibly be done by hooking the paste event and logging each paste event, along with what text was pasted. This would not be foolproof but it might be helpful in detecting or investigating some cheating cases.

Especially in courses such as CS186 and CS162 where students are required to fill in large tables or code blocks, having the ability to split the exam window into 2 panes (one side with the question and one with the code/table input) would be very helpful.

With examtool 2.0.15, examtool cheaters reported a mismatch in one answer, where the answer used a different alternative from the one listed in the question. This appears to be correct, as verified with examtool substitutions and by looking at their specific answer.

With examtool 2.1.7, examtool cheaters no longer detects anything for that student. This appears to be a false negative. I'm not sure how to narrow down why it failed to detect the problem. Any ideas on how to troubleshoot why it failed?

(Overall, examtool 2.1.7 produced about 6x less output from the cheaters subcommand than 2.0.15 did. I don't know how to assess that in aggregate, but I can verify the specific instance mentioned above.)

Copied from Cal-CS-61A-Staff/examtool-web#17:

For some cheating investigations, it would be useful to know whether the student has copied anything from the page (e.g., to share with others). This could plausibly be done by hooking the copy event and logging each copy event, along with what text was copied. This would not be foolproof but it might be helpful in detecting or investigating some cheating cases.

I have heard, but cannot confirm, that no alternative should be a substring of any other, otherwise it will trigger lots of false positives for the cheat detection. For example, # DEFINE VNAME Customer CustomerName is apparently no good. Is this correct?

If this is correct, this should be documented in exam-write/README.md. The restrictions on choosing targets and alternatives needed to ensure the cheat detection works well should be documented there.

I have one student where examtool logs shows no output for them (beyond their email address), and examtool save-logs doesn't download any logs from them, but I do have a submission from them, and examtool submissions does work. How do I diagnose what's gone wrong?

The exam is data8-fa20-mt-regular and the student with missing logs is the last student alphabetically (i.e., email address is [email protected]).

Feature suggestion: When using #DEFINE <target> <alt1> <alt2> ..., it is not clear how to define an alternative that contains spaces in it. It would be handy to be able to support that, and for that to be documented.

Copying from Cal-CS-61A-Staff/examtool-web#14: https://exam.cs61.org/ loads https://apis.google.com/js/platform.js, which is blocked by some ad blockers, causing the page to not load. I had multiple students in Data 8 report this issue. It would be useful to load that script from elsewhere, so the page isn't blocked by ad blockers.