Kiran Bhadury, Vishal Chakraborty, Francois Demoullin, Norm Matloff, Monte Misa, Rylan Schaeffer, Tiffany Ching L Yuen

Thong Le, Darya Semyonova

• What is OMSI?
• Benefits for students
• Benefits for instructors
• How to use this package
• Software tools for grading
• Future enhancements

OMSI, short for Online Measurement of Student Insight, is a software tool for conducting and grading examinations in a manner that is both secure and conducive to high-quality measurement of student insight. It is suitable for small or large class exams, be they based on essays, writing code, math analysis or multiple choice questions.

Students come to the classroom at the regular class time, just as with a traditional pencil-and-paper exam. However, they use their laptop computers to take the exam, using OMSI. The latter downloads the exam questions, and enables the students to upload their answers.

As detailed below, this arrangement has significant benefits for both students and instructors. The system is easy to install and use, and can be run on Macs, Windows and Linux.

Here is a screenshot of an example screen from the student's point of view:

OMSI will often result in your getting a higher grade in an exam than you would have in the traditional pencil-and-paper format:

• With essay questions, you have a chance to edit your answers, producing more coherent, readable prose. No cross-outs, arrows, words squeezed in above a line, no points off for unreadable handwriting. :-)
• With coding questions, you can compile and run your code, giving you a chance to make corrections if your code doesn't work.

In both of these aspects, OMSI gives you a better opportunity to demonstrate your insight into the course material, compared to the traditional exam format.

OMSI facilitates exam administration and grading. It has two components:

• Exam administration. This manages the actual process of the students taking the exam. You get electronic copies of the students' exams, eliminating the need for collecting and carrying out out a large number of papers, and making work sharing much easier among multiple graders. As noted in "Benefits for students" above, OMSI enables the student to turn in a better product, and this benefits the instructor as well: Better exam performance by students is both more gratifying to the instructor and also makes for easier, less frustrating grading.
• Exam grading. OMSI does NOT take the place of instructor judgment in assigning points to exam problems. But it does make things much easier, by automating much of the drudgery. For instance, OMSI automatically records grades assigned by the instructor, and automatically notifies students of their grades via e-mail. Gone are the days in which the instructor must alphabetize the papers, enter the grades by hand, carry an armload (or boxload) of papers to give back to students in class, retaining the stragglers not picked up by the students, and so on.

Contents:

• Terminology
• Installation (instructors and students)
• Directions for students
• Directions for instructors

The Linxu term directory is synonymous with folder for Mac and Windows.

The term terminal window from Mac and Linux is synonymous with Command Prompt for Windows (cmd).

You will need Python installed on your machine. For Mac and Linux systems, it should already be there, but it is easily downloadable for Windows.

You must have Python in your executable search path. To check this, open a terminal window and type

python

into a terminal window. Python should start, displaying the >>> prompt.

To install OMSI, download the .zip file. Unzipping it will produce a directory/folder omsi-master, where all relevant files reside.

Connecting to the server:

Download and install the package as above. Then change to the omni-master directory, and run

python OmsiGui.py

from a terminal window.

Then connect to the server and get the exam questions by selecting File | Connect. State your student e-mail address and the host and port provided by the instructor.

After you connect to the server, the exam questions will be downloaded to your machine.

NOTE: There should be two separate windows on the right of the GUI, one on top for the question prompt and one below to write the answer. The boxes are resizable and the question box may default to occupying the entire right side of the screen on some systems. If this is the case grab and drag the bottom of the box to resize the bottom window.

Saving answers:

Click the question number on the left of the OMSI screen, and select File | Save.` Note that saving is NOT submitting. Saving your file does NOT upload your answer to the instructor's server; the save operation merely saves to a file on your local machine. See "Submitting your answers" below for uploading to the instructor.

Running code:

Save the code first, then select File | Run. A new window will pop up, displaying the results.

Submitting answers:

Submit the answer to a particular question by clicking on the question number on the left side of the OMSI screen, and selecting File | Submit. This will upload your answer to the instructor. You can also submit all answers with File | Submit All. A dialog box specifying whether submission was successful will then be displayed.

Tips:

Be sure to try OMSI in an actual session before your first OMSI-based exam. Start up the server, playing the role of the instructor (see "Directions for instructors" below), and then start the client, playing the role of the student as above.

Make sure your laptop is fully charged before the exam.

Again, remember that saving an answer does NOT submit it. You must do that separately.

When you begin the exam, your very first action should be to click on Description. The instructor may have directions there that will make a night-and-day difference to your grade! So first take a few seconds to read this.

Python and R won't be running in interactive mode here, so you must write explicit ##print## operations. Graphical displays will not work, for the same reason.

When you submit an answer, make sure the Success! window then pops up to confirm receipt.

Within the directory omsi-master, there will be a directory InstructorDirectory. You place your exam questions in that directory (sample files are included there). The format for specifying the questions is detailed below.

During the exam, a subdirectory of InstructorDirectory will be created for each student, using the e-mail address provided by the student. In a student's subdirectory there will be an answer file for each question. e.g. omsi_answer1.txt or omsi_answer2.java

At the start of the exam period (not before), start the server from a terminal window by issuing the command

python OmsiServer.py [portNumber] [quoted exam name]

from within the omsi-master directory, e.g.

python OmsiServer.py 5000 'Fall 2014 Midterm 1'

The port number must be above 1024.

The server Internet address and port number will need to be distributed to the students at the start of the exam.

At present, this is simply done via ctrl-C.

Before the exam, the exam questions must be placed in a file called Questions.txt in the InstructorDirectory. The file should contain a description and the questions for the exam. If there are notes the instructor would like to write to him/herself in the file they can be placed at the beginning, in the section labeled NEW.

When parsing the file, OMSI will go through line by line and search for keywords NEW, DESCRIPTION or QUESTION. The roles are as follows:

• NEW: Optional. The lines that follow will contain private notes for the instructor.
• DESCRIPTION: Required. The lines that follow will contain instructions to students, which would normally go on the front page of a printed exam. The students will be able to view it by clicking Description in the menu.
• QUESTION: Have one of these for each exam problem. For problems involving code, directions for compiling or running the code go on this same line. The following lines contain the question, to be viewed by the students.

Example Questions.txt file:

DESCRIPTION

Students: Remember that Python and R won't be running in
interactive mode here, since you must call 'print' explicitly.

QUESTION -ext .py -run "python omsi_answer1.py"

Write a function half() that will return x/2, and 
freestanding code will print half(3).

QUESTION 

What does 'D' stand for in "UCD"?

In this example, two exam questions are defined, one requiring Python code and one requiring an essay. When a student writes and submits the answers, they will be saved in files omsi_answer1.py and omsi_answer2.txt in InstructorDirectory, in that student's subdirectory.

The suffix in that first file name arises from the specification -ext .py in the QUESTION line; otherwise the default suffix is .txt,

Specifying compile and run options

The compile and run functionalities are by default disabled. If any question requires compile and/or run then the -com and -run flags may be specified in a simillar manner as the -ext file while adding the QUESTION keyword, e.g.

QUESTION -com gcc -flags "-Wall -g" -run ./a.out

Tips:

For instructors making use of compilers and interpreters, it is suggested that the instructor have the students launch the client from a machine on which these are installed, say having them log in remotely using ssh -Y for a client running on a Unix-family machine.

OMSI should greatly improve the examination process for both instructors and students. However, as with any automated system, some care is needed to insure that things run smoothly.

It is crucial that the instructor, before the first OMSI-based exam, require the students to go through a dry run. The instructor may wish to incentivize this.

"Stuff happens." What if, say, there is a sudden power failure or the local WiFi access point suddenly goes out? Some students will not go through the dry run, even with major incentives/disincentives, and will have trouble using OMSI in the first exam. The following are worth considering:

• Paper copies of the exam prompt should be distributed at the start of the exam, and students should have the option of turning in their answers on paper. Some students may not even have laptops. (See note on tablets below.)

• Students should be told to click the Submit option "early and often." (Each submission would overwrite the last.) This is especially important if there is concern that the local WiFi access point might be flooded with traffic at the end of the exam period, as many students turn in last-minute work.

• Students should bring a USB key (thumb drive, memory stick). In an emergency, they can copy their local copies of their omsi_answer* files to the key, then turn in the key.

And what about cheating and other forms of academic dishonesty? In my experience, the vast majority of students are honest, and they resent the cheaters. What can be done about the latter?

• The OMSI window is designed to fill the student's entire screen. This prevents a student from communicating with others during the exam. The students must be given strict instructions not to move, iconify or resize OMSI window during the exam. This is easily monitored by proctors.

• As usual, there is the problem of a student looking at his/her neighbor's work, in this case a laptop screen rather than a piece of paper. A common technique for dealing with this in the paper exam case is to require students to sit in specific seats that are randomly assigned to them. This can be done with OMSI as well.

The answers submitted by the students and collected by the server may be graded by hand as usual. However, tools to facilitate electronic grading are available in this package. As noted earlier in this document:

OMSI does NOT take the place of instructor judgment in assigning points to individual exam problems. But it does make things much easier, by automating much of the drudgery. For instance, OMSI automatically records grades assigned by the instructor, and automatically notifies students of their grades via e-mail. Gone are the days in which the instructor must alphabetize the papers, enter the grades by hand, carry an armload (or boxload) of papers to give back to students in class, retaining the stragglers not picked up by the students, and so on.

The basic idea is that the software will display on the screen, for each student and each exam problem, the student's answer. In the case of coding questions, the software will also run the code and display the result. In each case, the instructor then inputs the number of points he/she wishes to assign:

for each student:
   for each exam problem:
      if problem involves coding:
         compile and/or run code according to the
            QUESTION line of Questions.txt, displaying result
      display student answer
      instructor inputs number of points 
   record grade for this student (individual problems and total)

The main tool is Grading/AutoGradeOMSI.R. This code is in R, as it is an adaptation from an earlier system I (NM) had developed; also, its statistical and graphic capabilities facilitate analysis of the exam grades. (There is also a file AutoGrade.py in that directory, but it is under development.)

Detailed directions are given in the comments at the top of the file.

Much improvement is possible. Contributors of new code are welcome!

Here are some points of improvement:

• Tablet-friendly version of OMSI.

• Code to check a student's e-mail address against a class list upon connection, with passwords being required.

• Code to confirm the integrity of the OMSI client code (i.e. verify that the source code hasn't been changed), and to upload each student Run operation to the server, to guard against the student running surreptitious communication code.