ASST1: Robots and System Calls

Overview

I'm writing essentially a blog post about marking of assignment 1, which might help answer some of the many questions we've had about it.

Writing and running the initial tests was pretty easy. We've communicated the results of the initial auto-test run to students and quite a few of you are a bit confused about what has happened. Here is a quick overview of things that have cost students marks or created admin complications.

The General Shape of the Tests and Output

In the spec we mentioned a few times that we would test your kernel and user code with a different version of the "rob161" simulated robot. We built each student submission with the "rob161" file replaced with a testing version.

The main difference in the testing version, by default, is that it traces its events via a separate logging API, as well as producing messages on the console. This avoids issues with debug kprintf commands conflicting with the test output.

The test version also knows how to vary the event sequences relevant to Parts 3 and 4. Some of the tests cover cases in which the subsystems are ready (for Part 3) or report completion (for Part 4) in a different order.

The testing "rob161" also uses a different notify thread model for Part 4, with one notify thread for each subsystem.

Typical Test Failures

We have spotted a few common patterns in student code that has failed a number of tests.

Concurrency in Part 4

The testing "rob161" has more notify threads than the provided one. If this causes the student synchronisation code to lock up, they may fail many or all of the Part 4 tests. We continue to consider whether there might be a way to tweak the marking scheme to give students more partial marks here.

Unfortunately in many of the cases where Part 4 fails there isn't a lot of evidence in the logs or console output what the issue is.

Early Termination in Part 4

A few students have wondered why the Part 4 logs are ending early with an "out of protocol" error when they have issued all the required frequencies. We've looked at a couple of these "async_scale" binaries and spotted that they finish their main control sequence a little too soon, once all the commands have been issued but before the final completion notification has been received. This can lead to the testing robot simulator objecting to an attempt to finish the test out of sequence.

More to Come

We are continuing to examine patterns in the testing output and will put more "hints" here of what the issues might be. Dealing with the issues below has taken up a bit of time that we would normally put into investigating this kind of thing.

Issues with Git and Submitted Versions

We understand that the setup with git, gitlab, git bundles and "give" creates a number of steps and room for confusion.

A number of students have apparently pushed their assignment 1 work to gitlab but not submitted via "give", or have submitted the wrong thing. After a flurry of emails and other enquiries on this topic, we did a scan, and found that 75 students (!?!) had pushed a most recent commit to gitlab that isn't present in the bundle they submitted with give. More on that below.

Marking on Gitlab

Could we always just mark the most recent gitlab commit? This would create a lot of ambiguity. Firstly, the gitlab system doesn't give us good timestamps for when it received a commit. The commit contains a timestamp, but these can be entirely forged. Secondly, when using branches, it may be unclear which branch to test. We've spent a bit of this week following up with various students who have had issues submitting the correct branch. We don't want to stop students using a branch/merge development approach, but this means we need students to clearly tell the system which branch they are trying to submit.

Checking Gitlab

Could we keep the "give" based submission system, but have it cross-check the state of gitlab? Alternatively, could we have students push to gitlab and just "give"-submit the name of the branch or checksum of the commit that they want to have tested?

Unfortunately this hits a snag with the on-submit tests. The test script that runs when a student submits with "give" runs using that student's account. That means the test script won't have access to gitlab. We have access to all the gitlab repositories we have created, but that requires our credentials. Each student has relevant credentials as well, somewhere, but we can't assume we know where they are in their account.

We need the "give" on-submit tests to keep working. Without them we could run into a scenario (which we have had in the past) where we had to follow up with many students who had submitted something slightly broken.

Changes We made to the Marking Process

The main issue we realised at our end was that our assignment 1 on-submit test permitted the student to submit any single branch. This was a slight departure from what had been done in the past, and checked in assignment 2. We also forgot about it and tried to test "main", the expected branch, which led to issues for a few students.

Marking Gitlab Pushes

Our scan detected 75 students who had something on gitlab not covered by what they had submitted to give. This is a big enough number to put some effort into intervening. We wrote some new code to fetch and test these submissions, and have compared the results. It turns out that roughly 50 of these test exactly the same. Maybe these students did some rebasing, or pushed their "doc.txt" to gitlab after submitting in give, or similar.

We are left with 5 students who didn't submit in "give" at all (but did push to gitlab), 3 who managed to submit the initial sources we sent (but pushed something sensible to gitlab), and 5 who had some further fixes on gitlab that would have got them some more points. We will correspond with all the above students eventually. We will treat the first two groups as having submitted, but impose a substantial late-equivalent penalty. The final group miss out, as the late-equivalent penalty adds up to more marks than the difference between the two versions.

It also seems that 3 students managed to re-submit assignment 1 (weeks late) while trying to submit assignment 2. We've never seen this before and have now managed to undo it.