Screen Version

Course Surveys 2011

Gernot's Comments on CATEI Survey

This was impressively positive and at least as good as previous years. However, with a return ratio of 47%, not too much can be read into this.

Free-form comments

Nice to see all the positive comments.

The improvement suggestions all appear in my detailed survey, I will comment on them there.

Gernot's Comments on LiC Survey

Many thanks to all students for taking the time to answer the course survey. This year we achieved a 95% return rate!

The results mostly speak for themselves. The course seems to be in good shape, overall satsifaction was high.

Repeatedly-made comments:

• The biggest gripe (by a huge margin) was the three-hour lecture block. This is a real problem to change, but I accept that it is a major pain for students. I'll see whether something can be done about it, but I'm not overly optimistic, given that we're at the mercy of centralised time tabling.
• Interestingly, the biggest gripe of previous years, documentation, has all but gone away, despite moving from a commercial platform back to a research system! This is a compliment to the NICTA folks who produced the docs.
• The question about whether om was a good thing or not had more or less the expected response: Yes, it would be good to get closer to the hardware, and yes, the project is hard enough as it is. Still, it is a shame that you don't get to deal with caps and many of the seL4 abstractions.
  We'll definitely re-consider this point for next year, and see whether we can provide some more minimalist library without making the project harder. For this year I'm glad we did what we did, as changing the whole setup is always a big risk. In the end, all groups got a working OS together (and almost all on time), I consider that a big success!
• The exam was also considered hard. I think this year it was a bit on the hard side, as both were SOSP papers (which are among the longest around and tend to have no faults that are easy to spot). However, I hadn't seen anything else recently that was a good fit to the course. Hopefully, next time is better.
  Having said that, people need to understand that this is an elite course and you can't expect things to be easy. It focusses on providing insights, and this is what the exam tests. And it's the reason why having passed AOS is known to help in the job market!
• Not too much lecture time on seL4 specifics, and milestones not too well specified: These are features! We want you to work things out for yourselves, including how exactly the kernel ticks, and what the best design is (and that includes forcing you to experiment).
  However, we will examine whether there is a mismatch between milestone specs and tutor expectations, if so we'll fix that. (But also notice that very rarely were demo marks docked, so I don't think this is a big issue.)
• No tutor access during week-ends and milestones due on Monday: Fair point, I'll consider having milestone demos due on Tue.
• Chairs in seminar room: This is a complaint (by a minority of students) which had been made last year already (but not in the previous 4 years we've been using that room). Maybe this is an indirect consequence of the 3-hour lecture block...

Relevant one-off comments:

• Not enough emphasis on research paper critical reading and evaluation during the semester
  We listed 1—3 relevant papers for each week's lecture, so you were clearly directed to relevant literature. I also always encourage people to read these papers, but inevitably hardly anyone does. I can't read them for you, so what do you expect us to do?
  What we can do in the future is to provide the relevant papers a week or so before the lecture, in the (probably vain) hope that someone will read them. Worth trying...
• Some of the specifications were far too vague, and were assessed on criteria that were not revealed to the students.
  It is a core feature of this course that we do not spoon-feed students. We intentionally provide only rough guidance, combined with extensive feedback by tutors. We consider this a central part of the course experience.
  As far as assessment goes: No-one's milestone demo got marked down because they chose a non-ideal design or implementation, but the tutors might have told them why their choice was bad and advised them to change it. This is very different from assessing based on undisclosed criteria, and something we will not change!
• Some more exciting drivers would be nice.
  I agree, but hard to fit in without making the project even harder. Also, the platform is limited in that it only provides trivial and complex devices, not much in between.
  With possibly a new hardware platform next year, we will revisit this point.
• Auto marking scripts are helpful to resolve conceptual problems
  Providing you with a test harness and test cases would only work if the milestones were defined much more narrowly. We do not want this, as constraining the project too much would compromise the learning experience as it would drastically limit your design space, and avoid having to think through all the trade-offs. This is core systems work.
  Furthermore, testing your system is your job, and when thinking about your design you should think about how to test it. Taking this off you would be counter-productive!
• Too much microkernels? Yes, the course is very biased towards microkernels, but that is design. Discussion about why they failed commercially? Well, in fact, they didn't! The majority of embedded multitasking systems are microkernel-based, and seL4 is likely to change things further in favour of microkernels.
• Too much overlap with normal OS: This comment pops up every now and then. I guess there are parts of the system which are similar to what you've done before, but would it be better if we supplied big slabs of code? (We all agree that we supply too much OM...)
• Om bugs: True, that's the risk of a new project. Should be better next year.
• Material such as audio, internals of other OSes, etc: Yes, would be nice to have, but we have limited time, and there's no consensus on what to drop. Also, what exactly we offer depends on the expertise we have in house, which varies (and we presently don't have a Windows or audio-driver expert). We really want to ensure that all the lecturers are highly competent in what they teach!

All up, that was a lot of useful feedback. Thanks to all of you!

Gernot