Screen Version

On-Line Survey 2012

Survey ID 1319 Title COMP9242 12 Description Course Evaluation Survey for COMP9242 Advanced Operating Systems. Version for Session 2, 2012. Anonymous Yes Fill Ratio 80% (12/15) # Filled 12 # Suspended 1 # Not Filled 2

indicates required field

Your comments will help us to assess and improve our courses, not only for future generations, but for your further study in CS&E. We really look at the results and appreciate your feedback! Several changes to the course over the years were a direct result of student feedback. And, as always, we'll publish the uncensored results on the course web site. Note: Please do not enter "no comment" or something similar into comment boxes. If you don't have anything to say, just leave the box empty.

1. Quick Evaluation

1. Give a high rating if you have a good opinion of something (e.g. interesting, useful, well-structured, etc.). Give a low rating if you have a bad opinion of something (e.g. too slow, confusing, disorganised, etc.)   Question type : Single answer -- Radio Button   Excellent Satisfactory Poor Gernot Heiser 11 (92%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Kevin Elphinstone 11 (92%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Guest lecturer Leonid Ryzhyk 5 (42%) 3 (25%) 4 (33%) 0 (0%) 0 (0%) Guest lecturer Toby Murray 7 (58%) 3 (25%) 2 (17%) 0 (0%) 0 (0%) Guest lecturer Peter Chubb 6 (50%) 4 (33%) 2 (17%) 0 (0%) 0 (0%) Guest lecturer Ihor Kuz 7 (58%) 4 (33%) 1 (8%) 0 (0%) 0 (0%) Tutors/demonstrators 3 (25%) 5 (42%) 4 (33%) 0 (0%) 0 (0%) Exam 7 (58%) 2 (17%) 3 (25%) 0 (0%) 0 (0%) Course web pages 4 (33%) 5 (42%) 3 (25%) 0 (0%) 0 (0%) Reference material 3 (25%) 6 (50%) 2 (17%) 1 (8%) 0 (0%) Computing resources 5 (42%) 4 (33%) 3 (25%) 0 (0%) 0 (0%) COMP9242 overall 10 (83%) 2 (17%) 0 (0%) 0 (0%) 0 (0%)

2. General

2. Which factors most influenced your decision to enrol in this course?   Question type : Multiple answer -- Check Box Interest in operating systems as an area of study 12 (100%) Chance to build a system 8 (67%) Chance to get fingers really dirty 8 (67%) Would like to do some systems research 7 (58%) Looking for a challenge 11 (92%) Looking for an easy course 0 (0%) Friends told me it was good 3 (25%)

3. Other factors not mentioned above? Question type : Short-answer Answer at the bottom page (1 comments)

4. Would you recommend this course to another student such as yourself?   Question type : Single answer -- Radio Button Yes 12 (100%) No 0 (0%)

5. The course is heavy on design and implementation issues. It also tries to remain close to present research issues (although that aspect has suffered with the move to 12 teaching weeks). What do you think about the content allocation?   Question type : Single answer -- Radio Button   Too much Just right Too little Theory/general principles 0 (0%) 0 (0%) 10 (83%) 1 (8%) 1 (8%) OS design and implementation 0 (0%) 2 (17%) 9 (75%) 1 (8%) 0 (0%) Current research issues 0 (0%) 0 (0%) 12 (100%) 0 (0%) 0 (0%)

6. What were the best things about this course? Question type : Long-answer Answer at the bottom page (9 comments)

7. What were the worst things about this course? Question type : Long-answer Answer at the bottom page (10 comments)

8. How does the workload in this course compare to workloads in other ...   Question type : Single answer -- Radio Button   Much Lighter Similar Much Heavier COMP courses at this level 0 (0%) 0 (0%) 2 (17%) 6 (50%) 4 (33%) COMP courses in general 0 (0%) 0 (0%) 1 (8%) 1 (8%) 10 (83%) Courses in general 0 (0%) 0 (0%) 1 (8%) 1 (8%) 10 (83%)

9. How does the overall quality/value of this course compare to other ...   Question type : Single answer -- Radio Button   Among the best Average Among the worst COMP courses at this level 9 (75%) 2 (17%) 0 (0%) 1 (8%) 0 (0%) COMP courses in general 10 (83%) 1 (8%) 0 (0%) 0 (0%) 1 (8%) courses in general 10 (83%) 1 (8%) 0 (0%) 1 (8%) 0 (0%)

10. What background knowledge do you think you were missing that would have helped you in this course? Is distinction in COMP3231/9201 a suitable preparation? Is it too harsh? Question type : Short-answer Answer at the bottom page (8 comments)

3. Content/Syllabus

11. Please rate the relevance/appropriateness of the lecture topics.   Question type : Single answer -- Radio Button   Very relevant Average Inappropriate N/A Microkernels and seL4 11 (92%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Caches 9 (75%) 1 (8%) 1 (8%) 1 (8%) 0 (0%) 0 (0%) Threads and Events 12 (100%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Virtual Machines 8 (67%) 3 (25%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Microkernel Design 8 (67%) 3 (25%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Performance Evaluation 10 (83%) 2 (17%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Device Drivers 8 (67%) 3 (25%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) SMP and Locking 8 (67%) 3 (25%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Security 7 (58%) 4 (33%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Multiprocessors 2 8 (67%) 2 (17%) 2 (17%) 0 (0%) 0 (0%) 0 (0%) Linux Internals 7 (58%) 4 (33%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Linux Scalability 6 (50%) 5 (42%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Local Systems Research 7 (58%) 2 (17%) 3 (25%) 0 (0%) 0 (0%) 0 (0%) Sample paper analysis 8 (67%) 2 (17%) 2 (17%) 0 (0%) 0 (0%) 0 (0%)

12. Please tell us how interesting you found the lecture topics.   Question type : Single answer -- Radio Button   Very interesting Ok Boooooring! Skipped Microkernels and seL4 10 (83%) 2 (17%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Caches 8 (67%) 1 (8%) 1 (8%) 0 (0%) 2 (17%) 0 (0%) Threads and Events 7 (58%) 5 (42%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Virtual Machines 10 (83%) 2 (17%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Microkernel Design 10 (83%) 1 (8%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Performance Evaluation 7 (58%) 3 (25%) 1 (8%) 1 (8%) 0 (0%) 0 (0%) Device Drivers 5 (42%) 3 (25%) 3 (25%) 0 (0%) 1 (8%) 0 (0%) SMP and Locking 5 (42%) 3 (25%) 4 (33%) 0 (0%) 0 (0%) 0 (0%) Security 6 (50%) 4 (33%) 2 (17%) 0 (0%) 0 (0%) 0 (0%) Multiprocessors 2 8 (67%) 3 (25%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Linux Internals 7 (58%) 3 (25%) 2 (17%) 0 (0%) 0 (0%) 0 (0%) Linux Scalability 7 (58%) 4 (33%) 1 (8%) 0 (0%) 0 (0%) 0 (0%) Local Systems Research 6 (50%) 5 (42%) 0 (0%) 1 (8%) 0 (0%) 0 (0%) Sample paper analysis 6 (50%) 3 (25%) 2 (17%) 0 (0%) 0 (0%) 1 (8%)

13. Which material do you think will be most useful to you in the future?   Question type : Long-answer Answer at the bottom page (12 comments)

14. Which material, not currently in this course, would you liked to have seen covered? Question type : Long-answer Answer at the bottom page (6 comments)

15. Which of the current topics would you like to see scaled back or excluded? Question type : Long-answer Answer at the bottom page (3 comments)

4. Lectures

16. What factors caused you to attend lectures?   Question type : Multiple answer -- Check Box I had enough spare time 3 (25%) The lectures were too good to miss 10 (83%) Given the pace and lack of a textbook, I could not afford to miss the lectures 3 (25%) It was as good a place as any to take a nap 0 (0%) I wanted to be seen to be there 1 (8%) None, I skipped most 0 (0%)

17. What were the reasons for skipping lectures?   Question type : Multiple answer -- Check Box Overall workload in this and other courses 3 (25%) Lecture notes and references cover the material adequately 0 (0%) Lectures are boring 0 (0%) There was not enough material to justify attending lectures 0 (0%) First half of the course was more interesting than second half 0 (0%) None, I attended (almost) all 9 (75%)

18. Any suggestions for improving lectures? Question type : Long-answer Answer at the bottom page (4 comments)

5. Project

19. What was the level of difficulty various parts of the project?   Question type : Single answer -- Radio Button   Too easy Just right Too hard Milestone 0 2 (17%) 4 (33%) 6 (50%) 0 (0%) 0 (0%) Milestone 1 1 (8%) 3 (25%) 8 (67%) 0 (0%) 0 (0%) Milestone 2 0 (0%) 1 (8%) 10 (83%) 1 (8%) 0 (0%) Milestone 3 0 (0%) 1 (8%) 10 (83%) 1 (8%) 0 (0%) Milestone 4 0 (0%) 0 (0%) 12 (100%) 0 (0%) 0 (0%) Milestone 5 0 (0%) 0 (0%) 9 (75%) 3 (25%) 0 (0%) Milestone 6 0 (0%) 0 (0%) 5 (42%) 5 (42%) 2 (17%) Milestone 7 0 (0%) 0 (0%) 7 (58%) 4 (33%) 1 (8%) Milestone 8 0 (0%) 0 (0%) 9 (75%) 2 (17%) 1 (8%) System documentation 0 (0%) 1 (8%) 5 (42%) 4 (33%) 2 (17%) Project overall 0 (0%) 0 (0%) 8 (67%) 3 (25%) 1 (8%)

20. How well was the project specified?   Question type : Single answer -- Radio Button   Very clear Ok Confusing Milestone 0 7 (58%) 1 (8%) 4 (33%) 0 (0%) 0 (0%) Milestone 1 2 (17%) 3 (25%) 3 (25%) 2 (17%) 2 (17%) Milestone 2 2 (17%) 4 (33%) 5 (42%) 1 (8%) 0 (0%) Milestone 3 2 (17%) 4 (33%) 3 (25%) 3 (25%) 0 (0%) Milestone 4 2 (17%) 4 (33%) 4 (33%) 1 (8%) 1 (8%) Milestone 5 0 (0%) 0 (0%) 9 (75%) 2 (17%) 1 (8%) Milestone 6 0 (0%) 3 (25%) 6 (50%) 2 (17%) 1 (8%) Milestone 7 1 (8%) 4 (33%) 4 (33%) 1 (8%) 2 (17%) Milestone 8 1 (8%) 2 (17%) 5 (42%) 1 (8%) 3 (25%) System documentation 1 (8%) 2 (17%) 4 (33%) 4 (33%) 1 (8%) Project overall 2 (17%) 3 (25%) 6 (50%) 0 (0%) 1 (8%)

21. What was the quality of...   Question type : Single answer -- Radio Button   Excellent Ok Poor Documentation/reference material 1 (8%) 5 (42%) 4 (33%) 2 (17%) 0 (0%) Supplied code 0 (0%) 5 (42%) 5 (42%) 2 (17%) 0 (0%) Hardware platform 1 (8%) 5 (42%) 5 (42%) 0 (0%) 1 (8%) Consultation time help/support 5 (42%) 4 (33%) 2 (17%) 0 (0%) 1 (8%) On-line help/support 3 (25%) 4 (33%) 4 (33%) 0 (0%) 1 (8%)

22. Last year was the first year we used seL4 as the base for the project. Given the extreme low-level nature of the seL4 API and the risk inherent in any change, we were overly protective of students and provided a layer (object manager) which, in hindsight, hid too much of the interesting bits. This year we provided much more low-level library code and forced students to work more directly with the seL4 API. While you are obviously unable to compare with last year's setup, we'd like to hear what you think about the libraries we supplied, whether they gave you an appropriate level of abstraction, abstracted too much or too little. We are also interested in any other suggestions for improving the project. Question type : Long-answer Answer at the bottom page (11 comments)

6. Anything Else

23. Any other comments/suggestions that might help us to improve the course in the future? Question type : Long-answer Answer at the bottom page (7 comments)

3. Other factors not mentioned above? 1: Signaling

6. What were the best things about this course? 1: Designing an operating system from near-scratch, and learning about a cutting edge microkernel 2: implementing aos, papers, and exam 3: High quality lectures, coupled with challenging project material. 4: Get to build the low level parts of an OS and run it on real hardware. 5: Actually for real building an OS. As it says above - getting fingers REALLY dirty. 6: Interesting lectures presenting what is currently being done and how research got to that stage. Helpfulness of tutors. Exploring different ideas and concepts. Exam was engaging and a good test of knowledge. 7: Relaxed lecures that are very interesting, project is engaging and challenging 8: I learnt A LOT... 9: Excellent teaching quality. The guest lectures, which aren't really guest lectures, were really good. The second is the opportunity to work on a project for a semester that requires actual delivery.

7. What were the worst things about this course? 1: It is a bit time consuming. Also, this isn't really a problem with the course, but I feel like I didn't take away too much from the lectures (in terms of details). Normally I would either focus during semester (not now, too busy with project), or use stuvac to cement the course. Again, not specifically the course's issue. 2: Different tutors have different ideas aboout what each milestone requires. Repeatedly lead to inconsistent marking and at times avoidance of certain tutors. This was due to their expectation of functionality that, at times, was poorly defined or felt like an unreasonable expectation. Often marks were deducted for functionality that was part of the following week's work. 3: Try to get the tutors to know more about what's going on 4: It was somewhat unclear what was meant by some terms used in milestones, and the outcomes of the milestones were sometimes explained in insufficient detail. 5: hard project 6: If I had to pick something is that it is just incredibly time consuming. This is not a negative - I think all the students knew this when we enrolled, if not, it became clear very quickly. Perhaps consider 12UOC? 7: Confusing specification of what was expected in milestone demonstrations. Not so much about what was to be implemented but how it was to be demo'ed. Especially with the last milestone which specified certain content and tutors expected full system demonstration. Some of the guest lecturers were pretty poor/dull. Can not remember which ones. Content itself was not bad but delivery needed improvement. 8: With some of the milestones, the speck was vague, which is fine, but the marks was more on having specific feature working. eg: with file permissions 9: Nothing in particular. The required hours and energy is significantly more than what a 6 UOC course usually asks for, but that's expected. 10: Dealing with untyped memory and UT manager =P

10. What background knowledge do you think you were missing that would have helped you in this course? Is distinction in COMP3231/9201 a suitable preparation? Is it too harsh? 1: As a bio science student I didn't really know deep detail about cache and other hardware. Not something that hindered me too much in the end. Distinction in 3231 is good. 2: Background requirements are sufficient 3: D in 3231 is good, maybe some program design pre-req like 2911 though? designing a system this big from nearly scratch requires a lot of design/modulatiry and proper abstraction that isn't needed in the 3231 assignments 4: It's fine 5: The prereq is reasonable. 6: OS is a easy course to do well in, but low marks don't necessarily mean you can't succeed in AOS. The main requirement is clear thinking and problem solving ability. 7: Suitable 8: I feel like the main prereq. is the willingness to work very hard - and consistently - for the duration of the semester.

13. Which material do you think will be most useful to you in the future?   1: Mircokernel design and VM were very interesting topics of which i'm glad to have knowledge to utilize in the future. However the most useful was probably the lecture on performance evaluation and then also the sample paper analysis. 2: Too hard to isolate a specific thing. The entire course as a whole was invalueable 3: Threads + Events, VMs 4: VM, Microkernel design and Linux internals. 5: Skills learnt in the project. Strong OS basics. 6: I think the paper analysis will be the most useful. I never really knew how to read papers before, and I find myself questioning articles etc. allot more now. 7: Microkernel stuff, Hypervisor/VM stuff, security stuff 8: Not sure really, although I am not intended to work in the operating system field, some of the techniques solving problems might be useful. Otherwise performance evaluation should be useful. 9: Threads and Events. 10: Systems paper analysis, perf. evaluation 11: microkernel design, cache, performance evaluation, locking, thread & event, virtual machine 12: Microkernels and seL4 and virtual machines, mostly due to their well established importances in the research community.

14. Which material, not currently in this course, would you liked to have seen covered? 1: Real-time systems. 2: Some coverage of non-L4 non-linux operating system intenals 3: None 4: Don't know. 5: Maybe a bit of integration with signals processing courses. 6: Filesystems

15. Which of the current topics would you like to see scaled back or excluded? 1: Lecture on caches was mostly repeated stuff from a wide range of CSE courses. Could be sped up. Also some of the locking was repetitive and dull. 2: All good. 3: None

18. Any suggestions for improving lectures? 1: None 2: Some of the guest lecturers need to improve delivery. Gernot and Kevin were great though. 3: Three hours lecture is in generally tough. I don't see how it can be done better given the scopes of the lectures though. 4: Split them up I guess, 3 hours is a long time to pay continuation attention to a speaker

22. Last year was the first year we used seL4 as the base for the project. Given the extreme low-level nature of the seL4 API and the risk inherent in any change, we were overly protective of students and provided a layer (object manager) which, in hindsight, hid too much of the interesting bits. This year we provided much more low-level library code and forced students to work more directly with the seL4 API. While you are obviously unable to compare with last year's setup, we'd like to hear what you think about the libraries we supplied, whether they gave you an appropriate level of abstraction, abstracted too much or too little. We are also interested in any other suggestions for improving the project. 1: it's good, but maybe a little bit less abstraction. 2: Project content itself was sufficient. Otherwise, see worst things about this course. 3: I did like the level of abstraction we were given. Once you get your head around sel4, it isn't too bad... 4: I liked it. Sorry for the short response. 5: It is a tough call. I enjoyed working at the higher level (any higher would have been crap however), I don't think anyone could get away from actually understanding down to the lower level based on the way everything was this semester. The seL4 interface is VERY low level and keeping in mind that students are coming from regular OS, it will be very new - it look me all semester to understand the low level. Having said that, exposing more of the lower level can't be a bad thing. 6: Libraries were generally pretty good. They provided the right level of abstraction. However they could be better specified in some cases to what they do and how they go about it. 7: the provided libraries are good sample code sometimes, but most of them are not well-documented. They're difficult to read. libserial limits the implementation of SOS, unless rewrite it. It might be interesting if some architecture specific stuff added to the project. 8: Lots of code (and therefore lots of bugs) was attributed to the management of caps. The learning curve for caps leveled off after a few weeks of using them, and their management became a chore. 9: The ut_manager requires the memory to be 16KB aligned, before it allocates object. This policy intensifies the external fragmentation. The capability library hangs the system if the ut_alloc fails. It should return error instead. 10: Using the seL4 API with no om was a strange level of abstraction, it felt so far away from bare-metal, but it when anything went wrong it was nearly impossible to debug. The documentation tells us to look in the IPC MRs for error information, but the auto-generated syscall methods use custom MRs for IPC, meaning there is no error info in our MRs when we look, in addition, these were difficult to modify, there was no documentation about the auto-gen syscall stubs and a bunch of times I tried to modify them, they were just generated from the xml again... Similarly the documentation lacked in many areas (is 1 a high process priority or low?), and when we ran into massive seL4 errors due to the order in which we deleted process caps, attempting to debug this just lead to more and more seL4 internal assertion failures.... Really I think the main way om would have been nicer was it's management of seL4 VM stuff, it was difficult to manage UT memory well, and errors relating to Page Tables missing and etc were difficult to debug even when the syscall stubs were modified. 11: It was reasonable to deal with most of the time. I do have two complaints. Getting error messages was unreasonably hard, even in cases where seL4 does give error messages. The second one is the seL 4 manual sometimes lack details. For example it doesn't say that the CSpace must be deleted after the TCB is deleted.

23. Any other comments/suggestions that might help us to improve the course in the future? 1: If the project is lighter, I can read more papers. 2: I felt there were inconsistencies between markers each week. We would get full marks one week, thinking everything was fine. The next week, we'd get a different marker who didn't like our design, even though we were given the OK the week before. Obviously, the "implementation details" should be left out of the assignment spec, because that is often the fun/interesting part - but it would be nice if there were more concrete requirements placed on the "final product" of each milestone. Sometimes it was unclear how much (or how little) was expected, and again it would differ between which marker you got. 3: Tutors were exceptional and aided greatly in what was a difficult but fun course. Only wish would of been to be able to see and explore the sel4 source code. 4: None 5: Having the ability to launch applications onto different serial ports was very handy for testing, and reasonably easy to do (well... it is easy to do once you throughout libserial and write it from scratch). I would recommend having this as part of a milestone. Also I would of loved another week working on it... but then again, the end of semester is when other subjects are due in... 6: It's a great course, easily one of the best I've ever done =) maybe calling it 6 UoC is a bit low? 12 UoC would be more reasonable possibly. 7: Between Gernot and Kevin as well as the fantastic tutors, the course is in safe hands.