Screen Version

On-Line Survey 2013

Survey ID 1376 Title COMP9242 13 Description Course Evaluation Survey for COMP9242 Advanced Operating Systems. Version for Session 2, 2013. Anonymous Yes Fill Ratio 79% (15/19) # Filled 15 # Suspended 3 # Not Filled 1

indicates required field

Your comments will help us to assess and improve our courses, not only for future generations, but for your further study in CS&E. We really look at the results and appreciate your feedback! Several changes to the course over the years were a direct result of student feedback. And, as always, we'll publish the uncensored results on the course web site. Note: Please do not enter "no comment" or something similar into comment boxes. If you don't have anything to say, just leave the box empty.

1. Quick Evaluation

1. Give a high rating if you have a good opinion of something (e.g. interesting, useful, well-structured, etc.). Give a low rating if you have a bad opinion of something (e.g. too slow, confusing, disorganised, etc.)   Question type : Single answer -- Radio Button   Excellent Satisfactory Poor Gernot Heiser 14 (93%) 1 (7%) 0 (0%) 0 (0%) 0 (0%) Kevin Elphinstone 11 (73%) 4 (27%) 0 (0%) 0 (0%) 0 (0%) Guest lecturer Ihor Kuz 5 (33%) 8 (53%) 2 (13%) 0 (0%) 0 (0%) Guest lecturer Toby Murray 5 (33%) 9 (60%) 1 (7%) 0 (0%) 0 (0%) Guest lecturer Peter Chubb 8 (53%) 5 (33%) 2 (13%) 0 (0%) 0 (0%) Tutors/demonstrators 11 (73%) 4 (27%) 0 (0%) 0 (0%) 0 (0%) Exam 6 (40%) 7 (47%) 0 (0%) 1 (7%) 1 (7%) Course web pages 5 (33%) 3 (20%) 5 (33%) 2 (13%) 0 (0%) Reference material 2 (13%) 5 (33%) 7 (47%) 1 (7%) 0 (0%) Computing resources 6 (40%) 5 (33%) 4 (27%) 0 (0%) 0 (0%) COMP9242 overall 11 (73%) 4 (27%) 0 (0%) 0 (0%) 0 (0%)

2. General

2. Which factors most influenced your decision to enrol in this course?   Question type : Multiple answer -- Check Box Interest in operating systems as an area of study 12 (80%) Chance to build a system 11 (73%) Chance to get fingers really dirty 12 (80%) Would like to do some systems research 2 (13%) Looking for a challenge 15 (100%) Looking for an easy course 2 (13%) Rupert told me to 12 (80%) Friends told me it was good 8 (53%)

3. Other factors not mentioned above? Question type : Short-answer Answer at the bottom page (1 comments)

4. Would you recommend this course to another student such as yourself?   Question type : Single answer -- Radio Button Yes 15 (100%) No 0 (0%)

5. The course is heavy on design and implementation issues. It also tries to remain close to present research issues (although that aspect has suffered with the move to 12 teaching weeks). What do you think about the content allocation?   Question type : Single answer -- Radio Button   Too much Just right Too little Theory/general principles 0 (0%) 0 (0%) 11 (73%) 4 (27%) 0 (0%) OS design and implementation 0 (0%) 1 (7%) 12 (80%) 1 (7%) 1 (7%) Current research issues 0 (0%) 1 (7%) 8 (53%) 6 (40%) 0 (0%)

6. What were the best things about this course? Question type : Long-answer Answer at the bottom page (10 comments)

7. What were the worst things about this course? Question type : Long-answer Answer at the bottom page (9 comments)

8. How does the workload in this course compare to workloads in other ...   Question type : Single answer -- Radio Button   Much Lighter Similar Much Heavier COMP courses at this level 1 (7%) 0 (0%) 0 (0%) 4 (27%) 10 (67%) COMP courses in general 1 (7%) 0 (0%) 0 (0%) 0 (0%) 14 (93%) Courses in general 1 (7%) 0 (0%) 0 (0%) 2 (13%) 12 (80%)

9. How does the overall quality/value of this course compare to other ...   Question type : Single answer -- Radio Button   Among the best Average Among the worst COMP courses at this level 12 (80%) 3 (20%) 0 (0%) 0 (0%) 0 (0%) COMP courses in general 13 (87%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) courses in general 15 (100%) 0 (0%) 0 (0%) 0 (0%) 0 (0%)

10. What background knowledge do you think you were missing that would have helped you in this course? Is distinction in COMP3231/9201 a suitable preparation? Is it too harsh? Question type : Short-answer Answer at the bottom page (8 comments)

3. Content/Syllabus

11. Please rate the relevance/appropriateness of the lecture topics.   Question type : Single answer -- Radio Button   Very relevant Average Inappropriate N/A Microkernels and seL4 8 (53%) 5 (33%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) Caches 10 (67%) 4 (27%) 1 (7%) 0 (0%) 0 (0%) 0 (0%) Threads and Events 10 (67%) 4 (27%) 1 (7%) 0 (0%) 0 (0%) 0 (0%) Performance Evaluation 11 (73%) 4 (27%) 0 (0%) 0 (0%) 0 (0%) 0 (0%) Microkernel Design 5 (33%) 9 (60%) 1 (7%) 0 (0%) 0 (0%) 0 (0%) Scalability and Linux Internals 5 (33%) 4 (27%) 5 (33%) 1 (7%) 0 (0%) 0 (0%) Security 2 (13%) 5 (33%) 8 (53%) 0 (0%) 0 (0%) 0 (0%) Virtual Machines 6 (40%) 5 (33%) 4 (27%) 0 (0%) 0 (0%) 0 (0%) Real-Time Systems 4 (27%) 5 (33%) 5 (33%) 1 (7%) 0 (0%) 0 (0%) SMP and Locking 4 (27%) 6 (40%) 5 (33%) 0 (0%) 0 (0%) 0 (0%) Multiprocessors 2, Drawbridge 5 (33%) 5 (33%) 5 (33%) 0 (0%) 0 (0%) 0 (0%) Sample paper analysis 4 (27%) 4 (27%) 6 (40%) 0 (0%) 0 (0%) 1 (7%) Local Systems Research 3 (20%) 4 (27%) 8 (53%) 0 (0%) 0 (0%) 0 (0%)

12. Please tell us how interesting you found the lecture topics.   Question type : Single answer -- Radio Button   Very interesting Ok Boooooring! Skipped Microkernels and seL4 8 (53%) 4 (27%) 3 (20%) 0 (0%) 0 (0%) 0 (0%) Caches 9 (60%) 4 (27%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) Threads and Events 9 (60%) 5 (33%) 1 (7%) 0 (0%) 0 (0%) 0 (0%) Performance Evaluation 5 (33%) 8 (53%) 2 (13%) 0 (0%) 0 (0%) 0 (0%) Microkernel Design 8 (53%) 6 (40%) 1 (7%) 0 (0%) 0 (0%) 0 (0%) Scalability and Linux Internals 8 (53%) 4 (27%) 2 (13%) 1 (7%) 0 (0%) 0 (0%) Security 4 (27%) 6 (40%) 4 (27%) 0 (0%) 1 (7%) 0 (0%) Virtual Machines 7 (47%) 3 (20%) 5 (33%) 0 (0%) 0 (0%) 0 (0%) Real-Time Systems 5 (33%) 6 (40%) 4 (27%) 0 (0%) 0 (0%) 0 (0%) SMP and Locking 6 (40%) 2 (13%) 6 (40%) 1 (7%) 0 (0%) 0 (0%) Multiprocessors 2, Drawbridge 5 (33%) 5 (33%) 3 (20%) 1 (7%) 0 (0%) 1 (7%) Sample paper analysis 1 (7%) 4 (27%) 7 (47%) 2 (13%) 0 (0%) 1 (7%) Local Systems Research 5 (33%) 3 (20%) 7 (47%) 0 (0%) 0 (0%) 0 (0%)

13. Which material do you think will be most useful to you in the future?   Question type : Long-answer Answer at the bottom page (15 comments)

14. Which material, not currently in this course, would you liked to have seen covered? Question type : Long-answer Answer at the bottom page (2 comments)

15. Which of the current topics would you like to see scaled back or excluded? Question type : Long-answer Answer at the bottom page (1 comments)

4. Lectures

16. What factors caused you to attend lectures?   Question type : Multiple answer -- Check Box I had enough spare time 9 (60%) The lectures were too good to miss 13 (87%) Given the pace and lack of a textbook, I could not afford to miss the lectures 5 (33%) It was as good a place as any to take a nap 3 (20%) I wanted to be seen to be there 6 (40%) None, I skipped most 0 (0%)

17. What were the reasons for skipping lectures?   Question type : Multiple answer -- Check Box Overall workload in this and other courses 5 (33%) Lecture notes and references cover the material adequately 0 (0%) Lectures are boring 0 (0%) There was not enough material to justify attending lectures 0 (0%) First half of the course was more interesting than second half 0 (0%) None, I attended (almost) all 11 (73%)

18. Any suggestions for improving lectures? Question type : Long-answer Answer at the bottom page (5 comments)

5. Project

19. What was the level of difficulty various parts of the project?   Question type : Single answer -- Radio Button   Too easy Just right Too hard Milestone 0 5 (33%) 5 (33%) 5 (33%) 0 (0%) 0 (0%) Milestone 1 0 (0%) 5 (33%) 9 (60%) 1 (7%) 0 (0%) Milestone 2 0 (0%) 5 (33%) 10 (67%) 0 (0%) 0 (0%) Milestone 3 0 (0%) 3 (20%) 10 (67%) 2 (13%) 0 (0%) Milestone 4 0 (0%) 4 (27%) 10 (67%) 1 (7%) 0 (0%) Milestone 5 0 (0%) 2 (13%) 8 (53%) 5 (33%) 0 (0%) Milestone 6 0 (0%) 1 (7%) 4 (27%) 7 (47%) 3 (20%) Milestone 7 1 (7%) 1 (7%) 8 (53%) 4 (27%) 1 (7%) Milestone 8 0 (0%) 4 (27%) 9 (60%) 2 (13%) 0 (0%) System documentation 0 (0%) 3 (20%) 10 (67%) 2 (13%) 0 (0%) Project overall 0 (0%) 0 (0%) 12 (80%) 3 (20%) 0 (0%)

20. How well was the project specified?   Question type : Single answer -- Radio Button   Very clear Ok Confusing Milestone 0 9 (60%) 2 (13%) 2 (13%) 1 (7%) 1 (7%) Milestone 1 8 (53%) 5 (33%) 1 (7%) 0 (0%) 1 (7%) Milestone 2 4 (27%) 5 (33%) 5 (33%) 0 (0%) 1 (7%) Milestone 3 5 (33%) 3 (20%) 5 (33%) 1 (7%) 1 (7%) Milestone 4 4 (27%) 5 (33%) 3 (20%) 0 (0%) 3 (20%) Milestone 5 4 (27%) 4 (27%) 5 (33%) 1 (7%) 1 (7%) Milestone 6 3 (20%) 3 (20%) 3 (20%) 5 (33%) 1 (7%) Milestone 7 4 (27%) 2 (13%) 6 (40%) 1 (7%) 2 (13%) Milestone 8 5 (33%) 2 (13%) 4 (27%) 1 (7%) 3 (20%) System documentation 5 (33%) 2 (13%) 3 (20%) 4 (27%) 1 (7%) Project overall 3 (20%) 5 (33%) 4 (27%) 2 (13%) 1 (7%)

21. What was the quality of...   Question type : Single answer -- Radio Button   Excellent Ok Poor Documentation/reference material 2 (13%) 2 (13%) 5 (33%) 5 (33%) 1 (7%) Supplied code 2 (13%) 3 (20%) 8 (53%) 2 (13%) 0 (0%) Hardware platform 8 (53%) 3 (20%) 4 (27%) 0 (0%) 0 (0%) Consultation time help/support 9 (60%) 4 (27%) 2 (13%) 0 (0%) 0 (0%) On-line help/support 6 (40%) 3 (20%) 6 (40%) 0 (0%) 0 (0%)

22. This year we introduced a new hardware platform, which required updates to the project and some of the supplied code/system. It also provided a potential for new breakage. Do you have any comments on the platform, or the readiness of the hardware and software? We are also interested in any other suggestions for improving the project. Question type : Long-answer Answer at the bottom page (11 comments)

6. Summer Internships

23. Each year we encourage the students in this course to do a summer internship with us (i.e. the Trustworthy Systems group at NICTA). Typically less than half take up the opportunity, and we'd like to understand why. Question type : Single answer -- Radio Button I am doing a ToR/NICTA Summer Internship with Trustworthy Systems 3 (20%) I am doing a ToR/NICTA Summer Internship with another group 2 (13%) I received an offer but declined 0 (0%) I applied but did not get an offer 2 (13%) I did not apply 7 (47%) N/F 1 (7%)

24. If you are not doing a ToR/Summer Internship with us, please let us know why. Question type : Long-answer Answer at the bottom page (9 comments)

7. Anything Else

25. Any other comments/suggestions that might help us to improve the course in the future? Question type : Long-answer Answer at the bottom page (5 comments)

Back to Summary

3. Other factors not mentioned above? 1: Johnny G told me to do it

6. What were the best things about this course? 1: Project Very friendly and helpful tutors 2: Project that had weekly goals and allowed you to build a system on work you had previously done, not common for university courses. Excellent teaching staff, lecturers and tutors answered all questions very helpfully and provided extra knowledge where it was needed. 3: Getting our hands dirty 4: * Being forced to use my brain. This course requires you to not only comprehend a topic but apply and adapt it to your needs. * Having a working little OS that you can be proud of and understanding fundamentally how it works from the ground up. * Learning through doing. 5: Got to build an operating system 6: The project involved a lot of work, but was ultimately very rewarding. 7: - I felt like I became a better programmer - Learnt a lot more about OS related topics - I got a chance to build a system running on real hardware - Consolidated my OS knowledge from basic course - Free beer 8: Chance to build something really challenging in a new context. OS development is hard to just jump in to. 9: hard. i am awesome now. 10: The satisfaction of completing a milestone, particularly the later ones.

7. What were the worst things about this course? 1: hard. debugging man... force us to learn tools earlier. 2: Not enough time. Would have preferred it to run longer 3: - it's a time sink - it's only 6UoC 4: Exam was a little haphazard. 5: It's very difficult to catch up once behind. 6: * Documentation was sometimes lacking * Milestones could have had more links to reference material. 7: Amount of hours required 8: Debugging the project Sometimes documentation was lacking Exam 9: Content seemed to jump around a lot.

10. What background knowledge do you think you were missing that would have helped you in this course? Is distinction in COMP3231/9201 a suitable preparation? Is it too harsh? 1: Background knowledge is suitable. 2: It was good. 3: Distinction seems fair, there 4: Fine 5: suitable. 6: I did not feel that my lower score in 3231 affected my ability to understand the content of this course. 7: yes 8: The course throws you in the deepend and sees if you can swim. I think that is a good approach and the filter is not too harsh.

13. Which material do you think will be most useful to you in the future?   1: Caches, smp/locking, security; VM when I get time to play around with virtualisation. 2: Real time systems. 3: "Is it.. caches?" VM, Multiprocessors, microkernel design (in terms of what it's trying to achieve and how), caching, threading v event loop v continuations, and performance analysis/benchmarking. 4: Knowledge about hypervisors, as well as design considerations around threads vs events. Workings of caches. 5: Caches, threads and events Knowing how VMs work is also pretty cool 6: Caches 7: Caches, Virtual machines, threads and events. 8: Microkernel design 9: systems development experience. 10: Microkernel design and Linux internals seem very useful 11: This is hard to say as I am not sure what I am going to be doing in the future. However caches and threads, SMP and Locking seem to most generally applicable 12: Understanding real life OS internals 13: Design of modern operating systems, and how to best take advantage of them when writing/designing user-space programs. 14: Concurrency and threading/event models for general computing. Performance evaluation and benchmarking for general life. 15: Threads/events, SMP and locking, virtual machines, scalability and linux internals

14. Which material, not currently in this course, would you liked to have seen covered? 1: Boot process (from hardware, not within seL4) 2: n/a

15. Which of the current topics would you like to see scaled back or excluded? 1: n/a

18. Any suggestions for improving lectures? 1: Have your own slides for threads vs events vs etc. The ones currently used don't flow smoothly with the lecture presentation due to the constant need to point out which aspects are out of date, etc. 2: Guest lecturers tended to run with only one or no breaks - preferred a break every hour to absorb the information. 3: Possibly increase the level of interactivity, offer more problems to be solved that relate to the task at hand to build better intuition of certain topics to keep it easy to stay engaged with the content. 4: More anecdotes and real life examples (war stories) always make content more interesting. 5: They're fine :)

22. This year we introduced a new hardware platform, which required updates to the project and some of the supplied code/system. It also provided a potential for new breakage. Do you have any comments on the platform, or the readiness of the hardware and software? We are also interested in any other suggestions for improving the project. 1: The Sabre seemed fairly solid. The plastic case could be flimsy at times. The loader works well. Overall seemed good. 2: Spurious interrupt! 3: it was pretty good actually. there were no breakages or really dodgy things happening. 4: Seemed to work well 5: There were a few snags with provided kernel blob; otherwise provided code is reasonably solid. Overall, quite good. 6: No issues here. 7: Was very frustrating to get the platform working at home. There were a couple of points where I had to reinstall drivers, and essentially each time I wished to use the board from home, it required a bunch of reconfiguration (even though I hadn't changed anything since previous use). However it was super easy to get going at Uni 8: The hardware was great I loved playing around with it. The case was a little bit dodgy and I constantly feared it smashing into 1 million bits. The hardware manual was off in some places like how to initialise the clocks. 9: UDP over serial sucks, badly - especially for trying to debug output. This was carried over from the old Slug platform. Alex K was talking about actually doing TCP here. Otherwise, hardware itself worked perfectly fine. My serial->USB was slightly broken though. 10: Serial port and carrier detect during bootloader screwed up frequently enough to be annoying. Everything fine once the OS actually booted. 11: It worked pretty well

24. If you are not doing a ToR/Summer Internship with us, please let us know why. 1: I applied for Taste of Research but got declined I would still like to do the internship though if that is arrangeable. 2: Graduating 3: I've signed a contract for the summer already. Next year, I am torn between NICTA and Google -- will talk to Kevin/Gernot re summer projects before making a final decision. 4: I already have a graduate job. 5: Interest in Driving Simulator project under Machine Learning group. Also, need a break from seL4. 6: Google internship 7: External employment over summer. I'd love to do a NICTA internship in future, if the opportunity arises. 8: I am doing an internship somewhere else (google) 9: I have an internship at another organisation, but I would've really liked the chance to work at NICTA.

25. Any other comments/suggestions that might help us to improve the course in the future? 1: Certain milestones could be fleshed out a bit more. However saying that the ambiguity forced creativity and really thinking about the scope of the problem which in retrospect is probably a better model of the real world. 2: The exam could have been deployed in a more timely manner. 3: One of the most challenging and rewarding courses of my degree. Reading papers is no longer scary. Thank you for treating us like competent individuals, not mindless dolts. This approach makes the course very valuable. 4: force debug tools learnt earlier. 5: Keep doing what you're doing!