hw7
Homework 7 - Planning an experiment
Due Nov 10 at 22:00
Required reading:
Walter F. Tichy. Should Computer Scientists Experiment More? Computer, 31(5):32–40, 1998.
Additional reading:
Matti Tedre and Nelli Moisseinen, Experiments in Computing: A Survey, (Links to an external site.) Scientific World Journal. 2014; 2014: 549398.
Peter J. Denning. ACM President’s Letter: What is experimental computer science? (Links to an external site.) Commun. ACM, 23(10):543–544, 1980.
D. G. Feitelson, Experimental Computer Science: The Need for a Cultural Change (Links to an external site.)
Assignment:
1. Introduce and state a research question that can be investigated in an experiment. It could be related to any field of computer science that has an experimental aspect, ranging from for example software development, programming languages, hardware performance, networks, computer security, and machine learning to visualization and interaction design. You are free to choose any question that you find interesting.
2. Find at least two peer-reviewed papers that are relevant to the suggested experiment, and relate them to your research question.
3. Formulate a hypothesis.
4. Then operationalize your hypothesis.
5. Suggest an experiment to test your hypothesis. Note that an experiment does not test the hypothesis itself but an outcome that can be deduced from it (the HD-method).
6. In particular, discuss the evaluation methods that will be used. For example, what kind of statistical evaluation will be done?
7. Describe the different requirements to actually carry out the experiment. These could range from access to data, legal permissions, funding, access to human subjects, access to special hardware to additional expertise that would be needed (or you would have to acquire).
8. The article by Walter Tichy mentioned above lists sixteen objections to experimentation in CS. Read the article and discuss whether any of the objections apply to your experiment.
Describe each step above in some detail in your submitted answer.
Handing in your solution
Please save your solution as a pdf file and hand it in BOTH here in Canvas as Homework 5a and on the Peergrade page. Do not write your name in the pdf file.
Note - this is a backup procedure to ensure that all your homework is graded by the TAs, since it is likely that Peergrade does not yet work for all students.
Peer grading
You will be asked to review the homework of three other students. Your solution will also be reviewed in this way.
Feedback from your TA
Your TA will grade your submission and report the result in Canvas.
Complete means you have passed the assignment.
Incomplete means you have to hand in a revised version.