Homework 7 HT2022

Homework 7 - Planning an experiment

Due Nov 16 at 17:00

Start by reading:

Walter F. Tichy. Should Computer Scientists Experiment More? Computer, 31(5):32–40, 1998. 

TichyShouldExperiment1989.pdf.   Download TichyShouldExperiment1989.pdf.  

Additional suggested reading:

Matti Tedre and Nelli Moisseinen, Experiments in Computing: A Survey, Links to an external site. Scientific World Journal. 2014; 2014: 549398.

Peter J. Denning. ACM President’s Letter: What is experimental computer science? Links to an external site. Commun. ACM, 23(10):543–544, 1980.

D. G. Feitelson, Experimental Computer Science: The Need for a Cultural Change Links to an external site.

Then work through and submit the following assignment:

1. Introduce and state a research question that can be investigated in an experiment. It could be related to any field of computer science that has an experimental aspect, ranging from for example software development, programming languages, hardware performance, networks, computer security, and machine learning to visualization and interaction design. You are free to choose any question that you find interesting.

2. Find at least two peer-reviewed papers that are relevant to the suggested experiment, and relate them to your research question.

3. Formulate a hypothesis.

4.  Then operationalize your hypothesis, i.e., define all the concepts and variables involved clearly so that the hypothesis can be stated in a concrete and measurable way.

5. Suggest an experiment to test your hypothesis. Note that an experiment does not test the hypothesis itself but an outcome that can be deduced from it (the HD-method).

6. In particular, discuss the evaluation methods that will be used. For example, what kind of statistical evaluation will be done?

7. Describe the different requirements to actually carry out the experiment. These could range from access to data, legal permissions, funding, access to human subjects, access to special hardware to additional expertise that would be needed (or you would have to acquire).

8. The article by Walter Tichy lists several objections to experimentation in CS. Read the article and discuss whether any of the objections apply to your experiment. 

Describe each step above in your submitted answer. Your submission should follow general 
standards of academic writing.

Handing in your solution

Please save your solution as a pdf file and hand it both in Canvas (for grading) and Peergrade (for peer review). 

Peer grading

You will be asked to review the homework of two other students in Peergrade. Your solution will also be
reviewed in this way. The peer review is a mandatory part of the course. 

Feedback from your TA

Your seminar leader will grade your submission and report the result in Canvas. This may happen before
the associated seminar, but if your seminar leader is busy it will 

Complete means you have passed the assignment.

Incomplete means you have to hand in a revised version.

Fail means that you will have to submit a new version and attend the make-up seminar.
The Fail grade will only be applied in exceptional circumstances such as plagiarized work.