This page provides detailed information on the third experiment in our experiment series. Additionally, the raw data and analysis scripts for the first two experiments are given here.

Experiment 3: Effort for creating and reusing PCM performance models

Design

Participants

We asked 2 student research assistants, 1 student working on a Master thesis, and 1 first-year PhD student. We asked the participants for their programming experience in years and how many PCM models they have built before. The results below are not ordered by the partipants, but each entry is ordered by values (i.e. you cannot deduce that the participant with 12 years programming experience has created the 1 PCM model).

Programming experience: 3, 8, 10, and 12 years
PCM Models created: 0, 0, 0, and 1 model

All participants had previously heard presentations and other information on the method PCM.

Preparation

The preparation task description is available at [1]. The participants used that wiki page for preparation.

The material includes an step-by-step example, and two training examples. The participants were asked to model the second training examples on their own (without reuse), the other example were optional. However, they could only ask questions on the method in general and the optional examples. Thus, by completing the second training example on their own they showed that they had become familiar with the tool.

Experiment Tasks

First, the participants received a sheet describing the first half of the task. Only after they completed that part, they received the second sheet with the second half of the task.

First half of the task: Create the PCM models for the components Cache, Shop, and Watermarking: Media:Mobppexp_ExtendedMediaStore_Part1.pdf‎
Second half of the task: Build a system with all components (the ones created before and the pre-existing components AudioDB, Billing, Encoding, and Store) Media:Mobppexp_ExtendedMediaStore_Part2.pdf‎

Results

Data

Duration for creating / reusing components: Media:Reusestudy-time.tsv.txt
Complexity of the components: Media:Reusestudy-complexity.tsv.txt

Analysis scripts

R script for the linear model: Media:Reusestudy.R.txt
R script for the descriptive analysis: Media:Reusestudy-descriptive.R.txt

Experiment 1

Results

Data

Predicted mean response times: Media:Koziolek-times-predicted.tsv.txt‎
Measured mean response time: Media:Koziolek-times-measured.tsv.txt

Scripts

R script for accuracy analysis: Media:Pooled-times.R.txt

Experiment 2

Results

Data

Predicted mean response times: Media:Martens-times-predicted.tsv.txt‎
Reference mean response time: Media:Martens-times-reference.tsv.txt
Rankings created by the participants: Media:Martens-ranking.tsv.txt

Scripts

R script for accuracy analysis: Media:Pooled-times-exp2.R.txt
R script for effort analysis: Media:Duration-exp2.R.txt