__The domain and computer simulation learning environment__

**Introduction**The last row of the table contained what we called “conclusion starters”. These sentences were added to support
students in drawing conclusions from the table. Compared with the LOOK phase the students’ statements were more
precise.
Second, students were prompted to take a careful look at representations from the simulation such as formulae and
diagrams.

They were, e.g., asked to calculate the impedance XC for two values of the frequency ω1 and ω2 (with ω2 =
2ω1). Students were also asked to draw diagrams (e.g., the resistance diagram in Figure 1) for different values of the
frequency and draw conclusions.

Third, students were given “prediction-starters” to support them in thinking deliberately about the consequences of a
change, e.g., “when the frequency becomes higher, I think the output voltage will…..”

__1- Procedure__Students in both conditions had three weekly two-hour lessons on the subject of low-pass and high-pass filters,
which was part of their regular curriculum. The fourth lesson was used to administer the knowledge test. Class 1
participated in the study first, and a few months later Class 2 participated. The same procedure was followed for both
classes.

The same domain content was covered in both conditions.
In three two-hour sessions, students in the experimental condition went through the simulations of each of the three
filters. At the beginning of the first lesson, the experimenter introduced the students to the

SIMQUEST learning
environment. For the design task, the experimenter explained the three phases in the design approach and told the
students how to use the LED-Sheets. During the first lesson, students worked with the simulation of the first filter.

At
the end of the lesson, all LED-sheets were collected. At the beginning of the second and the third lessons, the LEDsheets were returned to the students and students continued where they had stopped the lesson before. Near the end

**2- Third phase:**DESIGN
In this phase, the main goal was to design an assignment about the observations made and the knowledge acquired
during previous phases. Students were supported in using this knowledge and making it explicit in their design.

In
generating a question, they were instructed to pose a question about the observations they had made. In formulating
the answer, they were advised to check the correctness of the answer with the help of the simulation.

In generating
the explanation for their assignment, they were advised to explain the answer in detail, and to make use of
calculations, representations, and observations. For each interface, except for the fourth one, students went through
the three LED phases.

**3- Knowledge test**Knowledge was assessed using a paper-and-pencil (post-)test. The knowledge test consisted of two parts: one set of
items intended to measure conceptual (insight) knowledge, and a second set of items focused on measuring
procedural (calculation) knowledge. All items were scored by a rater who was blind to the condition of the
participant who had taken the test.

Both the test and the answering key were developed together with the teacher.
Conceptual knowledge (insight into the cause-effect relations in the domain) was measured by items in which
students were asked to predict or explain the effect of a change. Students received points for correct answers and for
their reasoning.

**4- In the example shown in**Figure 2, the student not only had to choose a situation, but also had to give
a reason for their choice. There were a total of 28 conceptual items, with a maximum total score of 50 points; the
maximum point value per item depended on its complexity (13 items with a maximum of 1 point, 9 with a maximum
of 2 points, 5 with 3 points and 1 with 4 points).

Reliability analysis of the test resulted in a Cronbach's alpha of 0.80.
Two judges independently scored the answers to the conceptual knowledge items for ten percent of the data, with
inter-rater agreement reaching 0.70 (Cohen’s kappa).

Procedural knowledge was measured by test items in which students were asked to perform calculations. Students
received points for the calculation procedure and the correct answer. There were a total of 6 procedural items with a
maximum total score of 15 points; the maximum point value per item depended on the its complexity (1 item
question with 1 point possible, 3 with a maximum of 2 points, 2 with a maximum of 4 points).

An example of a
procedural item is presented in Figure 3. Reliability analysis of the test resulted in a Cronbach's alpha of 0.64. Two
judges independently scored the answers to the procedural knowledge items for ten percent of the data, with interrater agreement reaching 0.76 (Cohen’s kappa).
There were a total of nine introductory items, that were used to “warm up” the students. These items referred to
general domain knowledge and were not analyzed.

__5- Of the third lesson__Students were asked to have a look at the transfer functions of each filter (they were not supposed
to design assignments about transfer functions). At the end of the third lesson, LED-sheets were collected. For both
classes the students’ own teacher was available during all lessons to answer students’ questions.

In the control condition, students received three two-hour lessons, from their own teacher. They did not use a
computer simulation but received conventional instruction. The teacher used the blackboard for explaining the
domain and students completed calculation exercises from their textbook.

Informal observations of activities in the
class were made during all lessons by the experimenter.
Results
In the results section, we first present the exam scores for both conditions in each class, as a way to establish the
comparability of the experimental and control groups in terms of prior domain knowledge. Next, we present the
results of the knowledge post-tests. Finally, to gain understanding in the way the students used the scaffolds, we
analyzed students’ completed paper-and-pencil design sheets.

**6- Exam scores Table 2**Gives an overview of the mean exam scores on the subject of electricity for both conditions and for each
class. The exam scores (which could range from 1-10) for this subject were made up of the scores on a number of
tests from the students’ regular curriculum that they had taken before the experiment began.

The way these exam
scores were determined in both classes was not similar (both followed a different curriculum) but data for each
condition were normally distributed for both Class 1 (Experimental: Shapiro-Wilk W = 0.959, df = 11, p = .754 ;
Control: Shapiro-Wilk W= 0.969, df = 13, p = .754 ) and Class 2 (Experimental: Shapiro-Wilk W = 0.915, df = 10, p
= .316 ;

Control: Shapiro-Wilk W= .928, df = 15, p = .252). No difference between the experimental and control
condition was found (Class 1: F(1,22) = .623, p = .438) Class 2: F(1,23) = .439, p = .514). From this we may
conclude that the experimental and control condition entered the experiment with comparable prior knowledge.

**7- Because students came from two different**Classes with different backgrounds, we performed an analysis of the
results on the knowledge tests for the two classes separately.
Class 1. Table 4 shows the results of the knowledge tests for the two conditions in Class 1.

Statistical tests for
detection of outliers showed that one student in the experimental condition appeared to be an outlier for the
conceptual items (with a score greater than 2 SD below the mean score). This student was removed from further
analyses.

**conclusion**In both conditions results on the conceptual test remained normally distributed after the removal of the
outlier (Experimental condition: Shapiro-Wilk W = 0.939, df = 10, p = .542; Control condition: Shapiro-Wilk W
= .899, df = 13, p = .129). The test results for the procedural knowledge test were affected by the removal of the
outlier

(Experimental condition: Shapiro-Wilk W = .790, df = 10, p = .011; Control condition: Shapiro-Wilk W
= .950, df = 13, p = .597). Therefore, the non-parametric Kruskal-Wallis test was used to examine the differences
between conditions within Class 1. A significant difference between the two conditions was found on conceptual
items (H(1) = 5.044, p = .025) but not on procedural items (H(1) = -.117, p = .732).

## Post a Comment