forrja: The domain and computer simulation learning environment

The domain and computer simulation learning environment 

Introduction 

The last row of the table contained what we called “conclusion starters”. These sentences were added to support students in drawing conclusions from the table. Compared with the LOOK phase the students’ statements were more precise. Second, students were prompted to take a careful look at representations from the simulation such as formulae and diagrams. 

They were, e.g., asked to calculate the impedance XC for two values of the frequency ω1 and ω2 (with ω2 = 2ω1). Students were also asked to draw diagrams (e.g., the resistance diagram in Figure 1) for different values of the frequency and draw conclusions. 

Third, students were given “prediction-starters” to support them in thinking deliberately about the consequences of a change, e.g., “when the frequency becomes higher, I think the output voltage will…..”

1- Procedure 

Students in both conditions had three weekly two-hour lessons on the subject of low-pass and high-pass filters, which was part of their regular curriculum. The fourth lesson was used to administer the knowledge test. Class 1 participated in the study first, and a few months later Class 2 participated. The same procedure was followed for both classes. 

The same domain content was covered in both conditions. In three two-hour sessions, students in the experimental condition went through the simulations of each of the three filters. At the beginning of the first lesson, the experimenter introduced the students to the 

SIMQUEST learning environment. For the design task, the experimenter explained the three phases in the design approach and told the students how to use the LED-Sheets. During the first lesson, students worked with the simulation of the first filter. 

At the end of the lesson, all LED-sheets were collected. At the beginning of the second and the third lessons, the LEDsheets were returned to the students and students continued where they had stopped the lesson before. Near the end

2- Third phase: 

DESIGN In this phase, the main goal was to design an assignment about the observations made and the knowledge acquired during previous phases. Students were supported in using this knowledge and making it explicit in their design. 

In generating a question, they were instructed to pose a question about the observations they had made. In formulating the answer, they were advised to check the correctness of the answer with the help of the simulation. 

In generating the explanation for their assignment, they were advised to explain the answer in detail, and to make use of calculations, representations, and observations. For each interface, except for the fourth one, students went through the three LED phases. 

3- Knowledge test 

Knowledge was assessed using a paper-and-pencil (post-)test. The knowledge test consisted of two parts: one set of items intended to measure conceptual (insight) knowledge, and a second set of items focused on measuring procedural (calculation) knowledge. All items were scored by a rater who was blind to the condition of the participant who had taken the test. 

Both the test and the answering key were developed together with the teacher. Conceptual knowledge (insight into the cause-effect relations in the domain) was measured by items in which students were asked to predict or explain the effect of a change. Students received points for correct answers and for their reasoning. 

4- In the example shown in 

Figure 2, the student not only had to choose a situation, but also had to give a reason for their choice. There were a total of 28 conceptual items, with a maximum total score of 50 points; the maximum point value per item depended on its complexity (13 items with a maximum of 1 point, 9 with a maximum of 2 points, 5 with 3 points and 1 with 4 points). 

Reliability analysis of the test resulted in a Cronbach's alpha of 0.80. Two judges independently scored the answers to the conceptual knowledge items for ten percent of the data, with inter-rater agreement reaching 0.70 (Cohen’s kappa). 

Procedural knowledge was measured by test items in which students were asked to perform calculations. Students received points for the calculation procedure and the correct answer. There were a total of 6 procedural items with a maximum total score of 15 points; the maximum point value per item depended on the its complexity (1 item question with 1 point possible, 3 with a maximum of 2 points, 2 with a maximum of 4 points). 

An example of a procedural item is presented in Figure 3. Reliability analysis of the test resulted in a Cronbach's alpha of 0.64. Two judges independently scored the answers to the procedural knowledge items for ten percent of the data, with interrater agreement reaching 0.76 (Cohen’s kappa). There were a total of nine introductory items, that were used to “warm up” the students. These items referred to general domain knowledge and were not analyzed.

5- Of the third lesson

Students were asked to have a look at the transfer functions of each filter (they were not supposed to design assignments about transfer functions). At the end of the third lesson, LED-sheets were collected. For both classes the students’ own teacher was available during all lessons to answer students’ questions.

 In the control condition, students received three two-hour lessons, from their own teacher. They did not use a computer simulation but received conventional instruction. The teacher used the blackboard for explaining the domain and students completed calculation exercises from their textbook. 

Informal observations of activities in the class were made during all lessons by the experimenter. Results In the results section, we first present the exam scores for both conditions in each class, as a way to establish the comparability of the experimental and control groups in terms of prior domain knowledge. Next, we present the results of the knowledge post-tests. Finally, to gain understanding in the way the students used the scaffolds, we analyzed students’ completed paper-and-pencil design sheets. 

6- Exam scores Table 2 

Gives an overview of the mean exam scores on the subject of electricity for both conditions and for each class. The exam scores (which could range from 1-10) for this subject were made up of the scores on a number of tests from the students’ regular curriculum that they had taken before the experiment began.

 The way these exam scores were determined in both classes was not similar (both followed a different curriculum) but data for each condition were normally distributed for both Class 1 (Experimental: Shapiro-Wilk W = 0.959, df = 11, p = .754 ; Control: Shapiro-Wilk W= 0.969, df = 13, p = .754 ) and Class 2 (Experimental: Shapiro-Wilk W = 0.915, df = 10, p = .316 ; 

Control: Shapiro-Wilk W= .928, df = 15, p = .252). No difference between the experimental and control condition was found (Class 1: F(1,22) = .623, p = .438) Class 2: F(1,23) = .439, p = .514). From this we may conclude that the experimental and control condition entered the experiment with comparable prior knowledge.

7- Because students came from two different 

Classes with different backgrounds, we performed an analysis of the results on the knowledge tests for the two classes separately. Class 1. Table 4 shows the results of the knowledge tests for the two conditions in Class 1. 

Statistical tests for detection of outliers showed that one student in the experimental condition appeared to be an outlier for the conceptual items (with a score greater than 2 SD below the mean score). This student was removed from further analyses. 

conclusion

In both conditions results on the conceptual test remained normally distributed after the removal of the outlier (Experimental condition: Shapiro-Wilk W = 0.939, df = 10, p = .542; Control condition: Shapiro-Wilk W = .899, df = 13, p = .129). The test results for the procedural knowledge test were affected by the removal of the outlier 

(Experimental condition: Shapiro-Wilk W = .790, df = 10, p = .011; Control condition: Shapiro-Wilk W = .950, df = 13, p = .597). Therefore, the non-parametric Kruskal-Wallis test was used to examine the differences between conditions within Class 1. A significant difference between the two conditions was found on conceptual items (H(1) = 5.044, p = .025) but not on procedural items (H(1) = -.117, p = .732).