Maria Klawe
University of British Columbia
Summer 2000

Introduction
A recent research study conducted by E-GEMS investigated the effects of varying group size and input device on the level of engagement, understanding and learning of students playing an original E-GEMS prototype. This research study evaluated SMART Boards as possible input devices for teams playing a single user in a computer game. SMART Boards are interactive whiteboards that let the user control his/her computer from the Board’s touch-sensitive screen surface.

Discussion - Experimental Design - Software Used
The educational computer game involved in the study is Avalanche, designed for children aged 10 and up. Avalanche is a co-operative game played over a network by four players. The game is set in a town damaged by a minor avalanche from a nearby mountain. The players, acting as the Disaster Response Team, must solve math problems relevant to avalanche-prevention, e.g., finding areas of critical zones, computing the coefficients and calculating the snow volume, as well as climbing mountains to gather data. The members must communicate with each other to plan a strategy and assign peers to particular tasks in order to prevent future avalanches from destroying the town.

The specific activity used in this study is Prime Climb. The objective for the players is to reach the peak of the mountain and, in doing so, gather information needed to complete other puzzles. The player climbs a mountain with a partner connected to him/her by a rope. Each mountain is represented by hexagons (hexes) containing numbers. Players climb a mountain by stepping on numbers without common factors. If the climbers step on a pair of hexes with a common factor, the last player to move falls. The climbers take turns moving up the mountain until one of them reaches the top. Once this is achieved, information on the mountain is entered automatically into the pair’s notebooks and reward money (representing a full refund for climbing equipment fees charged upon entering the activity) is earned.

Prime Climb is preceded by a series of tutorial levels called Practice Cliffs. Players earn certificates and receive discounts on climbing equipment fees charged upon entering Prime Climb if they successfully complete a number of Practice Cliff activities. Players entering a Practice Cliff are led through the activity by a guide called Cool Guy. Cool Guy teaches the player how to climb (with a computer partner), regain position after a fall and use picks to reduce a hex’s value by 1. As the levels progress, the amount of instruction given by Cool Guy decreases. The techniques used in the Practice Cliff are also used in Prime Climb, which has a higher level of difficulty in terms of the numbers contained in the hexes and the size of each mountain.

Specific Research Question
The hypotheses tested in this study are the following:

1. that using a SMART Board in a group play session will positively affect the degree of student collaboration, and awareness of and engagement with mathematical content.
2. that students will find collaborative play of Avalanche by two teams where each team uses a SMART Board as effective in terms of motivation and learning as playing in pairs where each student plays on their own computer.

This study aims to determine whether the use of a SMART Board can enhance children’s interaction with the game by providing them with a new way to share control over the game in situations where small teams act the role of a single player.

Experimental Setting
In this study, a class composed of Grade 6 and Grade 7 students is divided into three teams of 8 or 9 members. Each student in the class is given two opportunities to play Avalanche in each of the Team Play and Pair Play configurations. In Team Play, two teams play a common game of Avalanche projected onto a SMART Board and co-operatively complete several Prime Climb and Practice Cliff levels. Each team plays as one character and uses the SMART Board to interact with the game. In Pair Play, two students play a common game of Avalanche, with each student playing one character at his/her own computer. Prior to playing Avalanche, the students complete a questionnaire and answer a pre-test that measures their knowledge of prime numbers and common factors. After all the play sessions, the students answer a post-test and an attitude questionnaire.

Experimental Procedures
For this study, a mixed Grade 6 and Grade 7 class is divided into three groups of about 8 members. The students are also asked to pair up for the Pair Play phase of the study. For Team Play, each session involves two groups playing Prime Climb and completing the appropriate number of Practice Cliffs for approximately one hour. The setup for the Team Play phase is as follows:

Figure 3: Setup for the SMART Board phase of the study

For Pair Play, each session involves two students playing Prime Climb and completing the appropriate number of Practice Cliffs for about one hour. The set-up for the Pair Play phase of the research study is as follows:

Figure 4: Setup for the pair play phase of the study

Prior to the play sessions, the students complete a questionnaire on collaborative techniques and their familiarity with computers and computer games. A mathematical pre-test is also administered before the study begins. For Team Play, a group works collaboratively as a single character in Avalanche. Two groups play a common Prime Climb game within Avalanche, using two SMART Boards placed back to back as the input devices for the two "characters". A video camera behind the students records group dynamics and the group’s interaction with the game using a SMART Board as well as the interplay between the two groups. For Pair Play sessions, two students play a common Prime Climb game sitting side by side at two computers. At the conclusion of the study, the students complete questionnaires to assess their attitudes towards the two playing configurations. The students must also complete a mathematical post-test.

Details of the Study
The research study was conducted at University Hill Elementary School, a public school in Vancouver, British Columbia. The Pair Play phase ran from April 10 - May 10 and Team Play sessions were held on April 10, 12 and 14. Play sessions run from 30 minutes to one hour. The participants were Grade 6 and 7 students with varying levels of experience with Avalanche. The students were divided into groups by their teacher.

Scoring System
The factors considered in this study are performance in the game, possible gain in the level of learning, and level of engagement with the game.

Performance scores are based on speed of completion, specifically the number of Prime Climb activities and the number of Practice Cliff levels successfully completed within the hour allotted.

A possible gain in the level of learning is measured based on mathematical pre-test and post-test scores. The pre-tests and the post-tests have content similar to that in the Prime Climb and Practice Cliffs activity.

The level of engagement is determined by ratings given on the post-play questionnaire, as well as an analysis of the video tapes of the team play sessions.

Findings and Observations
Pre-Play Questionnaire and Pre-Test Results
Prior to the first play session, the students answered a questionnaire on collaboration, game-playing techniques, familiarity with computers and computer games.

Figure 5: Distribution of student responses for the pre-play questionnaire

The students had used a SMART Board before in conjunction with a class Super Tangrams teaching session. Prior to that they had no previous experience in playing computer games in groups larger than pairs.

A mathematical pre-test was given before the first play session. It contained questions on identifying prime numbers and identifying common factors in a pair of integers. For purposes of the study, 1 and n, where n is any integer, is not considered as a pair of numbers that share a common factor.

Figure 6: Distribution of students according to performance in the mathematical pre-test

Among the participants, eight Grade 7 students had previous experience with an earlier version of Avalanche. These students were grouped together in Group C and were the last group in the class to play Avalanche as a team using the SMART Boards. The other students in the class had no previous experience with Avalanche.

The mathematical pre-test results indicate that the students have a sound knowledge of prime numbers and common factors, scoring from 5 to 20 out of a possible 20 points on the pre-test.

Modifications to the Setup
For the first phase of the study, the whole class was given an overview on Avalanche. The class split into three groups, with one group playing on a pair of computers . The other teams played Avalanche on the SMART Boards. The final setup of the SMART Boards is shown below:

Figure 7: Modified setup of the classroom for the SMART Board phase of the study

An E-GEMS observer was available to act as a resource person for ambiguous aspects of the game. The observers also acted as technical support.

The first play session was the first time the students had seen Avalanche. In order to orient the students with the game, they were permitted to watch the intro sequence where their character was briefed on the game objectives, to go through the press conference where the goals are reiterated, and to spend time on the map. This left the groups with 40-45 minutes to play Prime Climb and complete Practice Cliffs.

During the second play session, the projector wasn’t receiving a signal from the computer. The complete system had to be shut down, reconnected and rebooted. Resolving this problem left the two groups with around 40 minutes to play Avalanche.

The third play session was delayed due to a school activity. The two groups had 40 minutes to play Prime Climb and complete as many Practice Cliff levels as they could within that time.

For all three groups, the average playing time was around 40 minutes. The revised time is sufficient for groups to complete several Practice Cliffs and a few Prime Climb activities. This modification does not drastically affect the results since all groups had roughly the same playing time and comparing the results should show that their levels of achievement are relatively similar.

The pair play sessions were not recorded on video tape. For a few of the pair play sessions, a third student observed the pair and sometimes "coached" them through activities. Only a small percentage of the pair play sessions were supervised and these often were scheduled into the class’s regular computer period from 12:45 – 1:15 p.m. from Monday to Friday, so play sessions ran for 30 minutes instead of a full hour.

Observations for the SMART Board Phase
At the start of the first team session, the E-GEMS observer demonstrated on the SMART Board how toolbar features were used and how a character navigates around the map. The first pair of groups were permitted to view the introductory sequence, and run through the press conference. The other two pairs of groups, due to shortened play periods, had both the introductory sequence and press conference omitted and were brought directly to the first Prime Climb puzzle. If any questions on the game arose, the students were encouraged to ask their team mates, ask the other team, or consult the game’s Help panel.

Before the play session started, E-GEMS observers reminded the group that they would be playing one character, so decisions made by the character should be discussed within the group. One student was assigned the role of "keyboard person". The keyboard feature of the SMART Board was not used because in trial sessions, it was very difficult to enter text in an active window and keep the SMART Board keyboard window active at the same time.

The groups at the SMART Board were allowed to play with no instruction from the E-GEMS observer. During the play sessions, at most four students in a group interacted with the game through the SMART Board, while about three students stood watching the game on the computer monitor and sometimes controlling the game with the mouse, if they didn’t agree with the students at the Board. The other two students in the group watched the students at the SMART Board and offered suggestions occasionally. Rotation of players was encouraged, but at most five out of the nine students in the group would get to interact with the game at various points in the play session.

The students had difficulty in playing Avalanche on the SMART Board since they had to avoid getting in the way of the projector beam. This was awkward and the students tripped over the tip feet or knocked the Board out of alignment. The activities require exact movement so once the Board was moved, the game had to be interrupted to reset alignment. Students watched the monitor, occasionally controlling the game using the mouse, because the area in front of the SMART Board was crowded or the projected image was obstructed by other students.

The back-to-back setup of the Boards affected interaction between the teams. The "keyboard person" had to type in messages directing the other team. Communication between the teams was limited and members often walked over to the other side to see what their partner team was doing. Students became confused on whose turn it was to move, what number their climbing partner was currently on and where to move because of the lack of communication between the teams.

Allowing many players to control the game using the SMART Board was effective but also caused problems. If one player was controlling the team’s character, another player could take control simply by tapping on the SMART Board’s surface while the first player was doing his/her task. The SMART Board gave game control to the most recent event it recognized, such as a someone touching its surface. At times, three players would be tapping the surface and a fourth would try to control the game using the mouse, causing a system overload and a crash or a disappearing/flickering cursor.

Most of the interaction and discussion occurred among the students working at the SMART Board. The students standing by the side were more passive and quiet. The students at the computer often played with the keyboard and mouse, overriding the actions of the students at the SMART Board.

Observations for the Computer Phase
Students playing Avalanche in the pair play configuration were permitted to view the introductory sequence and to run through the press conference. It was stressed that the objective of the pair play sessions was to play Prime Climb and complete Practice Cliff levels and Prime Climb activities.

The pair play sessions were supervised by the students' teacher, who had little experience with Avalanche, so the students were implicitly encouraged to either figure puzzles out by themselves, ask their partners or consult the Help panels.

During the pair play session, it was apparent that the math in Prime Climb confused the students. The students, including those with strong math skills, had not discovered that numbers could not have common factors in order to be successful in climbing. Even after completing several Practice Cliffs, the students kept falling in Prime Climb.

The students playing in this configuration found interacting with each other quite easy. They used the chat box or coached each other verbally through the activity. A number of students, while working on the co-operative Prime Climb activity, would look over their partner's shoulder to show them which hexes to step on.

Post-Play Questionnaire and Post-Test Results
After the last play session, the students completed a questionnaire comparing level of engagement with the game for each input device, mathematical learning, and input device preference for future play sessions.

Figure 8: Distribution of student responses for the post-play questionnaire

The pair play configuration optimized motivation to continue playing and collaboration between players. Learning, collaboration and interest in using SMART Boards in a future session of Avalanche were rated low by the students. They preferred smaller group sizes because each player has more opportunities to interact with the game and it was easier to communicate and co-ordinate their actions with less people involved.

Figure 9: Distribution of students according to performance in the mathematical post-test

For the post-test, student scores ranged from 5 to 20 out of a possible 20 points. However, compared to the distribution of scores for the pre-test, the distribution of scores for the post-test was spread out much more evenly. For the post-test, 17 out of 23 students achieved scores of 15 to 20.

Of the 23 students, 12 students experienced a decrease in their post-test scores (compared to pre-test scores), 6 students experienced no change, and 5 students had an improvement.

Conclusion
Past studies on co-operative learning have shown that collaboration among students playing educational computer games resulted in a higher degree of involvement with and improved performance in the game. Along with these benefits, co-operative learning brings with it an interesting question: how would a group of players share the role of a single user in a computer game?

A possible solution to this question is the SMART Board. The SMART Board is capable of accepting multiple-user inputs concurrently and it also acts as a large surface that a computer game can be projected on for group play sessions. In evaluating the effectiveness of the SMART Board as a new type of multiple-user input device, one must consider factors such as its effect on group interaction, engagement with the game, learning and student interest in using it as an input device for future play sessions.

Another factor that is considered in the search for the configuration that best supports group play is the size of the group acting as a single player. This variable affects group interaction and engagement with the game. Group size also affected the way the SMART Board was used. Large group size and the absence of a game-plan caused system overloads, and confusion and frustration among players. Some members did not get a chance to use the SMART Board.

The SMART Board is a promising alternative input device. Any shortcomings that may have emerged from the study are more attributable to human dynamics than to limitations in SMART Board design.

Recommendations
For follow-up studies, an investigation on the effectiveness of the SMART Board as an input device should study the SMART Board in isolation. In an isolated SMART Board study, a change in the pre- and post-test scores would be attributable to this setup alone.

Interaction in the SMART Board would have been greatly improved if group sizes were decreased to 4 players. It was observed during the study that about 4 players would be interacting with the game using the SMART Board and collaborating in order to solve the puzzles in Avalanche, while other players were watching the group at the SMART Board or crowded around the computer. A four-player group is small enough to avoid crowding at the Board and keep all members engaged with the game for the duration of the session.

SMART Board interaction might also have been improved if a game-playing protocol had been strictly imposed. The students playing at the SMART Board would take turns at the Board, avoiding the situation where players would be tapping the Board at once and causing system overloads. Players can take turns in pairs, individually or as a whole group. The players, however, must be forewarned that too many users tapping the Board at the same time may cause a malfunction.

Finally, for group play of a computer game like Avalanche, a rear-projection SMART Board would have been ideal. This would remove the inconvenience of avoiding the projector beam. Placing the SMART Boards side by side would have positively affected interaction between the groups and could possibly have improved the performance of the groups in the game.

Biography
Maria Klawe is currently the Dean of Science at the University of British Columbia, having served there as a vice president from 1995 to 1998, and head of the Department of Computer Science from 1988 to 1995. She also holds the NSERC-IBM Chair for Women in Science and Engineering, one of five regional chairs across Canada. Her chair is responsible for British Columbia and the Yukon, and emphasizes research and programs aimed at increasing the participation of women in information technology careers. Prior to joining UBC, she spent eight years with IBM Research in California, and two years at the University of Toronto. She received her PhD (1977) and BSc (1973) in mathematics from the University of Alberta.

Maria’s current research focuses on the design and use of computer games and activities for mathematics education for grades four to eight. She leads the E-GEMS project, a collaborative project involving computer scientists, mathematics educators, teachers, children and professional game developers. E-GEMS has developed several innovative and successful prototype games, and has done seminal work in identifying important factors in the design of effective educational software.

Maria has also served on many boards and advisory councils, including the Board of Trustees of the American Mathematical Society (92-97), the Computing Research (90-96), the BC Premier’s Advisory Council on Science and Technology (93-present), and the ACM Council (98-present). Maria was elected as a Fellow of the Association of Computing Machinery in 1995, and received the Vancouver YWCA Women of Distinction Award in Science and Technology in 1997.

Background
In the fall of 1992, Maria Klawe, then head of the computer science department at the University of British Columbia (UBC), assembled an interdisciplinary team of researchers, teachers, and computer game designers to form the E-GEMS project. The primary goal in creating E-GEMS was to explore the possibilities of using specially designed computer and video games to increase learning and appreciation of mathematics by children aged 10 to 14. This age range was chosen because research has indicated that this is when most children, especially girls, lose interest in math and science. The team was interested in electronic games because of their appeal to children and because they offered excellent opportunities for visualization and exploration of complex concepts.

On the other hand, they had serious concerns about using electronic games. Most girls, especially aged 10 and older, seemed to be less interested in playing electronic games than boys, and less interested in using computers in general. Thus, the question of whether it was possible to create electronic games that would be attractive to most girls in this age range was an issue for E-GEMS researchers from the start. Over the last five years the topic of girls and computer games has received a great deal of attention in the popular media, but the number of successful commercial games that strongly appeal to girls is still very small.

In the past seven years E-GEMS has conducted a wide range of research studies and developed innovative prototype games that are available to educators for use in the classroom. E-GEMS results demonstrate that games can be very effective in increasing both motivation and achievement in mathematics learning. They also pinpoint the critical importance of detailed elements of game design, the role of the teacher, and the integration of computer games with other forms of mathematics education. E-GEMS work has identified important differences, as well as similarities, in girls’ and boys’ interactions with games and computers.  Visit E-GEMS.

Back to Introduction
Back to Project Library