EPUBs are an experimental feature, and may not work in all readers.

Natural sciences are an essential component of the liberal-arts educational experience. However, many students find the problem-solving skills that are an integral part of any introductory physics course difficult to develop (Halloun & Hestenes). In particular, working on homework assignments without the benefit of expert feedback is often a frustrating experience. In order to alleviate this situation, one possible remedy is to construct an intelligent tutoring system that is capable of providing immediate feedback and relevant hints to students (Kulik & Kulik; Larkin). One such system is Andes, a coached learning environment for classical physics that has been in development since 1996 by researchers at the Learning Research and Development Center of the University of Pittsburgh and at the United States Naval Academy. This intelligent tutor allows students to solve physics problems in an environment that provides visualization, immediate feedback, and procedural and conceptual help. It runs under both the Windows 98 and NT operating systems.

Andes consists of an authoring module, developed by the Naval Academy, and a student module, developed by the Learning Research and Development Center. This software is still under development. It currently tutors students in the areas of static forces, translational and rotational kinematics, translational and rotational dynamics, energy, and linear and angular momentum.

Authoring Environment

The knowledge base behind Andes solves the physics problems off line by generating all the equations necessary to obtain a solution for each problem. This base consists of approximately 600 rules of two types: goal rules and physics-knowledge rules. The goal rules are used to guide the system in the steps necessary to obtain a solution path, and the physics knowledge rules are used to provide the underlying domain concepts.

In addition to generating the equations, the knowledge base also produces a graph indicating all reasonable solution paths. These paths, together with the equations, are then used by the modules in the student environment of Andes to assess student actions and provide appropriate feedback and hints. Specifically, each of the rules has a hint associated with it that can be used to give direction to a student who chooses to use the help system. The solution graph is used to identify the point that the student has reached within the solution, and indicate what the next step should be.

Pedagogical Considerations

Three of the Naval Academy researchers are physics professors with more than 90 years of combined experience. All students at the Naval Academy, regardless of major, are required to take a two-semester, calculus-based general-physics course. Further, there are no teaching assistants at the Naval Academy, so the Naval Academy professors have accumulated considerable experience in both teaching and tutoring a wide variety of students.

Several pedagogical issues were considered in the design of Andes. Beginning students often confuse related concepts such as acceleration and velocity, mass and weight, or force and momentum. Some students even hold false beliefs, e.g., about what forces exist. One goal of Andes was to target these misconceptions. Good physics students analyze problems qualitatively and plan their solutions, whereas poor physics students plunge immediately into the algebra. Therefore the interface was designed to encourage students to follow successful problem-solving strategies. In selecting problems to include in Andes, it was considered important not only to choose some basic problems for each topic, but also to attempt to target problem types that students often find difficult.

In all of the problems, Andes encourages the student to draw diagrams when appropriate, and to draw and label vectors, etc. A design decision was made very early in the development of Andes that the knowledge base would solve problems using vector components, even when the problems could be solved one dimensionally. The developers felt that by encouraging students to use components, they would overcome many of the difficulties that arise when students transition to two-dimensional problems. This design also provided consistency in the system.

Fig. 1: A typical one-dimensional problem solved via components.Fig. 1: A typical one-dimensional problem solved via components.

This is not to say that students are required to use components if they are not necessary to obtain the solution. For example, if a student is solving the one-dimensional, static-force problem as shown in Fig. 1, she can define a variable, Ft, representing the magnitude of the tension vector and another variable, Fw, representing the magnitude of the weight vector. Then once the student has defined a set of axes, Andes automatically provides the variables Ft_x, Ft_y, Fw_x, and Fw_y to represent the vector components of the two forces. Since in this problem there is no x-component for either force, the knowledge base generates the equations Ft_y = Ft (the y-component of the tension vector equals the magnitude of the vector) and Fw_y = -Fw (the y-component of the weight vector equals the negative of the magnitude of the vector). The student can then generate either the equation Ft - Fw = 0 or Ft_y + Fw_y = 0. Either will be considered correct. A more detailed discussion of the pedagogical considerations in constructing the Andes knowledge base can be found in Schulze, et al. (00a).

One of the primary goals in developing the knowledge base was to provide a tutoring environment that allowed the same kind of student/tutor interaction that might be experienced in a one-on-one session with a professor. One important characteristic of such an interaction is flexibility in the allowed solution path. This concept of providing flexibility in solution paths exhibits itself in a number of ways. One of the ways is in the choice of axes for a free-body diagram (Schulze, et al., 98). For example, if the goal of a problem is to find the magnitude of a force whose direction is not along the standard xy-axes or a ninety degree rotation of it, the student can choose either the standard axis system or choose the positive x-axis to be in the direction of the force. When a choice of axes reasonably exists, the knowledge base will solve the problem using all reasonable axes choices. Flexibility further occurs when it is possible to solve a problem in two different ways, and in this case, the knowledge base will generate both solution paths and the corresponding equations for each path. A further example of flexibility occurs when the student has a choice as to which body to select when solving the problem. This frequently arises in problems that involve a compound body. Again, the knowledge base will generate a solution path and corresponding equations for each of the possible bodies.

"No step in a physics problem solution is so simple that it won't be difficult for a significant number of students"

This flexibility allows the student to determine her solution path without being forced to solve the problem in only one way. It also allows the system to provide the student better help. For example, if the student is solving the static-force problem in Fig. 2, she has a choice of creating a free-body diagram with the standard xy-axes, or choosing the axes in the direction of the tension force.

Fig. 2: A problem with two possible axes choices.Fig. 2: A problem with two possible axes choices.

If the student draws the free-body diagram using the standard axes, none of the equations generated by Andes that relate to the axes in the direction of the tension force should ever occur in the student's solution path. If the student eventually enters an equation that would result from the axes in the direction of the tension force, it will be marked wrong. If the student then asks "What's wrong?" with the equation, Andes responds that it could not interpret this equation and suggests the student check the form of the entry and try again. This occurs because the equation does not lie on the solution path down which the student has started.

Student Environment

The student environment contains five major modules: the action interpreter, the assessor, the interface, the help system, and the student model. A more detailed discussion of the student environment can be found in Schulze, et al. (00b).

Interface

The interface, referred to as the Andes Workbench, provides an interactive environment in which the student works the physics problem. It consists of several panes and multiple tools, as indicated in Fig. 3.

Fig 3: The Andes interface depicting a rotational dynamics problem.Fig 3: The Andes interface depicting a rotational dynamics problem.

The icons along the left-hand margin of the window can be used to construct free-body diagrams or motion diagrams in the left-most pane, or used to define any vector quantity, as required. The icons above the window allow the definition of variables required in a solution, the use of Greek letters, the use of an equation solver, and the request for help with a solution, among other things. The top right pane contains the variables that have been defined, and the student enters equations in the bottom right pane. A video demonstration of Andes opens in a new window. Students use the right mouse button to control playing and pausing the video, and to close the video window when the demonstration is over. The video lasts approximately three minutes.

Action Interpreter and Assessor

The action-interpreter module in the tutor provides immediate feedback to the student while she is constructing her free-body diagram and when she enters equations. If the action she takes in the left-most pane (such as drawing a vector) is correct, the vector will turn green; otherwise it will turn red. The student can enter equations only in the bottom-right pane, and the equations can contain only variable names that appear in the top-right pane. If she attempts to enter an equation containing an undefined variable, Andes will turn the equation red and inform the student that it does not recognize the undefined variable. With the current version of Andes, the student is not required to construct a free-body diagram in the left-most pane before she enters equations in the bottom-right pane. She may define her variables using the Variable Menu rather than by constructing them. However, future versions of Andes will provide the instructor with the ability to require the student to construct the free-body diagram before entering equations in the bottom-right pane.

"Giving effective hints or help is very difficult"

The assessor module maintains a long-term student model of the student's level of mastery of individual physics concepts. It interprets the student's problem-solving action in the context of the current problem and determines the type of feedback to provide. For example, if the student enters an equation in the bottom-right pane, it is compared to the set of equations produced by the knowledge base, and if a match occurs, the equation turns green. If there is no match, then it turns red. At the same time that Andes turns something red, it enables the toolbox button "What's wrong?" which the student can select to receive a hint as to what she has done incorrectly. If the student asks for help on a red object, the assessor determines where in the solution graph the correct object resides, and passes this information to the Help system. The assessor module then updates the student model.

Student Model

The student model is updated at the end of each problem (Conati, et al.; VanLehn, et al., 98). This model maintains the probability that a student understands specific physics concepts. For example, if a student has successfully solved several problems involving Newton's Second Law, then the student model will reflect that she has mastered this concept. However, if Andes determines the student does not understand a concept such as the normal force, a short lesson explaining that concept will appear on the screen. In addition, during the execution of a particular problem, the student's position in the solution path is maintained. Both of these features of the student model are used to guide the Help system.

Help System

The simplest portion of the help system concerns the Andes Workbench interface, and includes an annotated illustration of its use. The student can also choose to run a video demonstrating how to interact with the Workbench. However, when the student asks for help during a problem solution, Andes needs to know what the student is trying to do, and this is much more challenging. The help module tries to understand what plan or goals the student is pursuing as she does an activity by evaluating the student model's record of her solution path/actions. Although the current system will provide specific-enough hints that the student can solve the problem simply by continually clicking on the help buttons, future versions will remove this possibility. At that point, bottom-out hints (final hints) will refer the student to electronic reference pages that explain the relevant physics concept.

While the student is solving a problem, she can solicit two types of help. The first allows the student to ask for hints about what step to take next to construct a successful solution (Gertner, et al.). This component of the help module is tied to the student model and to the solution graph in the action interpreter and assessor modules. If the student asks for a hint, she is provided with both the hint and a "why" link that explains the hint and a "how do I do that?" link that provides a further hint. The student can usually receive three sequential hints.

Andes can also provide help when the student constructs an object in the left-most pane or enters an equation in the bottom-right pane of the Workbench. If the object or equation has been marked incorrect (turned red), she can ask "What's wrong with that?" If the student has entered an equation that comes close to matching one that the system knows about (i.e., the student equation matches Andes's equation except for a sign), then Andes will suggest something. However, if Andes cannot figure out what the student has entered (i.e. there is no close match to an Andes equation), then Andes states that it has no idea what the student is trying to do.

The student can also receive unsolicited help in the form of mini lessons if the assessor module determines from the student model that she does not understand a specific physics concept. When this occurs, a short, annotated lesson on the appropriate topic will appear as a separate window and the Workbench is disabled until the student closes the mini-lesson window.

Empirical Studies

A formal assessment of Andes was conducted at the Naval Academy in the fall semester of 1999. The assessment included 173 students using Andes as a part of their course work and a control group of 161, all of whom were taking the required general-physics course. The assessment tool was a 400-point, free-response examination that covered the first eight weeks of the course. When the raw scores were converted into percentages, the Andes group scored about 3% higher than the control group. In practical terms, this corresponds to about a third of a letter grade, which in an academic context is notable. Obviously, one goal of the project is to maximize this improvement.

A statistical analysis was performed on the data in the table below. The t stat for comparing the Andes group to the control group is 2.109.

Table 1: The results of an examination given to students using Andes and to the control group.
Andes Control
N 173 162
Mean 294.8 281.6
Standard Deviation 52.1 62.3

The p-value of 0.0174 for a one-sided t-test showed that the Andes group was better than the control group. Therefore, we can conclude that the improvement using Andes is statistically significant. For further details of the methodology and analysis of the study, see Shelby, et al.

Anecdotal Results

After five semesters of using Andes with students enrolled in the first-semester physics course at the United States Naval Academy, we have noted the following anecdotal truths:

  • Students are initially reluctant to define variables or to use consistent notation. Particularly with the early problem assignments, which tend to be relatively easy and deal with topics students often feel they understand well, having to define all the variables used in an equation can seem tedious and unproductive. Andes provides an immediate tool for enforcing such a regimen, because the tutor refuses to accept solutions that do not have the appropriate structure. After having used the Andes tutor for several segments of the course, most of the students recognize the usefulness of such strategies, and often ask for additional problems in areas where they are having difficulty.

  • No step in a physics problem solution is so simple that it won't be difficult for a significant number of students. For a variety of reasons, even quite simple problems will cause some students to ask for help at every step of the solution. For a certain number of students, uncertainty about even what the first step in a solution should be is an insurmountable difficulty, which can be overcome with a timely hint.

  • Giving effective hints or help is very difficult. Human tutors use a variety of clues to assess the level of understanding that a student possesses at a particular point in a solution and choose what type of hint to give in ways that are very complex and difficult to model. It has become obvious that, in most cases, hints that give guidance as to the general form of the solution are, in the long run, more effective than giving specific, possibly numerical hints that correct errors in individual equations.

  • Many of our students understand much less about the physics they are using in their problem solutions than many of us think. Analysis of action logs that record every key stroke that occurred during a problem-solving session indicates that even students who would appear to be achieving a satisfactory level of mastery often ask for help on routine steps in the solution path.

Future Directions

Currently, students are not required to draw free-body diagrams or coordinate axes, and are allowed to use numerical values in equations at any point. Future versions of Andes will, at the instructor's discretion, remove this flexibility and require the student to draw free-body diagrams and coordinate axes when appropriate and to draw and label all vectors. Similarly, students may be required to enter symbolic equations and delay using numerical values until the last step in the solution. In order to provide better help to the student, we may change Andes so it requires that the student identify the physics principle she is trying to use. Andes will provide appropriate feedback concerning the principle, both in terms of whether the principle is correct and whether its application has been completed.

"Scaffolding" in a tutor can be interpreted as meaning either the set of constraints placed on students in terms of how they solve a problem, or the support mechanisms, such as help, that are available to them. Effective tutors should reduce all their scaffolding (Anderson, et al.) and this is not a trivial task. Since Andes places little constraint on the students in terms of how they solve a problem, some of its scaffolding can be considered as already reduced. That has caused student-modeling difficulties that adversely impact Andes's tutoring. Future versions will provide more substantive scaffolding such as requiring that students construct free-body diagrams before entering equations.

Both human and machine-based tutors should integrate the knowledge they currently teach about how to solve problems with other important knowledge in the task domain. That is, they should encourage deeper learning. The authors believe that natural-language (English) processing is necessary for encouraging such learning. A new project, Atlas, is being developed at Pittsburgh's Learning Research and Development Center. It will contain natural-language-based enhancements for model-tracing tutors that are modeled after human tutorial dialog, and are intended to encourage deeper learning (VanLehn, et al., 00.) We hope to incorporate Atlas into Andes.


Demonstration

Video Demonstration of Andes. The video, which works best on a PC, opens in a new window. Use the right mouse button to control playing and pausing the video. Close the video window when the demonstration is over. The video lasts approximately three minutes.



Kay Schulze is a Professor of Computer Science at the United States Naval Academy. She received a B.S. from the University of Charleston (Mathematics, 1968), a MS from West Virginia University (Mathematics, 1970), and a MS and Ph.D. from Boston University (Computer Science, 1984, 1989). She was an instructor at Boston University until 1985, when she became an Assistant Professor of Computer Science at the University of Massachusetts, Dartmouth. She joined the Naval Academy faculty in 1989. She is active in the Computer Science Accreditation Commission. Her research interests include intelligent tutoring systems, computer science education, and computer ethics. You may contact her at schulze@usna.edu.

Robert Shelby is a Professor of Physics at the United States Naval Academy. He received a B.S. and M.A. from the University of Texas, Austin (Physics, 1959, 1961) and a Ph.D. from Catholic University (Physics, 1969). He has thirty-nine years of teaching experience at the post-secondary level and has been on the faculty at the Naval Academy since 1965. A long-time member of the American Physical Society and American Association of Physics Teachers, he has been active in physics-education research and development for the last twenty years and has concentrated on the use of classroom communication systems and computer-based tutoring systems for the last seven years. You may contact him at shelby@usna.edu.

Donald Treacy is a Professor of Physics at the United States Naval Academy. He received a B.S. from Boston College (Physics, 1963) and a M.A. and Ph.D. from Princeton University (Physics, 1965, 1969). He has thirty-three years of teaching experience at the Academy. He is a member of Sigma Xi. His current area of research is in methods of improving learning in Introductory-level physics courses. You may contact him at treacy@usna.edu.

Mary Wintersgill is a Professor of Physics at the United States Naval Academy. She received a B.Sc. and D.Phil. from the University of Sussex (Physics, 1974, 1977) in the United Kingdom. She was a Research Associate at Oklahoma State University until 1978 when she joined the faculty at the Naval Academy. Her research interests include intelligent tutoring systems and developments in physics pedagogy, as well as the electrical properties of polymer electrolytes and fuel-cell membranes. She may be contacted by e-mail at mwinter@usna.edu.

Kurt VanLehn is a Professor of Computer Science at the University of Pittsburgh, where he is also a Senior Scientist at the Learning Research and Development Center and Director of CIRCLE: Center for Interdisciplinary Research on Constructive Learning Environments [formerly http://www.pitt.edu/~circle/]. He received a B.S. from Stanford (Mathematics, 1974) and a Ph.D. from the Massachusetts Institute of Technology (Computer Science, 1983). He was a Research Associate at Xerox Palo Alto Research Center until 1985, when he joined Carnegie-Mellon University as an Assistant Professor of Computer Science and Psychology. He moved to the University of Pittsburgh in 1990. His research interests focus on applications of artificial intelligence to education and cognitive modeling. You may contact him at vanlehn@cs.pitt.edu.

Abigail Gertner was a Research Associate at the University of Pittsburgh Learning, Research and Development Center from 1996 to 1999. During that time she was the Project Coordinator for the Andes project. She currently leads the engineering group in artificial intelligence for the Information Management and Instructional Technology Department at the Mitre Corporation. She earned her Ph.D. in computer science from the University of Pennsylvania in 1995. Her primary research area is in student modeling and plan recognition in intelligent tutoring systems. You may contact her at gertner@mitre.org.


References

Anderson, J. R., Corbett, A. T., Koedinger, K. R., and Pelletier, R. (1995). Cognitive tutors: lessons learned. The Journal of the Learning Sciences, 4(2), 167-207.

Conati, C., Gertner, A., VanLehn, K., and Druzdzel M. (1997). On-line student modeling for coached problem solving using Bayesian networks. In A. Jameson, C. Paris and C. Tasso (Eds.), User Modeling: Proceedings of the Sixth International Conference, UM97, 231-242. Vienna: Springer.

Gertner, A. , Conati, C., and VanLehn, K. (1998). Procedural help in Andes: Generating hints using a Bayesian network student model. Proceedings of the 15th National Conference on Artificial Intelligence, 106-111. Cambridge, MA: The MIT Press.

>Halloun, I. A. and Hestenes, D. (1985). Common sense concepts about motion. American Journal of Physics, 53(11), 1056-1065.

Kulik, J. A. and Kulik, C. L. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58, 79-97.

Larkin, J. (1983). The role of problem representation in physics. In D. Gentner and A. Stevens (Eds.), Mental Models. Hillsdale, NJ: Lawrence Erlbaum Associates.

Schulze, K. G., Correll, D., Shelby, R. N., Wintersgill, M. C., and Gertner, A. (1998). A CLIPS problem solver for Newtonian physics force problems. In C. Giarratano and G. Riley, Expert Systems Principles and Programming, 3rd Edition & CDROM, Expert Systems/VIRTVONF/ PAPER14/paper14.pdf, 1-7. Boston, MA. PWS Publishing Company.

Schulze, K. G., Shelby, R. N., Treacy, D. J., and Wintersgill, M. C. (2000a). Andes: A coached learning environment for classical Newtonian physics. In J. A. Chambers (Ed.) Selected Papers from the 11th International Conference on College Teaching and Learning, 151-162, FCCJ, Jacksonville, FL.

Schulze, K. G., Wintersgill, M. C., Shelby, R. N., and Treacy, D. J. (2000b). The pedagogical considerations in constructing the Andes knowledge base (submitted for publication).

Shelby, R. N., Treacy, D. J., Schulze, K. G., and Wintersgill, M. C. (2000). A statistical assessment of a field trial of Andes (in preparation).

VanLehn, K., Niu, Z., Siler, S., and Gertner, A. (1998). Student modeling from conventional test data: A Bayesian approach without priors. In B. P. Goettle, H. M. Halff, C. L. Redfield, and V. J. Shute (Eds.), Intelligent Tutoring Systems: 4th International Conference, ITS98, 434-443, Berlin: Springer Verlag.

VanLehn, K., Freedman, R., Jordan, P., Murray, C., Osan, R., Ringenberg, M., Rose, C., Schulze, K., Shelby, R., Treacy, D., Weinstein, A., and Wintersgill, M. (2000). Fading and Deepening: The next steps for Andes and other model-tracing tutors. In C. Frasson (Ed.) Proceedings of ITS 2000. Berlin: Springer-Verlag (in press).


Links From This Article

Andes

http://www.pitt.edu/~vanlehn/andes.html

Learning Research and Development Center of the University of Pittsburgh

http://www.lrdc.pitt.edu/

United States Naval Academy

http://www.usna.edu