1	Constraint-Based Tutors: a Success Story Tanja Mitrovic Intelligent Computer Tutoring Group Computer Science Department University of Canterbury Christchurch, New Zealand
2	Student modeling
3	Learning from performance errors Declarative/procedural knowledge Learning phases: Error detection Error correction CBM: domain and student modeling
4	Constraint-based Modeling Ohlsson, 1994 The space of incorrect knowledge is vast Therefore: abstractions are needed Represent only basic domain principles Group the states into equivalence classes according to their pedagogical importance
5	Constraint-Based Modeling Domain knowledge represented by a set of constraints A constraint is a pattern of form <Cr, Cs> If a solution matches the Cr then it must also match the Cs, else something is wrong “Innocent until proven guilty” approach
6	Example constraints If you are driving in New Zealand, you better be on the left side of the road. If the current problem is a/b + c/d, and the student’s solution is (a+c)/n, then it had better be the case that n=b=d.
7	Advantages of CBM Very efficient computationally No need for an expert module No need for a bug library Insensitive to the radical strategy variability phenomenon Neutral with respect to pedagogy
8	SQL-Tutor Research began in 1995 Solaris version Developed in 1997 Used in COSC313 in 1998 MS Windows version (1998) downloaded by 1186 people (May 1999 – 2001) Web version (1999)
9	Problems with learning SQL Misconceptions about the relational data model Misconceptions about the SQL concepts The necessity to learn about DBMSs DBMS messages are difficult to understand DBMSs unable to deal with semantic errors
10	Architecture of SQL-Tutor
11
12	Students did learn from it!
13	Comparison of competence (1998)
14	History of ICTG SQL-Tutor Solaris version (1997) MS Windows version (1998) Web version (1999) CAPIT (2000) KERMIT (2000) Web version 2003 WETAS (2002) LBITS (2002) NORMIT (2002) ERM-Tutor (2003) COLECT-UML (2005)
15	The goals of ICTG Enhancing CBM Testing the applicability and generality Development methodology Authoring system
16	Developing Constraint-based Tutors: Theoretical Underpinnings How to represent the domain? How to model the student? What pedagogy? When should the ITS take an initiative? What to instruction to deliver?
17	Enhancing CBM Long-term student model Overlay model Probabilistic model Problem selection Problem generation Tailoring hints Animated pedagogical agents Open student models Supporting and modeling metacognitive skills
18	Evaluation Highest importance Always in authentic situations Pre-post test performance Log analysis Subjective data Difficult to plan Hard to control Paper in Session 9b
19	Evaluations of SQL-Tutor CBM and students’ learning (1998 - ) Effectiveness of feedback (May 1999) Probabilistic student model (October 1999) Animated pedagogical agent (October 1999) Self-assessment (2000) Open student models (2001) Teaching problem-selection (2002) Problem selection strategies (2003) Granularity of feedback (2004)
20	Mastery of constraints
21	Would you recommend SQL-Tutor to other students?
22
23	Generality of the approach Design tasks SQL Database design (EER model) Software design (UML) Declarative tasks (CAPIT) Procedural tasks Data normalization (NORMIT) ER-to-relational mapping (ERM-Tutor)
24	CAPIT: Capitalisation and Punctuation Intelligent Tutor English punctuation and capitalisation for school children (9-11 years) Basic usages of capitals, commas, full-stops, quotation marks Completion exercise: student must punctuate and capitalise an unpunctuated, uncapitalised piece of text
25
26	Evaluation of CAPIT One 45 min session over 4 weeks 3 classes of 9-10 year olds Group A: no CAPIT Group B: no student model Group C: probabilistic student model
27	Results
28	KERMIT ER is a widely used conceptual data model Requires extensive practice to excel in it Developed as a problem solving environment Student modelling using CBM Implemented in Microsoft Visual Basic
29
30	KERMIT’s Knowledge Base 88 constraints Syntactic constraints All entity names must be in upper case The weak entity participating in an identifying relationship should have a total participation Semantic constraints The student’s solution should consist of all the entities present in the ideal solution
31	KERMIT - evaluation Evaluation performed 24-27 July 2001 COSC226 (Introduction to databases) 57 students in two groups Control group: no feedback (only full solution) Experimental group had all levels of feedback Pre/post test + questionnaire
32	Pre/post Test Results
33	Problem-solving support via the interface Reducing the working memory load Visualizes the goal structure Providing domain-specific information Structures students’ thinking Enforcing good practices in the chosen instructional domain Provide a learning environment close to the real-world environment
34	Problem-solving support via the feedback Based on intelligent analysis of students’ solutions Various levels of detail Correct? Error flag Hint Detailed hint All errors Full solution Wording of feedback Common-sense vs theory-based feedback
35	Wording of feedback Use the underlying learning theory! An effective feedback message should tell the student: Where the error is What constitutes the error Reiterate the corresponding domain concept Theory-based feedback more effective than intuitive feedback Paper in session 5a
36	Supporting problem solving via self-explanation Inspired by Conati & VanLehn, Koedinger Supported in KERMIT (database design) and NORMIT (data normalization) Student required to explain during problem solving Results: SE increases declarative knowledge procedural knowledge motivation
37	Self-explanation in NORMIT Explanation required for every action performed for the first time, or when there is an error Explanations selected from given options If the explanation is wrong, the student is asked to define the underlying domain concept
38
39
40
41	Defining domain concepts
42	WETAS Web-Enabled Tutor Authoring Shell ITS web server All tutoring functions taken care of: Student Modelling Problem Selection/Generation Feedback Three types of interface support: Text-based (WETAS controls interface) HTML (Total user control) Applet (mixed)
43	Tutors built in WETAS SQL-Tutor (reimplemented) LBITS Radiology Tutor EER-Tutor (KERMIT) COLLECT-UML
44	New project: ASPIRE WETAS does not support authoring of domain models eCDF grant Authoring-System for developing Intelligent Learning Environments Web-enabled (both authoring and delivery)
45	ASPIRE The author describes the domain in terms of an ontology Syntax constraints are induced automatically from the ontology Semantic constraints induced with the author’s help Interactive demo on Thursday Paper in Session 6b
46	Commercializing efforts DatabasePlace Web portal (Addison-Wesley) www.databaseplace.com Access to the portal sold with AW books February 2003 (SQL-Tutor & NORMIT) February 2004 (ER-Tutor)
47	Number of registered DatabasePlace users
48	Comparing local to distant students
49	Comparing local to distant students
50	Comparing local to distant students
51	Current projects ML for constraint induction (Pramudi Suraweera) Adding support for collaboration UML-COLLECT (Nilufar Baghaei) Affective modeling (Konstantin Zakharov, Amali Weerasinghe) A constraint-based Java tutor (Jay Holland) Adding question asking facility to constraint-based tutors (Nancy Milik)
52	Come visit us! Some of our visitors Beverly Woolf (1999) Ken Koedinger (2000) Vladan Devedzic (2002) Stellan Ohlsson (2004)
53
54
55
56
57