M00:00:09

>>Jim Cooper:  I'm Jim Cooper, Professor of Graduate Education at California State University, 
Dominguez Hills.  We're on the campus of Cal State Dominguez Hills today talking with Dr. Tom 
Angelo, who's a Professor of Higher Education and Director of the University Teaching 
Development Center at Wellington University -- I'm sorry, Victoria University at Wellington in 
New Zealand.  Tom Angelo holds a doctorate from Harvard University where he first began 
working with his colleague, Dr. Pat Cross.  Their textbook, Classroom Assessment Techniques:  A 
Handbook for College Teachers, is one of the best selling and most influential books in higher 
education in the last 20 years.  As a result of this book and their workshops, their assessment work 
has been adopted in hundreds if not thousands of colleges both nationally and internationally.  
Tom's had a Fulbright Fellowship and in 1998 was named one of America's 40 young leaders of the 
academy by Change Magazine.  Tom's consulted with over 200 campuses and has keynoted over 75 
professional conferences addressing issues of teaching, learning, and assessment.  Tom, let's begin 
by talking about your work in assessment since you're best known for that work.  Some people 
think assessment is a very technical sort of thing that requires advanced knowledge of statistics and 
psychometrics.  Can you talk about, in general, what assessment is, and your particular take on 
assessment and how it could be used in the classroom?

>>Tom Angelo:  Well, I think that at the heart assessment is about asking questions and to ask why 
we need it is about like asking why do we need check ups with doctors.  We need to find out how 
we're doing and how well we're doing in education but, the key, I think, to assessment is to ask the 
right questions.  The technical aspects of it aren't rocket science.  There are people around always 
who can help with that, but for those of us who were classroom teachers or academic administrators 
the key is to know what we're trying to achieve, especially on the teaching and learning side, from 
my point of view, and then to figure out how we would know if we were achieving that.  So, if our 
goal is, for example, to have students become better critical thinkers through their university career, 
how would we know if they're better critical thinkers?  What are the indicators of that?  What's the 
evidence of that and how good is good enough? So, assessment really is about trying to figure out 
whether we're achieving the goals and the objectives that we've set for ourselves and for our 
students and helping us keep on track and navigate in that direction.  You don't have to be a 
technical gearhead, basically, to do that.  Although, sometimes, some technical expertise helps.

>>Cooper:  Just to sort of operationalize some of the informal classroom assessment techniques 
that you and Pat are best known for, could you identify a couple of those assessment techniques 
that could be used by a classroom teacher?

>>Angelo:  Sure, and with the caveat that I am not the person who came up with most of these 
good ideas and in some cases we don't know who came up with some of the ideas, but the most 
commonly used and the most popular of all the classroom assessment techniques is something 
called “the minute paper” which we believe was developed by Professor Charlie Schwartz, at 
Berkley a number of years ago, and it essentially just involves asking students two questions at the 
end of a lecture or it could be at the end of an assignment and those two questions go something 
like: What were the most important things that you learned from this class or from this reading or 
from this whatever?  And the second question is:  What questions do you have remaining?  What 
do you still need to know?  And simply by asking students those two questions, to reflect and write 
about those questions, what were the most important things you learned and what questions 
remained, teachers can get a lot of very useful feedback that can help them then structure how they 
respond to students.  So, the bottom line is this:  Rather than me, as the teacher, try to imagine that I 
can guess what students' questions are, and that I can tell from looking at their faces what they are 
learning and what they aren't, assessment propels me to ask them and to get data from them and, 
unless you are a mind reader, assessment is really the only way to find out what students are 
thinking.

>>Cooper:  So, it sounds like assessment doesn't have to be a terribly time consuming thing.  It 
could be done in three minutes, five minutes at the end of a one hour lecture or two hour lecture or 
whatever.

>>Angelo:  Well, realistically, if it isn't relatively easy and quick to do, most classroom teachers are 
not going to have time to do it.  You know, they're busy people.  They've got a lot of content that 
they want to try to cover.  So, one of the reasons that Pat Cross and I created the book that we did, 
and many other people have done these kinds of things, is to try to find practical and time effective 
ways to do assessment in the classroom.

>>Cooper:  We're doing a particular kind of assessment called the NSSE at Cal State Dominguez 
Hills.  NSSE stands for the National Survey of Student Engagement.  Today you are going to be 
talking to some of our administrators about this so-called NSSE and what value it might have for 
college campuses.  Could you talk a little bit about what the NSSE is and what practical 
implications the administration of the NSSE to our freshman and seniors -- what practical 
applications it might have.

>>Angelo:  Well, I know something about the NSSE, mainly because I've used it myself on 
campuses in the United States and the National Survey of Student Engagement was developed by 
Dr. George Kuh, at Indiana University at Bloomington and his colleagues over a number of years to 
basically get information on things that we know are highly correlated with student learning and 
success and, so, the NSSE asks students to respond to a number of questions about things like how 
many hours a week they study, how many books they read, how many papers they write, how much 
feedback they get from their professors, how often they talk to their student colleagues about things 
that are going on in the classroom, what they are learning, all things that we know that are 
correlated to learning and success in the college or university.  So, it's really the best instrument 
that I have ever seen, and I think one of the best instruments in the world, if not the best, that we 
have for gauging how engaged students are in learning.  I want to be very clear that this is 
something different than asking students how satisfied they are or how happy they are or how much 
they enjoy being on campus.  It's about asking them what are they doing to learn and how much of 
it are they doing and how well.  With that data, campuses and departments can really get some 
insights into how students are spending their time and, if you do it well, you can use that data to 
help you pinpoint areas in your program where you might be able to raise the bar and help students 
learn more effectively and create a climate in which students are more likely to learn.

>>Cooper:  Are you aware of campuses that have successfully administered the NSSE and made 
interventions or modifications in their program that you could talk about?

>>Angelo:  Well, I'm not going to name names.  There are plenty of things that you can find by 
looking at the NSSE website which has a terrific website with lots of examples on it, but what I can 
tell you is that several hundred campuses across the country have used the NSSE in the last few 
years and there are good examples of the ways that these campuses have used it to zero in on 
specific aspects.  For example, a number of campuses have found that students were less engaged 
than they thought in the academic life or the cultural life of campus, that students didn't feel 
particularly connected, that they didn't hang around after their classes were over and have done 
things to try to create an academic climate on campus and made a difference and with the NSSE, if 
you do the survey every year or every two years or even every three years, you can then get trend 
data that can show you whether different aspects that you focused on are improving, and I think 
that's a tremendous advantage rather than looking at something for one year and looking at it at one 
point in time, to actually measure the same factors over time, try to improve them and see if it's 
making any difference over time and most campuses that are using the NSSE seriously have done 
that -- just that.

>>Cooper:  There's a companion instrument called the FSSE which is for faculty members.  Can 
you talk a little bit about what the FSSE is and how it might be used in conjunction with the NSSE?

>>Angelo:  There are all these acronyms and the FSSE stands for the Faculty Survey of Student 
Engagement and what it is basically is a version of the National Survey of Student Engagement that 
allows faculty to respond to most of the same questions but in this way -- faculty are asked to 
respond by what do they expect students to do.  For example, it asks -- the NSSE asks students how 
many hours a week do you study, on average, and the faculty version of this asks faculty how many 
hours a week they expect their students to study for their courses and how many hours they actually 
think they do.  So, it's basically saying what's your expectation, what's your standard, and what are 
you predicting your students are going to say.  Once you have that data, then faculty are likely to be 
much more interested in seeing the actual results of those students and comparing those results 
against their standards.  Let's say I imagine that my students in my Introductory Statistics courses 
are studying six hours a week say for my class.  We're in class three hours a week, two hours out -- 
two hours for every hour in class would be six hours a week, and, so, I expect them to study six 
hours a week.  Say I imagine they're studying about four and imagine my students saying, in fact, 
they're studying two on average.  Well, that's going to tell me something about the reality of my 
expectations and maybe that my homework, the assignments that I'm giving them are not really 
effectively engaging them for enough time to learn.  So comparing one's standards, expectations 
and the reality can be a very powerful way to help us see if what we're doing is really working in 
the way that we imagine it is.

>>Cooper:  Later on today you're going to be speaking with some of our campus leaders, 
administrators and faculty leaders and so on about what the chairs, deans and vice-presidents can 
do to foster this engagement that we've been talking about, and I know that you've got an acronym, 
MOM, to sort of talk about the various strategies that could be used by campus leaders to foster 
engagement.  Could you talk a little bit about that? 

>>Angelo:  Well, the -- one of the things, and I think there're several things that academic leaders 
can do to help faculty engage in assessment and improvement efforts. One of those clearly is to be 
trustworthy because people don't engage in the risk taking that change requires if they don't trust 
their leaders and so I think you have to -- you basically set an example as an academic leader for 
the people who work with you and be trustworthy and be engaged.  Easier said than done, I 
suppose, but a second important thing is to, basically, for academic leaders to understand which 
things are important enough to faculty to engage in the hard work of trying to change and also to 
recognize that any change is going to take several years to achieve.  One of the challenges we have 
in American universities is that academic leaders typically move on in three or four years whether 
they're provosts or presidents, often deans and that's not long enough to achieve the kind of change 
that most of us have to do.  So, there are a number of challenges, and I have used this acronym, 
which is actually something that comes -- if you've ever seen, and you all have, police or detective 
shows in which they talk about felonies, they talk about MOM, and MOM stand for motivation, 
opportunity, and means and so they talk about, prosecutors often talk about did the alleged 
perpetrator have a motivations.  Did they have the opportunity to commit the crime and did they 
have the means to commit the crime, and I've said the same thing about change. Are people 
motivated to change?  Are faculty -- do they care enough?  Do they think that the problem is severe 
enough?  Let's say that a number of our students are not making it through first year Calculus or 
first year Algebra on a campus and the failure rate is 50 percent, the Math Department may think 
that that's fine.  That shows that they're being rigorous and the rest of us may think that that shows 
that there's something wrong with the Math curriculum.  So, we have to convince the Math 
Department, whose curriculum it is, that they need to take seriously that and provide some 
motivation for them to take that seriously.  Then they have to have the opportunity to make change 
and that sometimes means having the time and the resources to do what needs to be done and to 
have the means, the M in -- the second M in MOM means that they have to know how to do what it 
takes to change.  Most of us try to do a good job. Most of us do the best work that we can and, if we 
knew how to do a better job, we'd be doing it.  So, often asking people to change means that they 
have to have the opportunity to learn some new skills in order to do be able to do things differently. 
So that's the MOM; motivation, opportunity, and means to make change. 

>>Cooper:  Another area that you're identified with is the area of critical thinking, and I know that 
you've talked in workshops across the country, and you'll be talking today to our faculty about 
some strategies that we think are important in fostering or promoting critical thinking or deep 
learning as it's sometimes called, can you talk a little bit about the issue of critical thinking and 
what classroom strategies might be important in fostering critical thinking?

>>Angelo:  Well, critical thinking means different things to different people in different disciplines 
and in some disciplines critical thinking is the ability to persuasively argue a point and win an 
argument by marshaling your arguments and your points.  In other kinds of discipline it's -- in the 
sciences, in particular, it's the ability to deal with hypothesis and to bring facts to bear on those 
hypotheses to prove or disprove them.  Still other fields, and engineering is an example of this and 
many other applied fields, critical thinking is the ability to solve real world messy problems 
effectively and so depending on the discipline one's in, you might think critical thinking is one, at 
least, of those three things.  In any of those cases, critical thinking requires bringing evidence to 
bear on a problem and often evidence that's conflicting or incomplete and making decisions and 
judgments based on that evidence and that's of course a key part of life for any professional and for 
lots of people who are professionals and so to do that well, it seems to me, ought to be one of the 
hallmarks of an educated person and certainly university education and college education ought to 
help us to do those kind of things better.  There's been quite a bit of good research in the last 50 
years or so about critical thinking in its various forms, and I think there's some good news in that 
and one part of the good news, especially for a place like CSU Dominguez Hills, is that diversity, 
diversity of viewpoints, diversity of experiences, diversity of approaches to thinking and solving 
problems and -- can greatly aid the development of critical thinking in students.  So, by that I mean, 
when you can put students in the situation in which they can safely work together with other 
students who have different experiences than them and different skills and different backgrounds 
towards solving a common problem, they are more likely to have their assumptions challenged and 
stretch and grow and to become better critical thinkers than if they're in a completely homogeneous 
environment. If there are any of those left, there probably aren't too many.  So, diversity used well 
and used effectively can help students become better critical thinkers.  Learning a method, and it 
really turns out not to matter too much what method you learn.  You could learn the scientific 
method.  You could learn the nursing method. Different disciplines have different methods, the 
historical method.  You can learn the Socratic method that -- any systematic method within a 
discipline that you learn well tends to help develop critical thinking, and writing, extensive writing 
with feedback tends to develop critical thinking, and it's -- obviously it's clear that there are kinds 
of thinking that we can't hold in our memory, that are too complex that we can't pursue them 
entirely in memory and speaking and writing gives us a huge advantage in that and, so, helping 
students develop the ability to think through writing is one of the proven ways, again, to develop 
critical thinking.

>>Cooper:  Craig Nelson was here recently and Vince Tinto was as well and both of them talked 
about the importance of active and group learning in fostering some of these positive outcomes.  
Can you talk a little bit about that?

>>Angelo:  Well, again, I think that's quite well developed and established that most of our 
students in most situations learn better by working with peers than they would on their own or by 
only working with faculty and, in fact, if you look at the overall -- the three biggest influences on 
student learning in higher education, in general, the most powerful influence, the thing that makes 
the most difference in terms of student learning, is what students themselves do when they’re 
studying. That gives a lot of ammunition to teaching students to learn well, to study well, to study 
effectively.  We have to remember that studying in Biology is not the same thing as studying for 
English or History and studying in Physics, studying effectively for that is not the same thing as 
studying effectively for poetry and often our students don't know -- don't have a repertoire of study 
skills.  The second most powerful influence overall in student learning is what they do with other 
students. How they work with other students on their academic work.  So, again, I think we have a 
responsibility to train students to work effectively together with others to learn well and that's an 
area that I think is pretty underdeveloped.  It isn't the case that everybody knows how to work 
effectively in groups. Certainly most faculty members, probably don't know how to work 
effectively in groups and most of us haven't been trained but in many organizations that depend on 
team work and group work they spend a lot of time and effort and money training people to work 
effectively and to deal with the inevitable conflicts that arise. So students can learn better, most 
students, by working together in organized teams.  To do that, they have to know how to work 
together and to do that faculty members have to know how to create assignments and learning 
opportunities that take advantage of team work and that take advantage of cooperative learning and 
what people can bring to the table.  So, for example, if we give students assignments that 
individuals can complete more effectively on their own than in groups, in other words pretty 
simple-minded assignments, like answering questions in the back of a chapter, very often that's not 
going to be good group work.  So, writing good assignments, creating good assignments, training 
students at some point to work well together, giving them some guidance at least and then creating 
a grading scheme that manages to motivate students is the key. So, I would say, you know, again, 
we can go back to what's the motivation for students to work in groups?  Do they have 
opportunities to develop those skills and do they have the means to do what they need to do in 
groups?  Again, those things are key.

>>Cooper:  Craig Nelson, when he was here, talked about comparing two classes, one that's taught 
to a fairly explicit set of objectives with the master learning model and a clear delineation of what 
the tests would be like versus a class that's pretty undifferentiated and the students aren't clear as to 
what the course goals and expectations are, testing procedures and said, “Now imagine you're a 
student studying for those two classes; which one of those two classes would you invest more of 
your time in, one where you knew that you if you studied well outside of class and in class there 
would probably be a payoff because it's pretty clear what's going to be on the test versus the much 
more defuse situation.”  So, I think in terms of driving study behavior the issues of clarity and 
highly structured presentations and highly delineated classroom assessment techniques make a lot 
of sense.  Let me talk to you about another issue that's of concern at Dominguez Hills.  We're a 
very diverse student body.  Over 70 percent minority students.  We have real challenges with so-
called gateway classes in math and writing that have prevented many of our students from staying 
at the university.  Can you talk a little bit about that student population and those kinds of classes 
and what we can do to kind of increase engagement and ultimately student success in those classes? 

>>Angelo:  Well, I'm no expert on diversity, and I don't want to portray myself as one.  What I 
would say is that generally what I take from the research literature is that good teaching and good 
learning seem to be relatively common across cultural groups here in the United States.  There are 
cultural differences but the fundamental things that work for diverse students work for everybody 
and good practice, generally, is going to empower most of our students.  The issue here for many 
students who come to college as first generation students is that they don't necessarily know how 
the game works effectively. So, they don't have what many people call the cultural capital that a 
student might come from whose parents are educated.  I have to say I'm a first generation college 
student, was, many years ago, and my parents knew nothing of what colleges and universities were 
like and couldn't help me in that, but I had other kinds of advantages that many students don't have 
in that my high school teachers, I think, worked very hard to try to help prepare me for what was 
going to happen in college and so there are several dimensions to this and sometimes students are 
under-prepared in the fundamental skills that we expect of them in college.  So one approach to this 
would be to say:  What if instead of making time the invariant aspect of this and quality of the 
variable, that we made time the variable and quality the invariant, and I will say what that means.  
In most college courses if you can't learn the material in 12 to 15 weeks, you're in trouble.  So if 
you're under-prepared when you come to college, and it takes you longer to learn the material than 
that 12 to 15 weeks, you're out of luck.  You either have to repeat the course or just give up or you 
take a low grade that basically shows that you haven't mastered it and then when you try to move 
on you can't bring that knowledge to bear because you don't have it under control.  In a mastery 
learning situation, like Craig Nelson was talking about, what we basically say is:  However long it 
takes you to learn this material, within reason, as long as you learn it to a high level of mastery, 
then you're fine.  So, for example, I taught a statistics course in the last few years for students who 
were under-prepared in mathematics from high school and rather than teaching it three hours a 
week, I taught it four, and we had two hours of labs rather than one.  Now that meant that it took a 
lot of students' time and effort to get through this but, for those students who were under-prepared 
in math, it was really the only option that they had to get through statistics, and we raised the 
success rate dramatically for those students, the same exams, the same standards, but more time.  
So one of the things we have to accept is not everybody is going to move at the same speed and, if 
we want to help students who are less prepared, we're going to, probably, have to accept and help 
them accept that it will take more time and effort and more practice for them to get up to speed.

>>Cooper:  That idea is quite powerful.  If you do those procedures, the ones that you're 
suggesting, the ones that Yuri Triesman tried at Berkley with minority students see them move 
from about a 60 percent DF or withdrawal rate to about a 4 or 5 percent rate, initially, with minority 
students and then with larger populations.  So it's good evidence in a variety of content areas 
thinking of Eric Mazur, in physics and in other areas where the kinds of principles that you're 
talking about have shown to be effective even in very difficult content areas like chemistry, 
statistics and so on.  I want to talk to you a little bit about what resources might be available both 
for the sort of novice with regard to some of the concepts that you're talking about and maybe some 
of the more intermediate folks in terms of strategies that can foster engagement.  Certainly there's a 
Cross and Angelo Classroom Assessment book, but can you talk about some other resources 
whether it be websites or books, conferences, journals that you read or that you would recommend 
to people interested in finding out more about what we’ve been talking about?

>>Angelo:  Well, it's always a danger, Jim, to recommend things because -- especially websites 
because they come and go so quickly, but I think that we have a surfeit of information.  It isn't that 
we don't have enough information out there it's finding the time to read these things and finding the 
right ones.  The National Science Foundation of the United States has done a very good job of 
providing websites that are navigable and that can lead you to information that's very useful.  If 
you're in the sciences or related areas, what they call the stem areas, science, technology, 
engineering, mathematics.  The National Institute for Science Education has a terrific website and, 
you know, again, if you go to the web and Google these things you can find them pretty easily.  
The best book that I've read in the last few years about learning overall is a book that was put out 
by the National Academy of Sciences called How People Learn, an editted volume by a number of 
authors all of them eminent in cognition and psychology and so, again, How People Learn is a 
book that I found -- a lot of my colleagues have found useful, very readable.  So, I think that there's 
a tremendous amount of information out there.  The challenge is finding that information and 
having the time to actually think about it.  So, I think it's better to read one book probably that can 
be helpful to you and start there.  Another book many people have found helpful, and I certainly 
have over the years is Teaching Tips by Wilbert McKeachie, and his colleagues and, if you haven't 
seen Teaching Tips and you're interested in college teaching, I think, to my mind, it's the single 
most useful book that there is on college teaching.  So there are a number of good websites, more 
all the time.  There are a number of good books.  We're in a situation in 2005 where the wealth of 
information and the synthesis of the research that's available to us now is so much better than it was 
when you and I started that it's just incredible and so, I think, that's all good news. 
 
>>Cooper:  Before I ask you if you have any closing comments I want to talk about a topic that is 
known to some faculty but not to all. It's called the scholarship of teaching and learning.  Can you 
talk a little bit about what the scholarship of teaching and learning is and what it looks like?

>>Angelo:  Well, my colleagues at the Carnegie Foundation for the Advancement of Teaching and 
the Carnegie Academy for the Scholarship of Teaching and Learning can certainly do a better job 
of this than I can, but the idea is one that people have been discussing since about 1990 when 
Earnest Boyer, first began talking about the scholarship of teaching at that time and the idea is, 
basically, to elevate teaching to a scholarly activity, in other words, to use the same kind of 
thinking and the same kind of rigorous approach toward college and university teaching that we do 
in our own disciplinary research, and one way of thinking about this that Carnegie has proposed is 
to say there are at least three levels one can be at as a practicing teacher.  One level is to be an 
effective teacher and there are people who are very effective at teaching in colleges and universities 
who have no idea how they do what they do and can't explain it to anybody, but they do a good job, 
and they do a measurably good job, and we've all had those kinds of teachers, hopefully.  I was 
lucky enough to have several, and I think most people who went into academics as a career 
probably had several extraordinary teachers many of whom could not say how they did what they 
did and didn't really have any theory of teaching but somehow had learned to do it very well. 
Carnegie then says there's being a scholarly teacher.  That's a person who really understands and 
knows how teaching and learning work and has taken the time to read about how people learn and 
especially how they learn in their disciplines and some disciplines have a very deep and broad 
literature of learning and teaching.  So you have some effective teachers who are both effective and 
scholarly and there are probably some scholarly teachers who are not too effective.  They know 
what works, but they are not very good at it.  The idea would be to be effective and to understand 
and be able to explain it to other people and help younger colleagues develop and exceed one's own 
performance, and the third level Carnegie talks about is doing the scholarship of teaching and 
learning and this is actually doing what some people might call action research or classroom 
research or various sorts of inquiry into teaching and learning.  In other words, studying how 
students learn and don't learn in our own classrooms to try and figure out how to do that better and 
people have done that for many years, of course, and what the movement of scholarship of teaching 
-- for teaching and learning has done is to really elevate that and make it into something much more 
acceptable and to get to the point where on some of our most prestigious research university 
campuses faculty are actually being promoted based on research that they are doing on teaching 
and learning in biology or teaching and learning in chemistry or teaching and learning in history.  
The goal being to improve our effectiveness at causing or promoting student learning in those 
fields.  When you think about it, we ought to as teachers be as good at getting better in our 
profession over time as engineers are or as doctors are or as other professionals are, and we'll only 
do that we if we apply rigorous scholarly methods both to our teaching and to our areas of research.  
So that's what the scholarship of teaching and learning is about and there are lots of people doing 
very exciting things out there.  If you're interested in that, I would say go to the Carnegie 
Foundation's website for the CASTL Program, the Carnegie Academy for the Scholarship Teaching 
and Learning and you will find a wealth of information. 

>>Cooper:  One of the things I like about the scholarship of teaching and learning is, again, you 
don't -- it doesn't have to be a terribly technical, methodological intervention.  It might be the 
impact of a minute paper on retention in the class or a score on an exam or critical thinking or 
liking for the subject.  So, one of the things we're pushing on our campus is for our senior 
administration in the rank and tenure process to value the scholarship of teaching in ways similar to 
that they value basic research.  Okay.  Well, we're just about at the end of our time together, Tom, 
but are there any other issues that we haven't discussed that you'd like to share with our viewers? 

>>Angelo:  No, I think I just want to say that being at an advanced stage of my career and pretty far 
down the track that I just want to encourage all those folks who are maybe watching this, especially 
younger people, that things really are better in many ways in terms of the promotion of teaching 
and learning and respect for and the valuing of teaching and learning than they were in the past and 
that I appreciate and value -- and all of my colleagues who do continue to work day-in and day-out 
to improve student learning.  There's really no more important job and so thank you to all those 
folks who are trying to do that. I appreciate it and believe me your neighbors and your neighbors’ 
children and your own children will as well, thank you. 

>>Cooper: Angelo, thank you for being with us. 

>>Angelo: My pleasure.

^M00:33:48