How do you define success for a calculus MOOC?
I’m going to strike now, while the conversational iron is hot, and ask people to define success for a calculus MOOC.
Why?
I’ve already mostly explained why in this recent post, but just in case you missed it, I think mathematics is being threatened by calculus MOOCs, and although maybe in some possibly futures this wouldn’t be a bad thing, in others it definitely would.
One way it could be a really truly bad thing is if the metric of success were as perverted as we’ve seen happen in high school teaching, where Value-Added Models have no defined metric of success and are tearing up a generation of students and teachers, creating the kind of opaque, confusing, and threatening environment where code errors lead to people getting fired.
And yes, it’s kind of weird to define success in a systematic way given that calculus has been taught in a lot of places for a long time without such a rigid concept. And it’s quite possible that flexibility should be built in to the definition, so as to acknowledge that different contexts need different outcomes.
Let’s keep things as complicated as they need to be to get things right!
The problem with large-scale models is that they are easier to build if you have some fixed definition of success against which to optimize. And if we mathematicians don’t get busy thinking this through, my fear is that administrations will do it for us, and will come up with things based strictly on money and not so much on pedagogy.
So what should we try?
Here’s what I consider to be a critical idea to get started:
- Calculus teachers should start experimenting with teaching calculus in different ways. Do randomized experiments with different calculus sections that meet at comparable times (I say comparable because I’ve noticed that people who show up for 8am sections are typically more motivated students, so don’t pit them against 4pm sections).
- Try out a bunch of different possible definitions of success, including the experience and attitude of the students and the teacher.
- So for example, it could be how students perform on the final, which should be consistent for both sections (although to do that fairly you need to make sure the MOOC you’re using covers the critical material to do the final).
- Or it could be partly an oral exam or long-form written exam, whether students have learned to discuss the concepts (keeping in mind that we have to compare the “MOOC” students to the standardly taught students).
- Design the questions you will ask your students and yourself before the semester begins so as to practice good model design – we don’t want to decide on our metric after the fact. A great way to do this is to keep a blog with your plan carefully described – that will timestamp the plan and allow others to comment.
- Of course there’s more than one way to incorporate MOOCs in the curriculum, so I’d suggest more than one experiment.
- And of course the success of the experiment will also depend on the teaching style of the calc prof.
- Finally, share your results with the world so we can all start thinking in terms of what works and for whom.
One last comment. One might complain that, if we do this, we’re actually speeding on our own deaths by accelerating the MOOCs in the classroom. But I think it’s important we take control before someone else does.
mathbabe 
I am not a math guy as such. I took intro to Calc my freshman year of college and remember not a wit. I am numerate, though, and get concepts quickly. If I can be useful in any way on any experiment, use me. Want me to take a MOOC and give feedback? Or be interviewed through? Feel free to use this willing blank slate.
Also, my son just took calculus at a well known university and the professor, he and others say, was terrible. He is a strong student and got little to nothing out of it. I would argue a good MOOC would have majorly trumped this horrible and useless experience. The kids who had taken it in high school said that they were lucky but still often had to refer to their high school notes because he was so bad. I don’t know if a poorly taught calculus student could be useful but if he is somehow, let me know.
George
LikeLike
Did you see the article in the Notices about the Calculus Concept Inventory? This seems like the kind of test of calculus teaching effectiveness you are looking for.
LikeLike
Ooh no I didn’t thanks! And here’s the link: http://www.ams.org/notices/201308/rnoti-p1018.pdf
LikeLike
I think you have to decide what you measuring, how you are defining success, first. As a former college math instructor my first instinct would be to ask how the students do on the next course. I knew which of my collegues were good instructors and which were not based on how their former students performed in the follow up courses which I taught. (They could do the same with me,) The problem is that the bulk of calculus students take no math courses beyond calculus.
One approach would be to institute MOOC and non-MOOC first semester calculus and evaluate second semester work. A danger here is that if the second semester is non-MOOC the MOOC students may be disadvantaged by the strangeness. Vice-versa if the second semeter is MOOC.
I am skeptical about experimental evaluation of teaching methods unless done on a massive scale. Even in the largest university the number of instructors is so small that personal variation would muddy the waters too much. That said the CCI seems like the good start. I qualifiy this by saying I have friends and relatives who are K-12 teachers and I know about “teaching to the test” and the other effects of measing outcomes by a test.
Finally, how will universities determine if MOOC calculus is suitable? That is simpler. They will accept the cheaper per student MOOC if students accept it as a substitute while still paying full fees, or at least maintaining the margin.
LikeLike
Agreed that subsequent work is a great idea. Also interesting would be to see how often the Mooc vs. other students take another course.
LikeLike
Oops. Just saw yours. Yes.
LikeLike
I would add to how well they do they next semester, though I know it’s problematic and might have to be evaluated differently, how many people choose to actually take the next semester. That is, was it too daunting and confusing and unexciting to pursue? Had they reached far enough (i.e., I am going to be a lawyer)? Was it taught so poorly (the material and the enthusiasm) that they just bailed on math and moved on? Did they move on to statistics or some other math? That is, how successful was the teaching in that regard.
LikeLike
Google calculus concept inventory. There was a notices article on it recently.
LikeLike
I did look up the CCI (Calculus Concept Inventory). Here is a link to a survey article that mentions it. http://www.physics.indiana.edu/~hake/ImpactConceptInventoriesB.pdf
My take is that we really don’t know how to test for calculus concepts that matter – exactly what, exactly what types of questions, exactly how to maintain the integrity of any such test. If we want to compare MOOC’s and “traditional” classes on what matters we are stuck. If we are just interested in student performance on working standard problems look to comprehensive calculus finals or even AP calculus exams.
LikeLike
Sorry about the terse comment. Did it from my iPhone. I have some further comments (well, rants):
1) I do believe that it’s impossible to measure the effectiveness of a teaching method, such as MOOC’s, unless you first have an effective way to measure how well students learned the underlying subject, no matter what the teaching method is. And this is not at all easy.
2) In fact, going down this path is exactly what has led to the recent high-stakes testing fiasco. The buzzword is something like “outcomes testing”. Conceptually it’s a great idea; implementing it properly is, as far as I can tell, nearly impossible.
3) The so-called Calculus Concept Inventory (CCI) is based on the analogue in physics, called the Force Concept Inventory. I can’t prove that these test calculus or basic physics better than standard tests, but I am convinced that they do. They are not about “understanding concepts”, because “understanding” is too vaguely defined. Rather it is about testing whether someone can solve questions in physics using *only* the concepts. In other words, whether someone has the skills for doing physics. The CCI was designed with the same goal in mind, but for calculus. I strongly recommend that you all take a look at examples of problems from both exams.
Why de-emphasize calculations? For me it is because if a student can’t use the concepts effectively, why do I care whether they can do calculations or not? Of course, if a student has demonstrated the proper skills using concepts, then it is also appropriate to test them on using both concepts and calculations to solve problems that require both. So it’s not de-emphasis but doing things in a proper sequence.
I also believe that the CCI less important than similar tests for precalculus (i.e., using the concept and examples of functions effectively). To me if a student has a proper understanding of functions, then the concepts of derivative and integral are quite straightforward. But too few textbooks and courses make sure students have a gut understanding of what a function is.
The idea of FCI was probably inspired by the so-called Fermi problems (see, for example, http://en.wikipedia.org/wiki/Fermi_problem).
I’ve said this many, many times: I don’t mind teaching mathematical machinery (like calculus), but STEM graduates would be a lot more effective and valuable in the real world if we simply made sure that they knew how to use 8th grade mathematics really, really well. I have met too many college graduates and even Ph.D.’s in math who stunk at this. I don’t think the mathematical community understands how much this contributes to the negative image of mathematics and mathematicians in the society at large.
LikeLike
Two quick questions: Can you teach to the CCI test? And what does that look like?
In other words, if it (or any given test) becomes more than a diagnostic tool – if it’s an evaluation method – then calculus teachers will be in the same boat that high school teachers are now with the Common Core. On the other hand, if “teaching to the test” actually looks like learning core concepts, then maybe that’s ok.
LikeLike
Teaching to the test is, as far as I know, possible for any kind of test. And that’s never good under any circumstances. This is a big challenge to overcome. Jerry Epstein, who maintains the CCI, tries to restrict who gets to see the CCI problems, but this is obvious not a solution if the test is used on a large scale.
I don’t see any way to eliminate this problem entirely. However, in my department we did for a while mitigate the problem somewhat by simply having exams created by people who were not teaching the course itself. This allowed the teachers to teach whatever they wanted without any restraint without knowing what precisely would be on the exam. Of course, this is also easy to undermine, too. There’s a level of oversight combined with integrity that’s needed for all of this to work properly. That’s really really hard to ensure.
LikeLike
I’ll be my ornery self and say it’s the wrong first question. First question, which is too complicated, is how we measure the success of an education. Is it problem solving? Creativity? Critical thinking? Knowledge mastery? Ability to apply knowledge to physical situations?
Only if we agree on how we measure the success of an education can we measure the success of a MOOC on *any* topic. I personally feel that calculus does not serve the educational needs of a plurality, if not a majority, of students. Maybe I’m wrong. I think discrete math, voting theory, a really good course on symmetry/patterns/groups/beauty, statistics, mathematical modeling are all better things to offer many 18-year-olds. Calculus is like the Greek tragedies. Both are super-amazing and changed the course of human thought and civilization. But maybe Beat poetry or contemporary African literature are actually a better entree to the ways in which literature can change a student’s world or life. We provide a variety of literature so that people can be touched in any number of ways, but we don’t do the same for math.
So the success of a calculus MOOC I’d measure with a differentiation test, an integration test, and some open-answer questions about Riemann sums and limits. Meh. It’s fair, though culturally narrow and potentially a waste of time. I’d rather use open-answer written questions about the concepts of rate of change, net change, and relationships between changing quantities, but then it’s really a test of math writing rather than calculus per se — and that’s how I teach, explicitly, but I don’t think that’s easy with a MOOC that’s designed for content delivery rather than peer writing review.
LikeLike
I agree with the first 2 paragraphs. A large chunk of the Calculus students subsidizing math research are business majors at universities with modest reputations. Most would benefit from learning to apply basic algebra and middle school math skills to conceptually difficult “multi-step” problems rather than additional superficial exposure to higher level tools.
LikeLike
Business majors at most universities don’t take straight Calculus any more. They take a separate business calculus course, which is very, very different.
LikeLike
B-Calc should be Calculus I without trig. It’s hopefully taught by math profs. I’m sure it’s less than that at many schools. Regardless of what it might be called, these students are paying for a big chunk of math research, especially at schools lacking engineering.
LikeLike
Here’s an article about how the Khan Academy evolved that includes some commentary appropriate to this thread…
LikeLike
The MIT folks have presented their analysis of MOOCs at a variety of conferences. They do assess the effectiveness, although in terms of preparing students for other MIT courses.
The grossly simplified result is:
1) MOOCs are a good replacement for the lecture. Putting a few hundred students in a room for an hour to listen to a lecture is highly inefficient. Watching videos on the students’ schedules, with videos properly prepared and appropriate length for learning, is a better use of everyone’s time.
2) MOOCs are not a substitute for residential education. The results are clearly inferior. As an educational charity, MIT feels it to be appropriate to make materials available to the world, because even if the effectiveness were 10% compared with residential, the low cost and large number of people reached makes that 10% highly beneficial to the world.
3) The appropriate mix of electronic and in-person education contact is still being evaluated, and depends upon the subject.
4) MOOCs introduce the “tourist” participants. Over 90% of those who sign up for MOOCs are “tourists”. They want to get a feel for the subject and what it might be like to learn it. They are not seriously invested in learning the course material and generally drop out once their curiosity is satisfied. MIT doesn’t consider their lack of learning significant, because they were never more than tourists.
LikeLike