Sign In

Communications of the ACM


CS Classes Have Different Results Than Laboratory Experiments – Not in a Good Way

View as: Print Mobile App Share:
Mark Guzdial

Georgia Institute of Technology professor Mark Guzdial

I have collaborated with Lauren Margulieux on a series of experiments and papers around using subgoal labeling to improve programming education. She has just successfully defended her dissertation. I describe her dissertation work, and summarize some of her earlier findings, in the blog post linked here.

She had a paragraph in her dissertation's methods section that I just flew by when I first read it.

Demographic information was collected for participants’ age, gender, academic field of study, high school GPA, college GPA, year in school, computer science experience, comfort with computers, and expected difficulty of learning App Inventor because they are possible predictors of performance (Rountree, Rountree, Robins, & Hannah, 2004; see Table 1). These demographic characteristics were not found to correlate with problem solving performance (see Table 1).

Then I realized her lack of result was a pretty significant result.

I asked her about it at the defense. She collected all these potential predictors of programming performance in all the experiments. Were they ever a predictor of the experiment outcome? She said she once, out of eight experiments, found a weak correlation between high school GPA and performance. In all other cases, "these demographic characteristics were not found to correlate with problem solving performance" (to quote her dissertation).

There has been a lot of research into what predicts success in programming classes. One of the more controversial claims is that mathematics background is a prerequisite for learning programming. Nathan Ensmenger suggests the studies show a correlation between mathematics background and success in programming classes, but not in programming performance. He suggests over-emphasizing mathematics has been a factor in the decline in diversity in computing (see blog post here about this point).

These predictors are particularly important today. With our burgeoning undergraduate enrollments, programs are looking to cap enrollment using factors like GPA to decide who gets to stay in CS. (See Eric Roberts' history of enrollment caps in CS.) Lauren's results suggest choosing who gets into CS based on GPA might be a bad idea. GPA may not be an important predictor of success.

I asked Lauren how she might explain the difference between her experimental results and the classroom-based results. One possibility is that there are effects of these demographic variables, but they're too small to be seen in short-term experimental settings. A class experience is the sum of many experiment-size learning situations.

There is another possibility Lauren agrees could explain the difference between classrooms and laboratory experiments. We may teach better in experimental settings than we do in classes. Lauren has almost no one dropping out of her experiments, and she has measurable learning. Everybody learns in her experiments, but some learn more than others. The differences cannot be explained by any of these demographic variables. 

Maybe characteristics like "participants’ age, gender, academic field of study, high school GPA, college GPA, year in school, computer science experience, comfort with computers, and expected difficulty of learning" programming are predictors of success in programming classes because of how we teach programming classes. Maybe if we taught differently, more of these students would succeed. The predictor variables may say more about our teaching of programming than about the challenge of learning programming.


Alfred Thompson

Back in the 1970s when I was looking for my first software development job companies were using all sorts of tests and "metrics" to determine who would be a good programmer. I'm not sure any of them had any validity. I don't know that we have any better predictors today. In my classes these days I see lots of lower GPA students who do very well in computer science classes.Maybe it is how I teach. Maybe it is something else (interest?) but all I really know is that I want to learn better how to teach.

Alwyn Goodloe

I'm not in academia but work in a lab where we get to see lots of student interns. I think much more than say math or EE, there is a wide range of what gets taught and how it gets taught. It's very important for students to land in a place where both align well.

Most "programming" classes say they are teaching "principles" but lets be honest what most students take away is "syntax". I think the new CMU curricula really does aim at principles. Our labs are usually really and in most school run by TAs lacking in communication skills. My program was very picky about letting PhD get in front of students, but I've been around long enough to know this isn't normal. Students who can barely speak English get tossed into TA duties and equally bad native speakers who simply have no idea how to run a classroom. Invariably faculty don't want to hear about it.

Displaying all 2 comments

Sign In for Full Access
» Forgot Password? » Create an ACM Web Account