Online Versus In-class Teaching: Learning Levels Explain Student Performance! AND Are Some Students Better Suited for Online Class...?

I'm reporting on two sessions in this post because they are tightly interwoven and created by the same presenters.

Both sessions began by reviewing the growth rate in online learning (Sloan Report, 2007). In one, we responded to the question "Is online learning for everyone?" Some answers focused on convenience and flexibility (Who is Best Suited for Online Learning? Online Universities Weblog). Some focused on traits commonly cited such as independence, discipline, self-motivation, liking to read and write, ability to stay on task (Is Online Learning for You?). The group agreed that students with these traits would be successful whether learning online or in the classroom.

As to whether there are differences in learning from online vs. onground delivery, both sessions referred to findings in some studies that online is better, in others that classroom is better, and in many that there are no differences (as reported at WCET No Significant Differences Web site). (In this review of findings, there was no analysis offered of whether course design or instructional activities could be the reason for divergent results more than mode of delivery. However, participants in both sessions brought up the question of the alignment with instructional strategies.)

The focus for each session was whether students with different learning styles may perform better online (Index of Learning Styles Questionnaire by Soloman and Felder). To measure student learning performance in this study, the exam had items explicitly aligned with each level of the "old Bloom" taxonomy - 5 items for each level of learning. The exam creators checked reliability of their classifications of items into each of the learning categories.

The course, taught online and onground, had a common syllabus, common final exam, comparable number of students, and students with comparable GPAs. These were some of the differences:

• there were fewer students whose majors were quantitative (e.g., computer science) vs. qualitative fields (e.g., liberal arts) in the online delivery
• there was one instructor for the online course, and multiple instructors for the onground delivery
• the instructional activities and quizzes during the course were idiosyncratic to the preferences of the TA's, thereby making the instruction different in each section of the course

Using classic inferential statistical analysis, the study seems to show that extremes at either end of some learning styles, e.g., "active-reflective," do better than "balanced" learners in an online class.

While there seemed to be no differences in online vs. onground learning except in synthesis outcomes, the structure of the assessment design in this study holds promise for sharper analysis of impact on learning outcomes. The data produced some controversial speculation about whether there will be NSDs at all but the synthesis level; discussion in each session pointed out a couple of possible pitfalls in analysis - namely, if the course has not been designed and delivered with synthesis learning activities in the online as well as onground course sections, then it is not valid to conclude that either mode of delivery has produced better results than the other.

The data showed, after controlling for GPA and differences in qualitative majors, that online students did not perform as well on the synthesis items. But the study did not show the reasons.

The design also omitted the "evaluation" level of learning, so performance on that level was not tested.

The paper on this research should be developed in about a month, and I've requested a copy of the complete report from each of the presenters in each session. One of the presenters expressed interest in finding out more about instructional strategies known to work for different types of learning outcomes, about Webb's curriculum and assessment alignment model, the hierarchy of Gagne and Briggs, and re-examination of what is now defined as the highest level of learning in the "revised Bloom's taxonomy."

I think this was a well designed study and could serve as a good model for future exploration.

Presenter: Milind Shrikhande and Richard Felder Georgia State
University

Community, Voices, and Portals of Engagement

Our luncheon keynote speaker probably was thinking globally, while a biochemist Margie Paz, originally from the Phillipines, and now at Griffin, Georgia, talked with me, and we discovered that we both had attended Iowa State University in Ames, Iowa. And, we shared some stories about Ames, Des Moines, Griffin, and the Phillipines.

Our conversation was good prep for Dr. T’s fresh perspective on Yo Yo Ma’s “Silk Road Ensemble” and the dimensions of learning in that project. I was struck by the concept that a body of knowledge is unknowingly privileged by particular voices. Dr. T. showed us how threads of perception of the same event in historical images provide different knowledge about that event, and how students can be engaged in discovering, in a digital community, these different types of knowledge. She showed us the project “Perry Visits Japan.”

Dr. T focused on the hybrid of cognitive and sensory frameworks to transform comprehensive and critical analysis, and how these come to define “signature” pedagogies, such as “grand rounds” in medical teaching.

As I write these notes a day later, two teachers across the hallway are discussing spiritual understanding and personality in the teaching process. This is not unlike the concept that Dr. T. mentioned of designing “open-room for debate and interpretations” into the assignments that we create for students. She reflected on the challenge for many teachers of planning inquiry with open-ended studies and absence of conclusions into a study process.

In this context, layering communities and voices into portals of engagement led to the creation of “research teams” whose members came from different countries, who were provided with real data about a pervasive real-world health problem, and were invited to decide what to do with the data. This approach was much less prescriptive than many learning assignments. The teams had to
decide what was important to them in their personal lives, what they care about most, and then could proceed with deciding what to do about the data.

Dr. T. described the distance learning design-studio concept. She added questions about what to visualize from what perspective? What to scaffold? How to develop productive habits of mind? Reflection on McLuhan’s (1961) The Gutenberg Galaxy notion of extending our senses outside of us with the social world, and how this is like what happens when a new note is added to a melody. And what biases do we bring to the curriculum?

Keynote Speaker: Dr. Kathy Takayama

A Methodology for Developing and Validating an Assessment Rubric

The techniques in this study are commonly expected in the process of developing a legally defensible certification exam. The presentation, however, focused on exceptionally rigorous application of these techniques for assessment in a context of teaching.

With the goal of improving a rubric's validity and reliability, the process in this study began with development of learning objectives and a review of related professional guidelines and literature before engaging in academic discussions about the constructs in the rubric. Representatives from the academic area and from the profession were engaged in discussions about criteria and weights for the rubric. A process of consensus moderation known as the Delphi technique was used to reach agreement among review groups, thereby reducing variance in scoring.

Among the reported results, the research team reported the observation that there was more agreement in using the rubric with an "A" paper, but below that level, variance becomes greater as teachers consider how to give feedback without discouraging the lower-performing students. The questions after this observation: What is fair? Don't students deserve equitable evaluation? Or do we want to encourage a system in which students "section shop?"

The rubric in this study was a "News Release Evaluation Rubric" had a left column with five major constructs, each with a checklist of elements, and a right column with a % weight for each major construct.

Presenters: Sandra Allen, Columbia College Chicago and John "Ed" Knight, University of Tennessee at Martin.