1) 
Outcomes Do Matter, and a Unified Framework is the First Step
I strongly disagree with Aaron Doering. In the second section of Panel II: Implications of Cyberspace, he made what seemed to me to be an exceptionally strong stance on the need to measures outcomes from the implementation of new technologies for learning, stating that no student would ever face the decision between a media-empty and media-rich class, so measuring outcome differences between online and face-to-face courses is unneeded.
This is simply untrue. For example, in my home department, Psychology, we offer a course which can be taken either online or in person. It is an identical course. The only difference between the two sets of sections is that the lecture and discussion component is given entirely online for the online course. The lectures are the same - the online course receives a webcam-captured version of the face-to-face class.
So imagine this: if there was a consistent difference between online and face-to-face class outcomes (say, for the sake of example, students in online courses received consistently better grades with the same instructors and materials), students would be faced with the choice of taking a course that produces lower or higher grades (on average), which would be a very real and important decision for them. Our high-minded ideals of “but you learn the same amount either way” would amount to little for their decision and for their GPA, which does have important implications for future outcomes for them (graduate school, work after college, etc).
A gentleman also raised another important point in this session - “is there any conclusive evidence that online is better than face-to-face?” - which received a similar answer from, I believe, PZ Meyers. He claimed it didn’t matter to him, since an in-person component was present in his class anyway. This is also unrealistic when putting this in the larger perspective of education as a whole. Some classes DO have online components alone. This is an interesting and valid question - just because you could have a face-to-face component in addition to an online component doesn’t mean you will.
I don’t know about findings in education, but my home field of industrial psychology actually does shed some light here (industrial psychology is a relatively small field, but focuses on the application of psychological principles to the business world - such as training on the job). A meta-analysis on online vs. face-to-face training interventions by Sitzmann et al (2007) provided aggregated evidence across 96 studies that showed that, on average, online studies are no worse or better than face-to-face training. However - and this is an important however - when more training techniques were used in the online training, students did do better.
This leaves several unanswered questions. Is it simply because there are more training opportunities in the studies with better outcomes, or is is because the nature of those training opportunities is qualitatively better than traditional (usually lecture-based) training?
This is the focus of a study that I am conducting right now - and improvement and expansion on the Sitzmann meta-analysis, taking into account the actual composition of the training interventions. Do they use blogs? Do they use wikis? Is there a video component? I am sorting through several hundred studies and aggregate the results in order to understand more targetted questions, e.g. “Does online training that uses wikis produce a bigger online-vs.-face-to-face difference than online training that doesn’t?”
The major problem that I face with this is that most studies treat cyberspace-driven training interventions as “something we did,” and not as a collection of dependent features. For example, descriptions might be like “WebCT was used.” Well, what parts of WebCT were used? Did you do real-time assessments? Video presentation? Lecture materials as PDFs?
This is one of the MAJOR challenges that any multidisciplinary examination of these ideas must face. How do we define any of these inteventions? MySpace will come and go. Facebook will come and go. Google will (probably) come and go. But what makes these technologies similar or different? What is ACTUALLY changing when someone migrates from one social networking service to another for their training or education needs?
This is why we need to develop a unified taxonomy and framework to work within in order to even BEGIN to talk about these issues. We cannot talk about “the benefits of MySpace” because MySpace isn’t what’s powerful - it’s the tools that MySpace provides. This also escapes the trap of technological obsolescence - if we study the components and not the products themselves, then our research will not obsolete until the components fall out of favor - not just until Google’s stock dries up. This also keeps the research fresh, and the grant dollars coming in - as new technologies develop, they must be placed in the taxonomy and monitored as they develop. Creating this taxonomy and framework is what I would like to do, and it truly is a multidisciplinary question. We cannot address this in just one field without limiting ourselves artificially by the technologies available to and commonly used in that field.
So I suppose, in the end, this will sound like an AA introduction: My name is Richard Landers, and I need some collaborators!
(rlanders.filedrawer.org)
I tend to agree with you, Richard! Tech versus no-tech comparison studies are out of fashion in educational circles, I think in part because of the Clark-Kozma debate a number of years ago. One result of that debate was the conclusion that technology doesn’t matter, in and of itself, to educational outcomes — it’s the educational methods, the teaching-learning techniques and practices, that make the difference.
That said, I think that comparison studies can be quite revealing and useful, precisely because new technologies change the teaching-learning activities in classes. And it’s not quite true that all such studies show “no significant difference”. Several recent meta-analyses of comparison research show that different ways of using technology in education do make a difference — some positive and negative differences. My attempt at a brief synopsis with bibliography is at http://dmc.umn.edu/spotlight/tel-effectiveness.shtml.
Very interesting! I have to admit that I’ve never heard of this before - likely because industrial/organizational (I/O) psychology would appear to be several steps removed from Educational Technology. I will definitely take a look at the book you reference though.
We also largely abandoned significance testing at all in I/O a number of years ago, as they are so easily influenced by sampling error and sample size. Most of the research these days simply reports effect sizes, and there seem to be several small but meaningful effects in many places (although it is difficult to get a good estimate of their relative importance in a single study, of course - thus my current research).
I had heard the conclusion that the technology itself doesn’t matter before, but I think this may be a bit short-sighted. Most of that occurred in times when the primary purpose of the technology was to replicate the in-person experience, wasn’t it? (This is a guess!) These days, technology is enabling entirely new methods of instruction which may not be easily replicated across mediums (what is the in-person equivalent of a blog?), which might make this question more relevant? (Let me know if this sounds crazy!)
My recollection is that PZ’s answer was that his class has meetings, textbook, etc, all the features of a “regular” class plus the blogging, which he classified as outreach. So I don’t recall him saying that he didn’t care, just that the question as posed did not describe what he is doing.
HI! A healthy discussion is always a good thing. Just to be clear, I was referring to how we do research with technology as I reference the plethora of discussions regarding the “media debate” in our field. It is important that we measure the outcomes with new technologies and especially when looking at the various pedagogical strategies with these new technologies. Also, as Walker says above, there are many articles that reveal a significant and non-significant difference. Thus, the importance of not making sweeping comments (the “Big Wrench”), but being aware there are always two sides. Here are just some of the original articles and there are many more recent ones. As you can see, the debate has been around a very long time. Enjoy and have a great week!
Clark, R.E. (1994). Media will Never Influence Learning. Educational
Technology Research and Development, 42(2), 21-29.
Kozma, R. B. (1994). Will media influence learning? Reframing the debate.
Educational Technology, Research and Development, 42, (2), 7-19.
Morrison, G. (1994). The Media Effects Question: “Unresolvable” or Asking the Right Question.
Educational Technology, Research and Development, 42, (2), 41-44.
Tennyson, R. D. (1994). The big wrench vs. the integrated approaches: The
great media debate. Educational Technology Research and Development, 42 (3), 15-28.
Shrock, S. (1994). The Media Influence Debate: Read the Fine Print, But Don’t Lose Sight of the Big Picture.
Educational Technology, Research and Development, 42, (2), 49-53.
Reiser, R. (1994). Clark’s Invitation to the Dance: An Instructional Designer’s Response.
Educational Technology, Research and Development, 42, (2), 45-48.
Ross, S. (1994). Delivery Trucks or Groceries? More Food for Thought on Whether Media (Will, May, Can’t) Influence Learning.
Educational Technology, Research and Development, 42, (2), 5-6.
Jonassen, D., Campbell, J., & Davidson, M. (1994). Learning with Media: Restructuring the Debate.
Educational Technology, Research and Development, 42, (2),31-39.
I suppose the thing that’s bothering me about this whole debate is that most of my quantitative training over in psychology has supported the idea that statistical significance testing doesn’t really tell us anything. It is very unlikely that changing media types actually “doesn’t matter.” It obviously changes the qualitative experience, and accordingly there must be some degree of change in terms of learning outcomes, even if that difference is fairly small.
You are simultaneously supporting two views which I frankly can’t reconcile. Perhaps you could explain it to me? You state “Thus, the importance of not making sweeping comments (the ‘Big Wrench’), but being aware there are always two sides.” Assuming that there are two distinct, separate sides is itself a sweeping comment, isn’t it? Education is not a natural dichotomy. Isn’t it more likely that everything affects everything else to some extent? The convenience of concluding “there is a difference” or “there is not a difference” is only that - a convenience - and not a representation of the actual state of the world, so to speak. “No significant differences” should be a starting point - not an end in and of itself. All this tells us is that the differences are too small to be distinguished from chance levels with the sample sizes commonly found in these studies - it tells us little about how to minimize or maximize those differences.
(Sorry if this sounds like a sermon, but I teach statistics! Such issues are on my mind often.)
I will read into your field a bit to reconcile my perspective, but most of today was honestly baffling to me - the perspective of education towards all of these issues seems to be vastly discrepant from the way psychology/social science approaches them. I suppose this is the value of such a multidisciplinary initiative?
For reference, here are a few articles that exemplify my view on significance testing (I am unaware of how well known this is outside of psychology, so hopefully these will be helpful):
Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method. American Psychologist, 56(1), 16-26.
Cohen, J. (1994). The Earth is round (p < .05). American Psychologist, 49(12), 997-1003.
Hauer, E. (2004). The harm done by tests of significance. Accident Analysis and Prevention, 36, 496-500.
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods, 1(2), 115-129.