In higher education, online programs are chasing QM Certification. What does that mean? Quality Matters is a lot of things: a non-profit, a methodology, a rubric.
Instead of forcing instructors to all become instructional designers with QM training, all schools should have dedicated online education experts. Basic shells should be prepared by these instructional designers, supporting educators. Educators have enough to do without more responsibilities.
If the IT/ID teams want to adhere to QM methods, fine. But don’t ask instructors to become designers, in addition to being subject matter experts.
I dislike Quality Matters. It embodies many trends in higher education I dislike and resist, including: rubrics, checklists, processes over content, certifications, jargon, acronyms, and commodification.
Rubrics come in many forms, and most of them are flawed for a reason economists long ago recognized: you get what you measure. If you tell students in a science course that 20 percent of an essay grade is grammar and 20 percent is formatting, you’ve suggested to them that 40 percent of a grade has little or nothing to do with subject matter. At the same time, rebalancing the rubric might cause students to ignore language and formatting.
When I’ve been asked to use rubrics, I always include some “all or nothing” statements. If you fail to meet the assignment objective, that trumps the points of the rubric. There are good and really bad rubrics, but I have yet to see a great rubric.
I love checklists and following processes. You cannot bake cookies without a checklist and a clear process. That’s not the sort of situation in which “process over content” becomes a concern. Instead, I’m talking about when we emphasize a formal process inflexibly and end up with content that’s undesirable.
In computer programming courses I was taught to always flowchart a routine or function. The flowcharting received so much attention that students created beautiful charts… and broken code. That seems impossible, yet it happened time and time again. Flowcharts, design documents, procedural manuals, and other planning tools have value only if they support and contribute to a goal without getting in the way.
Change orders are a good example in programming. You could spend hours, days, or even weeks documenting a use change request for an enterprise project. By the time the change is approved, the user has realized a different function or feature better meets current needs. Companies that adopted strict procedures were slow to adapt. That’s how “agile” coding methodologies became popular: they emphasize functionality over process.
Certification insanity makes credentials the only accepted measure of expertise. That’s wrong, and most of us know it. Plenty of great instructional designers started before there were certifications in the field, much less degree programs in course design. Certified Peer Reviewer. Master Reviewer. Course Review Manager. QM offers plenty of certifications… all for a price. Someone benefits from the gatekeeping, especially those who have already endured the ritualistic hazing.
To make something a magical cult, it needs mysterious language. That’s where the acronyms and jargon enter. If Quality Matters is “QM” and the staff is talking about GSs, SRSs, CBEs, APPs, MRCs, CPRs, and so on.
A quick intro to QM from the Quality Matters website:
QM originated in 2003 as a Fund for the Improvement of Postsecondary Education (FIPSE) grant awarded to MarylandOnline, Inc. (MOL), titled “Quality Matters: Inter-Institutional Quality Assurance in Online Learning.” The purpose of the grant was to improve the quality of shared online courses by establishing an inter-institutional peer-reviewed quality assurance certification process.
Evolution
Interest in the grant and the process it was creating spread quickly beyond MOL. When the grant ended in August 2006, it was determined that the work begun during the grant should continue. In September 2006, the Quality Matters grant became the Quality Matters Program, a non-profit, self-supporting program of MOL. In 2014, the Quality Matters Program began operating as Quality Matters, a non-profit organization with its own Board of Directors.
Who could be against “quality” in education? And we’re going to assure it! How? By developing a standardized, uniform approach to “shared online courses” that offered credit across multiple college and universities. If everything is the same, across campuses, we’re somehow closer to quality. Oh, but not the content. Trust us. We won’t dictate the content, only the user experience with that content.
“Self-supporting” means we need to earn money. Hello, membership fees. Hello, certification fees. But don’t worry, no one is making any money. I view QM as “non-profit” in the same league as ETS (Educational Testing Services) and object to its approach to online education.
At the core of Quality Matters is the QM Rubric, which is used to evaluate online courses. There are 42 (ask Arthur Dent why) Specific Review Standards within eight General Standards. A 42-item checklist would be laughable, but I’ve had to complete the forms and perform a mock evaluation. It’s far too many items to be useful.
The eight General Standards in the Quality Matters Rubric are:
- General Standard 1: Course Overview and Introduction
- General Standard 2: Learning Objectives (Competencies)
- General Standard 3: Assessment and Measurement
- General Standard 4: Instructional Materials
- General Standard 5: Learning Activities and Learner Interaction
- General Standard 6: Course Technology
- General Standard 7: Learner Support
- General Standard 8: Accessibility and Usability [Of course accessibility is last.]
The “Course Worksheet” is a lengthy document, used by evaluators (and ideally the instructor) to determine the strengths and weaknesses of an online course design. It took me several hours to complete a CW for a mock peer evaluation. That experience was enough to never take the process seriously. Technical communication scholars know that long questionnaires encourage people to take shortcuts and omit data.
The major headings for the Course Worksheet (CW) include:
- Basic Course Information: Questions 5 – 11
- Course Format: Questions 12 – 18
- Course Learning Objectives, Assessments, & Instructional Materials: Questions 19 – 28
- Course Interaction Components: Questions 29 – 39
- Instructor Perspectives: Questions 40 – 44
- Items 1-4: Institution / Course information
I hate everything about QM, but I hate its embedded ableism most of all.
Accessibility as an Afterthought
It required 61 presses of the tab key to reach the “Module 5” quiz that we were expected to evaluate in the QM course. Imagine striking tab 61 times. Again and again and again. Plus, trying to remember where you are in the other materials. When you leave the page or tab between windows to get between the homework form and the sample course, the darn navigation resets. Therefore, 61 tabs (or more, since you need to get into the quiz) again. And… again…
And I missed the “right” link more than once with my palsy.
I decided to record my efforts and send them to the university IT department because I was so angry.
With adaptive technology, one of the quizzes required 2 hours and 20 minutes for me to complete. I passed (7/8) but the physical and cognitive costs were inexcusable. For several hours, I experienced palsy attacks and seizures, leading to a severe migraine. The system is not accommodative of physical and cognitive differences. The question I missed was the result of just giving up trying to navigate multiple tabs and screens to gather information.
QM is not, ironically, meeting universal design principles for actual users. They just checked off that they have captions or headings. Usability includes not forcing users to constantly switch about a screen.
Here is the expected schedule for the online training, from the QM schedule:
- 1.5 Hours, One Quiz
- 1.5 Hours, Three Quizzes
- 2.0 Hours, One Quiz
- 3.0 Hours, Two Quizzes
- 3.5 Hours, One 60-point Quiz
- 1.0 Hour, One Quiz
- 2.3 Hours, Two Assignments
- 1.0 Hour, One Quiz
- 1.0 Hour, Two Assignments
Almost 17 hours. I need at least twice the allocated time. This is not equitable, a core principle of online education. This clearly penalizes those not attending the other format and penalizes those with special needs even more. Obviously, my anger is increasing the more I reflect on this process and the model that was presented.
Pedagogy Models Stink, But QM Is NOT a Pedagogy
I’m not a fan of “pedagogy models” or diagrams suggesting hierarchies of learning stages. Yes, I have had to reference Bloom’s Taxonomy in my lesson plans. I’ve had to associate my course materials with various circles, triangles, and webs supposedly illustrating how we learn.
We do not learn following a linear path, or in stages represented in a triangle. That’s nonsense. I realize proponents argue that models are merely starting points, but many new teachers are told, “Students cannot evaluate and analyze until they’ve progressed upwards through the stages.” I had to memorize the mnemonic ESAACK: evaluation, synthesis, analysis, application, comprehension, and knowledge. The goal, the ideal, is evaluation. My education courses suggested most students never reach that tip of Bloom’s Taxonomy.
At least Bloom’s Taxonomy has something to do with how people learn. Quality Matters has absolutely nothing to do with how we learn. Nothing. It has everything to do with its 42-item rubric. As pedagogy scholar Jesse Stommel notes, administrators and regulators love models, even though there’s no evidence that models do much. A great teacher might learn from models and then develop a personal approach.
Stommel offers an insightful critique of Quality Matters. He begins with an overview of models in general:
Not Taking Bad Advice: a Pedagogical Model
Jesse Stommel
Executive Director, Hybrid Pedagogy, a nonprofit supporting better teaching (2011 – present)
26 Jul 2020The text for my flipped keynote at Digital Pedagogy Lab 2020.
I’ve never created a model for online, digital, or hybrid pedagogies. As long as I’ve been teaching, and as long as I’ve been teaching teachers, I’ve encountered, been
flummoxed by, or have cast off models. I have yet to see a single pedagogical model worth its salt. And yet I watch how quickly models spread. The neater and tidier the model, the more likely it seems to be broadly adopted by an institution: Learning Styles, Bloom’s Taxonomy, ADDIE, Scaffolding, Design Thinking, Quality Matters, Andragogy, HyFlex. Lists, frameworks, Venn diagrams, rubrics, templates. Six principles of Andragogy, five stages of the ADDIE development process, six levels of Bloom’s Taxonomy, forty-two review standards of the Quality Matters Rubric. And, even when these models are thoroughly debunked, they continue to retain traction. According to the Association for Psychological Science, “No less than 71 different models of learning styles have been proposed over the years … But psychological research has not found that people learn differently, at least not in the ways learning-styles proponents claim.” And, yet, the American Psychological Association found, “in two online experiments with 668 participants, more than 90 percent of them believed people learn better if they are taught in their predominant learning style.” In higher education, too many of us cling to other people’s models, because we have rarely been taught, encouraged, or given the support we need to create our own.
Quality Matters hides behind its complexity and mysterious language. As Stommel explains of such models:
Some of the more insidious models are fashioned as needlessly complex in order to create a mystique of intellectual rigor. Even when these models aren’t based in research, they are made to seem as though they are.
Pedagogy is the science and art of teaching. It’s when a teacher adjusts lessons mid-class session to meet students where they are… to guide students towards where they need to be. Pedagogy requires knowing your students as people and yourself as a teacher. I have taught four sections of a class in a semester, and all four were different. That’s pedagogy: applying research and best practices in a classroom.
Pedagogy is praxis, the intersection between the philosophy and practice of teaching. Best practices, which aim to standardize teaching and flatten the differences between students, are anathema to pedagogy. The most egregious example is the Quality Matters rubric, which the organization (a standalone non-profit as of 2014) calls the “QM Quality Assurance System,” which is said to “create a culture of continuous improvement” to “deliver the promise” of online learning. I have previously analyzed the marketing copy of Turnitin and various learning management systems. This work is not a mere exercise. When we’re deciding what tools we use, we should be looking careful at the discrepancies between what the companies say the tools do and what they actually do. At the points where these diverge, we begin to see the cracks in a tool’s promise.
From the front page of the Quality Matters Web site: “With online learning, everyone has a goal. Learners need to improve and grow. You work to nurture them with
well-conceived, well-designed, well-presented courses and programs.”
Hold on here, you should be thinking, pedagogy cannot be a single design approach. One process cannot be “best” for all groups of students or all teachers. Quality Matters promises the implausible.
When I’ve watched colleagues with the most complex rubrics, longest checklists, and most detailed syllabi, I’ve noticed they panic when anything deviates from the plan. Those instructors get lost in the process, at the expense of teaching content. We all know what happens with long online documents: “TL/DR” (Too Long / Didn’t Read).
If you have to tell readers something is clear… it is not clear.
Quality Matters is a special kind of model, almost the polar opposite of Bloom’s, because QM is decidedly not at all simple. It is, in fact, needlessly complex to the
point of being inscrutable, which ultimately helps QM sell an annotated version of the rubric, as well as courses, workshops, certification programs, subscriptions, and institutional memberships. In the rubric itself, the words “clear” or “clarity”
appear over a dozen times. (There are 42 items on the latest higher education rubric and 43 on the latest K-12 rubric, all with point values assigned to them, a mechanism for tallying a final score.) Item 5.4 on the higher education rubric, for example, says that “the requirements for learner interaction are clearly stated.” While I do think teachers should make their requirements for a course clear, writing longer syllabi, adding more items or levels to a rubric, and spelling out more requirements do not (in my experience) make anything more clear for students.
Teaching shouldn’t be like engineering widgets. Quality Matters tries to be the ISO 9000 of online learning as if our schools manufacture outcomes. Documenting everything often improves nothing. Quality Matters suggests, however, that documenting magically addresses problems with online course designs. Yet, the problems in education aren’t addressed by QM. Instead, QM documents and reinforces what administrators want.
Stommel looks carefully at the language of the QM Rubric.
In “The Trouble with Rubrics,” Alfie Kohn writes, “Consistent and uniform standards are admirable, and maybe even workable, when we’re talking about, say, the manufacture of DVD players.”
QM promises efficiency and objectivity, but actually creates more work and merely provides cover for all the biases and subjectivity that continue to exist (and are left unchecked) in spite of the rubric. The words “access,” “accessible,” and “accessibility” do appear five times on the Quality Matters rubric, but each use refers to either providing “policies” and “statements,” or having accessible “text and image files” and “multimedia content” that meet “the needs of diverse learners” (the only context
in which “diversity” is mentioned). This just doesn’t go even nearly far enough. The word “privacy” appears once, but the onus is put on students to protect their data and privacy; the course needs only to provide the necessary “information.” Here are a few words that do not appear anywhere on the Quality Matters rubric: “community,” “agency,” “inclusivity,” “flexibility,” “joy,” “compassion,” “question,” and “human.”
Quality Matters feels like Henry Ford trying to tackle online education, and flailing about miserably. There’s no innovation. There’s no revolutionary insight within the rubric or other QM materials. Let’s all be the same. Teachers, let’s all follow the same model. Students, learn this model and follow it. Learning will surely happen.
Our efforts toward building community should be directed toward the students who need that community the most, the ones most likely to be feeling isolated even
before the pandemic: disabled students, chronically ill students, homeless students, BIPOC students, LGBTQ students, etc. We need to build courses, and imagine new ways forward, for these students, the ones already struggling, already facing exclusion.Flexibility and trust are key principles of any pedagogy worth its salt, but they are particularly important when we’re in crisis. Right now, we need to focus on “teaching the students we have, not the students we wish we had.” We don’t need more models for hypothetical students we haven’t yet met. And so, my pedagogical model has exactly one point:
Stop looking for models and begin by talking to students.
About: Jesse Stommel is co-founder of Digital Pedagogy Lab and Hybrid Pedagogy: The Journal of Critical Digital Pedagogy. He has a PhD from University of Colorado Boulder. He is co-author of An Urgency of Teachers: the Work of Critical Digital Pedagogy.
Quality Matters repelled me. It is one of several reasons I’m not returning to an adjunct post. Are there good practices within QM? Yes. But, there are also some horrible ideas I reject, such as constantly referencing learning objectives and outcomes throughout a shell. No student will page back and forth to decode “LO 4.2” in an assignment description. The course shell becomes a text-dense maze students won’t read through. It is exclusionary, in practice.
As I wrote above, the training by Quality Matters was inaccessible. It was ableist, classist, and exclusionary.
The real accomplishment of QM is that it erects another barrier between students and teachers.