by Ellen Cushman, Northeastern University
Legacies of imperialist thought permeate understandings and uses of validity. This essay discusses the ways in which the concept of validity creates the colonial difference as it maintains social, epistemic, and linguistic hierarchies. Recent scholarship in writing assessment, particularly in this special issue, is considered in light of this critique as it offers ethical and methodological alternatives to help address issues of equity, representation, and fairness. Though attempts to revise the Standards to address issues of fairness have been made by the American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], these remain partial and incomplete. I propose a decolonial option to modernist notions of validity, one that asks scholars of assessment and test designers to dwell in the borders of the colonial difference that validity, as both hermeneutic and instrument, has helped to create and ensure. Such a decolonial approach would see validity evidence tools, not as a way to maintain, protect, conform to, confirm, and authorize the current systems of assessment and knowledge making, but rather as one way to better understand difference in and on its own terms.
“If we are to ask and answer the really tough questions about validity in terms of fairness, as the new sixth edition of the Standards attempts to do, we must understand the trajectory of its historical past” (Elliot, 2015, p. 677).
My entire professional life, I’ve been keenly interested in understanding the ways in which language and meaning making relate to power in the everyday lived realities of peoples in communities, classrooms, and Nations. My efforts have been to better understand where, how, for what reasons and with what outcomes knowledge can be made with peoples.
In The Struggle and the Tools, I investigated the ways in which change and linguistic striving were conceptualized and practiced every day in an ‘inner city’ in upstate New York. The Cadenses linguistic knowledge is evidenced in processes and practices unfolding in highly asymmetrical power relations. The stories they shared with me on the front stoop, around kitchen tables, and in social service waiting rooms ask for a scholarly recalibration of what counts as learning, triangulation, and rhetorical strategies. In similar fashion, I turned my attention to writing course curricula, outreach, and engagement asking if it is indeed possible to change pedagogies and content in ways that bring university students and community members into learning relationships with each other. And what about the tools used throughout these contexts, is there a way to level the hierarchies associated with these? This led to research with my tribe, the Cherokee Nation, in Tahlequah, Oklahoma to better understand how the Cherokee writing system still impacts learning and language use.
In The Cherokee Syllabary: Writing the People’s Perseverance (2012), I traced the ways in which the creation of the syllabary, an 85-character writing system, accrued instrumental, cultural, and historical value over time, lending to the perseverance of my peoples. This book led to further questions about where it is that social change takes place, why and how it gets stymied, and the ways in which various epistemologies and knowledge making practices and instruments come to be valued. In short, I’ve been dwelling in the borders created by Western imperialism.
Throughout it all, my goal has been to question and revise knowledge made in literacy studies in light of stories and practices in tribal and local communities as well as university classrooms. I’ve attempted to shift the paradigms of what is known as literacy, literate, and the letter (2012); tradition, artifact, collection (e.g. of data or objects, 2013); and story (2014). With Mary Juzwik, I’ve also been co-editing the flagship journal of the National Council of Teachers of English: Research in the Teaching of English. In that role, we have sought to decolonize English education by delinking each of the terms in the title of the journal. Thus, we have sought articles that help to reframe research as storying, teaching as relationships, and English as Englishes. The overarching goal has been to delink these concepts from their grounding in Western imperialism, to expand the content of what counts as these key terms, and to explore the tenets that structure them.
Taken together, my work has concerned itself with several key concepts that remain symptomatic of the imperialist legacies of knowledge making in academe today. At the crux of this work has been the decolonial assumption that what counts as knowledge and evidence of this knowledge—what is valid—is to be found in all peoples’ understandings and rhetorical struggles. As a Cherokee scholar of literacies and journal editor, my contribution to this special issue, then, is to think more broadly about the imperialist legacy of validity as a structuring tenet of modernist thought. My hope is to broaden the discussion of ethics in writing assessment and the critique of validity from these perspectives.
Following Elliot, I will trace the trajectory of the historical past of this term in order to ask the really tough questions about validity and fairness. Unlike Elliot, though, I’ll begin with the Renaissance, when so many of the tenets of imperialist thought took hold in Western Europe. I argue that validity works as a tenet of modernist thought by identifying both what is objective and what is evidence, and by so doing, creates and maintains imperialist hierarchies of difference. I’ll then consider ways in which current assessment tools and research methods have attempted to address questions and concerns this history brings to light. For example, the authors in this special issue have attempted to shift the onus to change onto scholars of assessment and designers of tests and have outlined methods and practices for doing so. While recent revisions to the Standards (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014) have attempted to redress concerns about equity, fairness, and bias in the interpretation of test results, these efforts remain partial and incomplete as the authors in this special issue amply demonstrate. I end the article by suggesting how a decolonizing approach can be part of a discussion of ethical, fair, and valid writing assessments.
Few terms matter as much to writing assessment and social justice as validity. Here are just a few ways in which validity emerges in day-to-day workings of the university. Writing assessment validation tools influence who has access to college, the placement of writers into college writing classes, the successful completion of undergraduate writing courses, and the students’ time to degree. “Thus, validation tools that help us understand the local effects of writing assessment can be critically important: use of these tools may mean the difference between college success and failure for many students” (Poe, Elliot, Cogan, & Nurudeen, 2014, p. 589). In addition to being central to the work of writing program administrators, validation tools are also key to determining the ethics and rigor of studies in journals, such as those published in Research in the Teaching of English. This journal is in its 50th volume year and it has always worked from the presumption that English is the only language of the nation. It has built its reputation by upholding the highest standards for validity in data collection, selection, and analysis. Questions relating to the validity of evidence presented in RTE articles permeate the reviews authors receive. With a 5% acceptance rate, 95% of the submitted articles go unpublished each year, in part because article writers fail to demonstrate valid research designs structured in ways to gather, select, analyze, and interpret sufficient evidence for what the researcher claims to be investigating. Validity is at the heart of everyday acts of institutional assessments of writing and writing assessment research quality, and as such, the concept of validity and method of making valid arguments form the transparent instrumental logic of academe. Validity is a naturalized concept and invisible instrument of rigor that totalizes the realities of students and researchers. Its taken-for-granted ubiquity makes it a particularly interesting concept and tool to look at through a decolonial lens.
Validity has its roots from “French valide (Old French valide, Spanish valido, Italian valido, Portuguese valido) or Latin validus strong, powerful, effective” (OED). During the 16th century, validity developed as a concept that totalized the Western imperialist reality. Validity identified what counted as authority in law and perfection in the church; as what became valued in well-founded arguments, proofs and warrants; and even as a person could be said to be ‘valid’ if s/he was in good, sound, and robust health—or if not, the person was said to be an ‘invalid’ (OED). It is no coincidence that the word “validity” took hold in these languages at a time when the French, Spanish, and Portuguese empires were busy creating and codifying the colonial difference. Validity is an imperial concept in the first place. It was developed to justify what counts as claim, evidence, and warrant, in support of the enunciation of empire in all realms of legal, moral, intellectual, and physical being.
Validity indicates the social and epistemic hierarchies of knowledge created as part of the colonial difference. What is deemed to be valid in arguments and therefore reliably consistent in its measures (Slomp and Fuite 2005, pp. 191-193) needs the necessary other of the invalid and unreliable to legitimize themselves. To pursue Slomp and Fuite’s (2005) metaphor, validity and reliability are two horns of the bull of knowledge making and testing. Scholars and writing assessment specialists have for the last 50 years been debating “which of the two horns of the charging bull they should stand before” (p. 191), validity on the left and reliability on the right. I’m asking a different set of questions all together: Why this bullfight? Who is sitting in this arena and benefiting from the fight they may have helped organize? And why is this inhumane event taking place? In other words, I am taking up both the technical and the hermeneutic views of stability and arguing that we need to understand the imperial legacies of both if we are to find decolonial means of, in this case, writing assessment.
These questions extend a trajectory of scholarship that questions validity (and reliability), already established in the important ethical questions raised by Moss (1994 & 1998) and Wiggins (1994) when they asked if reliability or validity should be of greater importance to writing assessment specialists. To their credit, these scholars did allow for fairness to be brought into the arena to complicate the relationships between reliability and validity. Moss (2004) continued to take up the need to see validity and reliability as related and mutually sustaining concepts that are part of a particular type of thought. Notwithstanding Mislevy’s (2004) more inclusive definition and methods for determining reliability, Moss (2004) argued reliability and validity stem from particular understandings of what counts as evidence and what types of inferences might be drawn from this evidence: “…whenever we reason from evidence to inference, we engage in particular practices, that can be located within particular theories of ‘credibility.’ These theories and the practices through which they are instantiated do indeed constrain what count as evidence and what kinds of inferences can be supported” (Moss, 2004, p. 246).
The concept of validity created the colonial difference as a tool, which was used to identify and exclude (thereby instrumentally manage) all forms of evidence that it itself had not identified as sufficiently indicative of the claim it wanted to make about knowledge, land, governance rights, morality, and health. “The logic of Western imperial epistemology consists in a meta-discourse that validates itself by disqualifying the difference” (Tlostanova & Mignolo, 2012, p. 198). Validity is on the one hand instrumental tool, which was established to manage peoples, knowledges, lands, governments, and institutions, and on the other hand, a meta-discourse which reified the social, linguistic, and epistemological hierarchies that made it possible, hence further securing its own position of authority to identify what counts as valid. Think of Columbus planting his flag in the Caribbean and Americas claiming all he sees for the king of Spain. The flag, symbol of validity of the ownership claim of the crown, simultaneously provides claim, evidence, and warrant as one and at once. Said another way, validity as a self-identified knowledge points to its own utterance as proof of the claim for what counts as ‘true’ evidence and, in doing so, it always already creates the social hierarchy that places itself at the top—differencing, denigrating, dismissing, disrespecting, devaluing all other forms of law, religion, knowledge, and being as it does. Validity worked as a meta-discourse and tool that brought itself into being by dismissing that which it identified as its necessary other. The ubiquity with which it’s used to categorize, rank, and track beings into social positions, institutional roles, and professional grades traces its legacy to Great Chain of Being hierarchical thought, which also came into fashion in the Renaissance. Validity helped to establish various knowledges, languages, peoples, and beings as standing in greater or lesser positions within the social hierarchy that was the Great Chain of Being, which is still present today in modern society.
Fast forward several hundred years to constructs of validity undergirding modern institutions. As Elliot (2015) observed, the notion of validity and its methods of justification serve as foundations to most modern institutions (e.g. the military, family, government, K-U education, hospitals, businesses); that is, any institution with hierarchically arranged organizational charts (p. 669). “Both the concept of validity and the arguments used to construct its presence,” have adapted “so well [across institutions] that we may wonder if they are not merely artifacts of the system but, rather, the System itself” (Elliot, 2015, p. 669). The concept of validity served as a tenet of the imperialist thinking and doings that fueled the industrial revolution and progressive era, thinking and doings that helped to dismantle the sovereignty of American Indian nations through the Dawes Allotment Act, Westward Expansion, and boomer and settler colonialism. The concept of validity worked as it always had, but this version was even more imbued with the “drives for technological innovation and economic efficiency … realized through [the] creation of systems” and formulated in a “particular genre of argument” popularized by Stephen Toulmin’s rhetoric that masks its own imperial workings by nodding superficially to contingencies (Elliot, 2015, p. 669). In short, validity, is both content and principle of imperial thought, both an ‘artifact of the system’ and a primary tenet of ‘the System itself’— it’s both an instrument of modernity and a structuring principle behind it. This leads us to another moment of epistemic delinking: how has the remarkable adaptability of validity as tenet and instrument tried to account for the ways in which it perpetuates social hierarchies and injustices?
To her credit, Moss explored the idea of validity as both a hermeneutic and technical one, in which multiple types and kinds of evidence gathering and interpretations are necessary to ensure the data needs and uses of educational professionals are met (Moss, 2013, p. 95). Studies of the uses of test data and interpretation point to the need for a broader focus to support local users, she argued, as they attempt to understand questions or problems that evidence can be used to address (Moss, 2013, p. 96). Building from this line of thought, the articles in this special issue take up issues of social (in)equity and fairness in assessment by posing a theory of validity based on deliberative action.
In doing so, these articles provide rigorous and compelling alternatives to imperialist tenets of validity by creating a theoretical frame for fairness under which the ideas of reliability and validity can be understood as mutually sustaining, contextualized, and attendant to social consequences. Within this ethical assessment framework, reliability and validity are taken into account throughout the entire process of designing assessment constructs. Norbert Elliot’s article leads the issue by providing a much-needed philosophy of fairness based upon principled action, in part through the creation of variable models, disaggregated by groups (§3.2.3). Elliot begins the analysis with a necessary overview of measurement traditions (§2.1) to show how the traditions of measurement may offer benefits for a theory of writing assessment. Although these traditions have not necessarily been based on theories of ethics of fairness, communal interaction, localism, and identification of least advantaged, they can serve as part of an ethics striving for social justice through methods that ensure fairness (see §3.1). Elliot moves on to bound the theory of fairness he’s creating, paying particular attention to requirements of principled action (§3.2.4) and rejecting dualism (§3.2.5), before exploring its order (§3.3) and foundations (§3.4). Elliot’s thought experiment in which he applies his theory suggests a powerful heuristic for understanding and integrating concerns of fairness in approaches to writing assessment. Importantly, the conclusions drawn from his thought experiment reveal a construct (§4.4.3), potentially capable of stemming from and creating a continuum of care for all involved in the design and implementation of the assessment construct. While much of the theoretical perspectives Elliot draws upon stem from Western philosophers, a decolonial approach to validity would not throw this baby out with the bath water of imperialism. Rather, a decolonial approach would recognize the possibility that fairness, equity, and attention to pluriversal knowledges might be possible from and within multiple epistemological standpoints, without seeking to privilege one epistemology over another. In the theory Elliot forwards, validity and reliability are wrested from their primacy over fairness and their exclusion of social consequences and instead become nested within these exigencies for change.
David Slomp’s article carries forward this philosophy of fairness by providing critique of an existing high school English writing test 30-1 in Alberta, using Kane’s framework (§4) to analyze the inferences made by designers of the assessment constructs, revealing a complex of problems associated with the validity of the test. He goes on to present a fresh alternative to developing assessments through a model of ethical assessment research in which he and his team of researchers collaborate with teachers and students at every stage of designing assessments (§5 and 6). A promising model for designing assessments emerges from the ashes of his critique of an actual assessment construct. One of the most powerful findings in Elliot and Slomp’s articles: The onus to change how, why, for whom, and in what ways assessments fail to account for their social consequences is on assessment designers rather than teachers and students.
Mya Poe and John Cogan’s article provides a powerful alternative to assessment policies and practices that would rather place blame for failure on those giving and taking the test. Exploring Supreme Court cases related to the Civil Rights Act of 1964, they outline the finer points of the disparate impact approach, which justices use to determine intentional and unintentional forms of discrimination unfolding in the everyday decision making processes of institutional representatives. The disparate impact approach, they argue, provides both concept and method for assessment researchers because it shifts the burden of having to prove tests have harmful consequences away from students and teachers and places that burden onto the shoulders of assessment makers and researchers.
Taken together, many of the ideas in this issue work from within the borders of assessment research, attempting to see the ways in which social hierarchies and injustices are perpetuated by current forms of assessment research, policy, and practice, while proposing compelling alternatives, methodologies, and heuristics for ensuring more equitable outcomes of assessment initiatives.
Validity as a tenet is used to claim, gather, and justify results with so many performance and survey tools, it has now more than ever been used to routinize inequities as naturalized parts of systems of educational access, predictions of success in school or on the job, psychological and intelligence measures, and as a foundation for knowledge creation in research studies. The symptoms of colonial thought have been identified, but remain largely untreatable to those who create educational assessments and psychological measurements. To their credit the authors of the 1954 version of the Standards for Education and Psychological Testing (APA) attempted to correct for manifestations and reproductions of social inequity by developing the concept of “construct validity.” The notion of construct validity was created in an effort to validate the theory underlying the assessment content, without recognizing that it would always already provide for its own validation. In other words, as Slomp has suggested, validation has a built in confirmatory bias to it, so that it’s always looking for evidence that supports the claims it wants to make in part because it cannot, as an instrument, just yet see outside of its own content and imperialist tenets. And so unfolds many efforts since to correct for bias through more open and transparent means of showing how an assessment maker does his/her work using “construct validity.” It’s as though transparency can correct for bias when it basically and simply says to those created by the colonial difference: “Let me show you how the making of this assessment will exclude you using our exclusive and exclusionary process of building this test.”
Authors of the Standards again attempted to account for the bias implicit in any research by introducing the idea that interpreting results was yet another place where bias might manifest: “The cultural shift is evident in the 1974 edition of the Standards”, in which attention was paid to the concept of fairness in terms of how test results were to be interpreted and used (AERA, APA, & NCME, 1974, p. 676). Developing sharper, more precise instruments to measure potential test bias, “the authors of the Standards acknowledged, in a powerfully empirical way, the need to examine the meaning of scores across different groups of examinees” (AERA, APA, & NCME, 1974, p. 676). In the wake of the Civil Rights movement, the authors of the standards were gradually getting clued in as to how testing might be contributing to systems of social stratification, or colonial matrices of power (Tlostanova and Mignolo, 2012). Their answer: create more empirically precise forms of validity measures to find the bias that was baked into the tenet and tool of validity as part of the creation of the imperial difference in the first place. The regression analyses created in this time to account for test bias in effect show, even more precisely, how validity works to replicate the social structures already in place, with two noteworthy exceptions being proposed early on by Cleary (1968) and more recently by Elliot (2011 and in §3.2.4). Subsequent issues of the Standards, Elliot found, admitted that a complete separation of the scientific from social issues and constructions of value was not possible. This might allow the Standards authors a vantage point from which to see validation as justifying all stages in the process of test design, testing of construct validity, and interpretation of results as it strives to increased awareness of fairness.
Yet, efforts at fairness risk making only token adjustments to the instruments of validity, as Slomp, Elliot, and Poe and Cogan have demonstrated in this special issue . For instance, fairness considerations include measures taken to prevent distortions between fact and interpretation in construct design, implementation, and interpretation of results. In the most recent version of the Standards (2014), fairness manifests itself in the very first chapter dedicated to delimiting ways to address issues of validity. Sections include steps to account for issues of fairness related to evidence obtained when considering the content of test questions, as this content relates to the construct it was intended to measure; a section on the evidence gathered by responses to the processes of testing; and a section on the evidence on the internal structure and consequences of testing (pps. 15-20). With each of these sections, validity as tool and premise of colonial difference attempts to justify itself within systems of difference it has had a hand in creating and preserving through the mask of neutrality and scientific objectivity.
Another example: The Standards authors attempted to address problems in the content of tests and interpretations of results, to control for “the extent to which construct underrepresentation or construct-irrelevance may give unfair advantage or disadvantage to one or more subgroups of test takers” (AERA, APA, & NCME, 2014, p. 15). So while questions on tests may use white, middle class, school-based content or phrasing in their questions, this attempt to address questions of validity by paying attention to the social with the scientific, still falls short of the fairness mark. This token nod to the social uses even more precise and narrowly scientific measures to address the problem of inequity and cultural difference that it itself has helped to create and perpetuate, while maintaining the tenet of imperial thought that is foundationally structuring all questions of validity in the first place.
Fairness can address content of particular questions, but it does little to adjust the overall ways in which validity measures themselves, from the start, are based on the colonial difference that they help to create and maintain. The problem has been, as Elliot’s article points out so well in §2.3.2, there is no latent variable of writing. He shows this to be a Platonic fiction, which only serves to reify existing social hierarchies. However, because of a hegemonic view of variable modeling—another artifact of imperialism—scores are not always disaggregated, and claims and inferences are made for a generality that does not exist. Instead, as Elliot argues, there are many, equally important variable models, that can be learned about through analysis of group differences and task analysis (§3.2.3).
In this instance, constructs will always be unrelated to the knowledges and language practices of the peoples made different by the construct and validity measures in the first place:
For this reason, the decolonial epistemic shift proposes to change the rules of the game—not just the content—and the reason for which knowledge is produced: Decolonization, instead of working toward the accumulation of knowledge and imperial management, works toward the empowerment and liberation of different layers (racial, sexual, gender, class, linguistic, epistemic, religious, etc.) from oppression and toward undermining the assumption on which imperial power is naturalized, enacted and corrupted. (Tlostanova and Mignolo, 2012, p. 63).
I’ve been arguing that validity is one such assumption by and through which imperial power has been realized in the everyday acts of assessment creation, testing, and interpretation and in knowledge making about assessment. As both tenet and instrument of imperialism, validity is a key player in the sanctioning and building of knowledge and in the maintenance of social hierarchies that have their roots in Great Chain of Being thinking. My goal in delinking from understandings of validity has been to work toward the “empowerment and liberation” of different groups from these socially arranging hierarchies of peoples, knowledges, dispositions to learning, and languages, especially as they are ranked, classified, managed, and tracked from before day 1 of their (educational) lives through assessments. This brings me to the last section of this essay, in which I take up a consideration of how the tenet and instrument of validity might be decolonized in potentially empowering ways within a theory and practice of fairness proposed by the articles this special issue.
So what might the Standards have offered with respect to both the conduction and the evaluation of writing assessment research? I opened article with a brief story of research and work that preceded this essay. I did so to return to a lesson I hope will be taken from these works to draw a different trajectory from the historical past outlined here: Individuals in communities and Nations have knowledges, histories, practices, values, and languages that can well shape future assessment creations and knowledge making practices. And they can do so when those in the positions to create assessments and research assessment do so with an understanding of pluriversality of languages and knowledges.
To begin, more scholars of writing assessment can begin to dwell in the borders created by the colonial difference, for “the main problem of the twenty-first century is not just crossing borders but dwelling in the borders” (Tlostanova and Mignolo, 2012, p. 72). I applaud the Standards attempts to use border-crossing methods to account for fairness and adjust tests and research methods accordingly. It seems there’s at least the acknowledgement that power and hierarchies are being remade, the hope that those who are most served by the hierarchies in place are beginning to recognize at least the ways in which they too are dehumanized by systems of oppression. Recent research into writing assessments suggests that validity threats are mounting (Dryer, 2016). Recognizing there’s a problem with the content of the system and its perpetual replication of inequity is a start. Yet, this recognition has led to small steps and piecemeal efforts to make change as described in these pages, including the shortcomings outlined in assessment theory (Elliot), construct design (Slomp), and in the ongoing methodological shortcomings of accounting for the social consequences of assessment research and design (Poe and Cogan). Rather than only crossing borders with greater and more robust technocratic instruments to test for validity of constructs, changing the constructs themselves can begin with dwelling in the borders. “That is, dwelling in the borders means rewriting geographic frontiers, imperial/colonial subjectivities, and territorial epistemologies” (Tlostanova and Mignolo, 2012, p. 72) — precisely the types and kinds of work the articles in this issue, as well as the final essay of this issue, attempt to do by presenting Elements for an Ethical Theory for Writing Assessment. The new definition of fairness offered in these pages, with the provision that resources be allocated to better understand the knowledges and means of expression of the least advantaged, offers a starting place to begin imagining and enacting a pluriversal possibility in assessment. When dwelling in the borders, all possible knowledges, languages, ways, and values become apparent, each with their own unique means for judging the veracity of statements, solving wicked problems, identifying quality use of sign tools (e.g. language, music, design, performance, video), and internally organizing structures and economies. This will be a world where many possible worlds coexist, but cannot be predicated upon a universal design that images one functioning world for everyone, nor a global world that insists on imperial global designs.
Dwelling in the borders helps to create a world in which many worlds coexist equally, in and on their own terms, predicated upon “pluriversality as a universal project. Critical border thinking and the decolonial shift are one road to that possible future” (Tlostanova and Mignolo, 2012, p. 79). Dwelling in the borders begins with the knowledge, languages, histories, and practices understood and valued by the people who live these realities. It would see validity evidence tools not as a way to maintain, protect, conform to, confirm, and authorize the current systems of assessment and knowledge making, but rather as a way to better understand difference in and on its own terms. Rather than making its own experience into a universal one, the baseline against which all Others are tested and their knowledges and languages are deemed deficit to, validity measures would seek to identify understandings in and on the terms of the peoples who experience them.
Broad’s article in this issue helps to identify the economic and social structural blindnesses of the current testing industry (see §4.1 & 4.2). Without a pluriversal view, scholars may also be blind to the theories and methods that ground knowledge making efforts. Dwelling in the borders, authors of the Standards, for instance, would seek to continue the important work of identifying, questioning, and challenging biases to strive for and help create a system that sustains and helps diverse knowledges persevere. They might seek to understand how various peoples themselves understand what counts as valid forms of expression, knowledge making, and evidence of this. The would take their cues from the teacher research on assessment construct design (Slomp), burden shifting analyses (Poe and Cogan), and dynamic criteria mapping (Broad 2003, Broad et al., 2009). They would see validity and reliability as mutually sustaining practices within an overarching theory of fairness as part of an ongoing continuum of care, as Elliot has so eloquently proposed. Dwelling with/in the borders of assessment research, these scholars have begun to create methods and theories for accounting for pluriversal knowledges and languages.
This good work, though not necessarily calling itself decolonial in any respect, is also happening from within the system of social hierarchies created by the imperialist notion of validity. Recently, a special issue of Research in the Teaching of English, guest edited by Mya Poe, created what we might see as pluriversal understandings of validities to address the limitations of current standards for validity when working with diverse populations. Slomp, Corrigan, and Sugimoto (2014) offered a framework for understanding the consequences a large-scale writing assessment might have on diverse populations in an effort to enable the creation of more integrated and integrative validity arguments. Inoue and Poe (2012) and Poe (2013) offered compelling research and arguments for innovations, such as racial validity measures, which seek to make writing assessments and their interpretations more equitable for all students. Inoue’s (2014) redefinition of nature and production of failure in the writing classroom dwells in the borders by offering a conception of “productive failure” that seeks to assess the ways in which failure is necessary for growth in student writing. They’re not throwing the validity baby out with the assessment research bath water, nor would I argue such is a necessary measure. Most peoples, it seems to me, have notions of what counts as a persuasive claim, who is authorized to speak in particular circumstances and how, and what are particularly well articulated uses of language, performance, art, media, and genres. The problem becomes when what counts as valid is always judged against a baseline that privileges one group of people’s knowledge and forms of expression to the necessary exclusion of others.
To encourage efforts to decolonize what counts as research in the teaching of English (Cushman, Juzwik, Macaluso, and Milu, 2015), the RTE editorial team has sought to publish several path-breaking articles that innovate the typical style of research genre and method recognized to create valid arguments from within the pages of the research journal in an effort to demonstrate in broader terms the ways knowledge might be made with, in and on participants’ terms (Orzaluk, 2015; San Pedro, 2016). Todd DeStigter (2015) took up precisely why it is that argument itself as genre and epistemic approach to valid thought has shaped the types of writing typically assigned and assessed in schools and universities. Though these authors may not characterize themselves as dwelling in the borders through epistemic delinking of key forms, methods, and tenets decolonize imperial knowledges, their work demonstrates the ways that those within the system of knowledge production have taken steps toward epistemic delinking.
My point is that you don’t have to be a person of difference to dwell in borders, to think of ways in which social equity and pluriversal understandings can be achieved in everyday knowledge work of assessment design and research on assessment. The important thing is to actively seeking out pluriversal (rather than universal) understandings, multiple and varied (rather than singular and narrow) ways of expression, integrated (rather than siloed) exercises in validity and reliability, whole and active (rather than atomized and static) language uses in an effort to name and respect a range of ontological, axiological, and epistemological perspectives.
American Psychological Association. (1954). Technical recommendations for psychological tests and diagnostic techniques (Vol. 51, No. 2). Washington, DC: American Psychological Association.
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (1974). Standards for educational and psychological tests (3rd ed.). Washington, DC: American Psychological Association.
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Broad, B. (2003). What we really value: Beyond rubrics in teaching and assessing writing. Logan, UT: Utah State University Press.
Broad, B. (2012). Strategies and passions in qualitative research. In L. Nickoson & M. Sheridan (Eds.), Writing studies research in practice: Methods and methodologies (pp. 197–209). Carbondale, IL: Southern Illinois University Press.
Cleary, T. Anne. (1968). Test bias: Prediction of grades of Negro and White students in integrated colleges. Journal of Educational Measurement, 5, 115–124.
Cushman, E. (2012). Learning from the Cherokee syllabary: A rhetorical approach to media research and teaching. JAC: A Journal of Rhetoric, Culture, and Politics, 32, 541–565.
Cushman, E. (2012). The Cherokee syllabary: Writing the people’s perseverance (Vol. 56). Norman, OK: University of Oklahoma Press.
Cushman, E. (2013). Wampum, Sequoyan, and story: Decolonizing the digital archive. College English, 76(2), 115–135.
Dryer, D. (2016). Appraising translingualism. College English, 78(3), 274-283.
DeStigter, T. (2015). On the ascendance of argument: A critique of the assumptions of academe’s dominant form. Research in the Teaching of English, 50(1), 11–34.
Elliot, N., Deess, P., Rudniy, A., & Joshi, K. (2012). Placement of students into first-year writing courses. Research in the Teaching of English, 46(3), 285–313.
Elliot, N. (2015). Validation: The pursuit. [Review of Standards for educational and psychological testing, by American Educational Research Association, American Psychological Association, and National Council on Measurement in Education]. College Composition and Communication, 66, 668–685.
Haswell, R. H. (1998). Multiple inquiry in the validation of writing tests. Assessing Writing, 5(1), 89–109.
Inoue, A. B., & Poe, M. (2012). Race and writing assessment. Studies in composition and rhetoric, Vol. 7. New York, NY: Peter Lang.
Inoue, A. B. (2014). Theorizing failure in US writing assessments. Research in the Teaching of English, 48(3), 330–352.
Mislevy, R. J. (2004). Can there be reliability without ‘reliability?’. Journal of Educational and Behavioral Statistics, 29(2), 241–244.
Moss, P. A. (2013). Validity in action: Lessons from studies of data use. Journal of Educational Measurement, 50(1), 91–98.
Moss, P. A. (2004). The Meaning and consequences of ‘reliability’. Journal of Educational and Behavioral Statistics, 29(2), 245–249.
Moss, P. A. (1998). Testing the test of the test: A response to “multiple inquiry in the validation of writing tests”. Assessing writing, 5(1), 111–122.
Moss, P. A. (1994). Can there be validity without reliability? Educational Researcher, 23(2), 5–12.
Orzulak, M. J. M. (2015). Disinviting deficit ideologies: Beyond “That's Standard,” “That's racist,” and “That’s your mother tongue.” Research in the Teaching of English, 50(2), 176–198.
Poe, M., Elliot, N., Cogan Jr., J. A., & Nurudeen Jr., T. G. (2014). The legal and the local: Using disparate impact analysis to understand the consequences of writing assessment. College Composition and Communication, 65(4), 588–611.
Poe, M. (2013). Making digital writing assessment fair for diverse writers. Retrieved from http://ssrn.com/abstract=2444400
San Pedro, T. (2015). Silence as shields: Agency and resistances among Native American students in the urban Southwest. Research in the Teaching of English, 50(2), 132–153.
Slomp, D. H., & Fuite, J. (2004). Following Phaedrus: Alternate choices in surmounting the reliability/validity dilemma. Assessing Writing, 9(3), 190–207.
Slomp, D. H., Corrigan, J. A., & Sugimoto, T. (2014). A framework for using consequential validity evidence in evaluating large-scale writing assessments: A Canadian study. Research in the Teaching of English, 48(3), 276–302.
Tlostanova, M. V., & Mignolo, W. D. (2012). Learning to unlearn: Decolonial reflections from Eurasia and the Americas. Columbus, OH: The Ohio State University Press.
Valid. (2015). In Oxford English Dictionary. Retrieved from http://www.oed.com/view/Entry/221190?redirectedFrom=valid#eid
 Thanks to Norbert Elliot for helping me sharpen this phrase. Many thanks to both David Slomp and Norbert Elliot for their insights, lessons, and thoughtful comments in developing this article.
Ellen Cushman is Dean’s Professor of Civic Sustainability at Northeastern University, Boston, Massachusetts, and may be contacted at firstname.lastname@example.org.