Volume 13, Issue 1: 2020

Understanding Proficiency: Analyzing the Characteristics of Secondary Students’ On-Demand Analytical Essay Writing

by Vicky Chen, University of California, Irvine; Carol B. Olson, University of California, Irvine; Huy Quoc Chung, University of California, Irvine

This study investigated the different characteristics of not-pass (n = 174), adequate-pass (n = 173), and strong-pass (n = 114) text-based, analytical essays written by middle and high school students. Essays were drawn from the 2015-2016 Pathway writing and reading intervention pretests and posttests. Results revealed the use of relevant summary was an important difference between not-pass and adequate-pass essays where significantly more adequate-pass essays used summary in a purposeful rather than general way. In contrast, major characteristics that set apart strong-pass essays from adequate-pass essays involved providing analysis and including a clear conclusion or end. Factors that affected these characteristics such as whether the writer made claims and comments about the text are discussed, and some instructional strategies are suggested.

Keywords: writing proficiency, writing instruction, adolescent literacy, text-based analytical writing, on-demand writing assessment


Introduction

Improving the writing skills of our students and the teaching of writing is a “vital component of students’ literacy achievement” (Graham et al., 2016, p. 1) and a key to college and career readiness. The 2010 Common Core State Standards (or CCSS), which lay out the educational expectations for students graduating from U.S. high schools, include rigorous standards for writing and communication. To prepare students for higher education and competitive employment in the 21st century, the CCSS requires that students be able to both analyze and produce a variety of texts for different purposes. In particular, the CCSS places heavy emphasis on text-based, analytical writing where students are expected to be able to analyze both the structure and content of a text—to “determine central ideas or themes,” “summarize the key supporting details,” “make logical inferences,” and “cite specific textual evidence when writing... to support conclusions drawn from the text” (CCSS Initiative, 2010, p. 35). Such writing requires students to think critically about what they read, make inferences, and construct their own opinions in a coherent and reasonable fashion—aspects of writing that students are often asked to perform as they move into higher education.

For many students, these expectations present major challenges. According to the National Assessment of Educational Progress (NAEP, 2011), only 27% of all 12th graders and 1% of English learners (ELs) scored at or above the level of proficient in writing (U.S. Department of Education, National Center for Education Statistics, 2012). This is cause for concern because the ability to write well has very real repercussions in both school and the workplace (National Commission on Writing, 2004). Most postsecondary institutions include some form of writing assessment in their admissions and/or course placement processes, and the more remedial classes students have to take, the less likely they are to persist and attain a degree (Dominick, Stevens, & Smith, 2007).

Before considering the specific challenges that student writers face, it is important to understand what we mean when we say that a writer is proficient. To do this, we examined the scoring criteria used by two major writing assessments from the last decade: the National Assessment of Educational Progress (NAEP, 2011) writing assessment and the writing portion of the Smarter Balanced Assessment Consortium (SBAC, n.d.) developed to address the CCSS.

Defining Writing Proficiency in the US

The NAEP is the largest nationally representative assessment of what American students in grades 4-12 know and are able to do academically (NAEP, 2017). It is conducted periodically in a wide variety of subjects and is known for its robust construct validity (Applebee, 2007; Wenglinsky, 2005). The framework used for the writing portion of the NAEP starting in 2011, developed over the course of several months by more than 500 experienced educators and experts in the fields of assessment and writing, aligns with the learning standards of most states regarding effective written communications (National Assessment Governing Board [NAGB], 2010). At the secondary level, the writing assessment, administered to 8th and 12th grade students, involves three different types of writing tasks that reflect a range of communicative purposes relevant to writing in both school and the workplace—persuading, explaining, or sharing an experience. Students are given two randomly selected prompts and have 30 minutes to complete each task (NAEP, 2016).

According to the item maps for the NAEP writing task, a piece of writing that demonstrates proficiency responds to the writing prompt in a manner that is clear and accomplishes its communicative purpose. The text should be coherent and well-structured. Most ideas should be developed logically with appropriate connections, and the voice of the text as well as its details and examples should be relevant to and support the main idea. Although errors may be present, these errors cannot interfere with the reader’s understanding of the text (NAEP, 2011).

Clarity, relevance, focus, and completeness are also reflected in the rubric used to score the writing portion of the Smarter Balanced assessment, designed to align with the Common Core and currently used by many states including California and Oregon. An “adequate” piece of informational-explanatory writing, according to this rubric, is clear with some evidence of structure. Some ideas may be loosely connected, but they should be clear and appropriate to the task. There should be adequate use of references and a progression of ideas from an introduction to a conclusion that gives an overall sense of completeness to the text. The response as a whole should be mostly sustained and generally focused.

Given the low percentage of students writing at or above the level of proficient, as well as the impact of writing assessment scores on students’ educational success, it is important to understand both what kind of writing students are producing and how proficiency is or is not reflected in their work. For this study then, we drew from conceptions of proficiency as defined by major writing assessments like the NAEP and SBAC, as well as the CCSS College and Career Ready Anchor Standards for Reading and Writing, to design an analytical writing rubric and administer a pre-post on-demand writing assessment in two large urban school districts with over 5,000 students in grades 7-12. We then examined a subsample of these essays scored by a site of the National Writing Project (NWP) to examine two questions:

  • What are some of the characteristics that might distinguish essays that are considered proficient from essays that are not?
  • How might essays that are just proficient differ from essays that are considered more advanced?

By gaining a deeper understanding of the characteristics that shape writing proficiency, educators will be more prepared to help students build a solid foundation for the writing skills they will need in college and the workplace. Teachers will be better equipped to focus their time and energy on the aspects of writing that are most beneficial to their students.

Theoretical Framework

The current study is informed by research on the composing processes of novice and experienced writers that began within the field of cognitive theory. This section will provide an overview of how researchers who have contributed to the cognitive theory of writing have conceptualized the writing process and some of the differences research has shown between how novice and experienced writers compose. Because most of these studies have compared published professionals such as university professors with college students who needed additional writing assistance, we will then discuss two different models of writing—knowledge telling and knowledge transformation—that expanded upon this research by focusing exclusively on students. Scardamalia and Bereiter (1987) proposed that these two different approaches to writing indicate where a student is in his or her development as a writer. Finally, we will discuss research that elaborates upon Scardamalia and Bereiter’s work by demonstrating that these approaches to writing should be seen as a continuum rather than only two distinct categories, and that these differences in development are reflected by varying patterns in expository texts written by students in Grades 1 through 9 (Hayes, 2011). The current study adds to this research on varying forms of knowledge telling and knowledge transformation in the text-based analytical writing of middle and high school students, identifying relevant text features and linking patterns with current definitions of writing proficiency.  

   In the field of cognitive theory, writing is primarily understood as a number of distinctive thinking processes writers employ throughout the act of composing (Flower & Hayes, 1981; Hayes, 1996). In this model, writing is highly recursive and driven by goals that writers set for themselves—goals that often change and evolve over the course of the writing process. Different composing processes can affect not only how a text is generated but also a text’s visible characteristics, such as the way ideas are organized and how the writer’s reasoning is presented (Hayes, 2011).

    By identifying and focusing on cognitive processes, or thinking skills, such as idea generation, goal-setting, planning, and monitoring, researchers are able to compare the strategies used by novice and experienced writers as they carry out a writing task (Flower & Hayes, 1981). This, in turn, helps researchers and teachers develop strategies that can support student writers as they learn to think and write as experts, an approach to writing instruction that has been shown to be especially effective for struggling writers and ELs (Olson, Matuchniak, Chung, Stumpf, & Farkas, 2017).

    Research drawing on cognitive theory has highlighted important differences between the writing and revising processes of novice and experienced writers. Flower and Hayes (1980) found, for example, that novice and experienced writers differ in the way they conceptualize a writing task. Experienced writers construct complex images of the situation or context of their writing and the audiences that might read their texts, fleshing these images out with their own experiences and understanding of writing conventions. Novice writers, in contrast, tend to have only sketchy images of the situation, and their representations of audience are often stereotypical.

     These differences in novice and expert writing processes are driven, in large part, by the goals writers set for themselves. For example, while expert writers develop extensive plans to affect their audiences over the course of their writing, novice writers tend to be tied to the topic (Berkenkotter, 1981; Flower & Hayes, 1980). In other words, skilled writers are solving a different rhetorical problem, one that involves creating reader-based prose that take readers into consideration as opposed to writer-based prose where writers are talking only to themselves (Flower, 1981).

    Novice and experienced writers also differ in their revision strategies. For instance, in a series of studies comparing inexperienced undergraduate writers and published professionals, Sommers (1980) found by analyzing think-aloud protocols and multiple drafts written by each participant that professional writers made many more structural and meaning-based changes during the writing process whereas the changes undergraduate novice writers made tended to be on a more surface, lexical level. In addition, while the revision strategies of novice writers were driven by an attempt to make their writing conform to a predefined meaning, experienced writers were more concerned with making their own meaning through engagement with their texts. Because of this, the revisions made by experienced writers are aimed at refining the meaning—and not just appearance—of their writing. This reconstructing of meaning is an important element of what Scardamalia and Bereiter (1987) call knowledge transformation.

     Rather than focusing on novices and experts, Scardamalia and Bereiter (1987) compared the composing processes of elementary school and college students—students at different levels of their development as writers, but who would not be considered masters of the written word. All students included in the study were considered good writers by their teachers, but the two groups differed in age and corresponding amounts of experience. The researchers proposed two models—or methods—of composing, called knowledge telling and knowledge transformation, that characterize the writing of these immature and mature writers. The key difference in these two models is in how knowledge is brought into the writing process.

   In knowledge telling, students simply recount information as they write in a relatively straightforward fashion. “Content retrieved from memory either is passed directly into the composition or else is rejected and other content is retrieved” (Scardamalia & Bereiter, 1987, p. 148) in an item-by-item manner without further elaboration to meet writing sub-goals. Writing is about getting preexisting ideas on paper and follows the writer’s train of thought rather than considering a reader’s needs. This process of retrieving and relaying information tends to lead to writer-based prose with only basic connections between ideas (Flower, 1981).

   In knowledge transformation, which was more common among mature writers, the presentation of and connections between ideas are more complex. Writing becomes about not only what to say, but also how to say it. Writers consider what a reader might think, and such goals as persuading come into play as information is reorganized, compared, analyzed, and transformed. This second model is important to creating reader-based prose (Flower, 1981), and to help students write better, teachers need to help them move from a knowledge telling model of composing to one of knowledge transformation (Scardamalia & Bereiter, 1987).

     Although these two models are useful, Hayes (2011) argued they could be further refined and useful distinctions could be made within these two broad categories. He elaborates upon the idea of knowledge telling and identifies three overarching patterns in the expository texts produced by elementary and middle school students at different levels in their development as writers. Hayes suggested that understanding these patterns—which demonstrate increasing levels of complexity and a shift from strictly linear to more recursive knowledge telling models—allows us to better assess where a student is in his or her development and focus on the specific strategies that would most help each student improve. In other words, Hayes demonstrated that broad categories like knowledge telling should be conceptualized as a continuum that becomes more complex as students move towards a knowledge transformation approach to writing.

   In the current study, we take a view similar to Hayes (2011) by looking for potential patterns in the not-pass, adequate-pass, and strong-pass essays written by middle and high school students in response to a text-based, analytical writing task—focusing on the inclusion or exclusion of specific textual elements such as summary, commentary, and analysis. These elements highlight different moves made by the writer that align with different forms of knowledge telling or knowledge transformation. For instance, making claims about the text is an aspect of knowledge transformation because it involves the writer drawing conclusions about the text’s meaning, whereas simply summarizing events from the text would align more closely with knowledge telling. We hypothesized that features associated with knowledge transforming would be more evident in essays that received passing scores whereas not-passing essays would conform with knowledge telling. We hoped to learn which of these elements, if any, were most strongly associated with essays that demonstrated writing proficiency.

Methods

Study Context

In this study, we examined student essays from the Pathway reading and writing intervention during the 2015-2016 school year in two large, urban school districts in Southern California with high percentages of ELs. Pathway is an intensive professional development program developed by an NWP site that takes a cognitive strategies approach to reading and writing instruction. Secondary school teachers learn to integrate strategy-based instruction into their classrooms to improve the interpretive reading and analytical writing of their students. In particular, the program aims to help mainstreamed ELs and Latinx students acquire the skills to help close achievement gaps between them and their native-English-speaking peers (Olson et al., 2017). The intervention is funded by grants from the Office of English Language Acquisition and Investing in Innovation.

Essays were written in response to two similar on-demand, text-based writing tasks. Both middle and high school students (7th-12th grade) were asked to read a piece of literary nonfiction, make a claim about the theme or lesson of the story, and support their claim with evidence from the text (see Appendix A). The assessment took place over two days. Students had one class period to read the text and plan for their writing, then used another class period on the following day to write their essays. Both districts served large proportions of Latinx students (District 1 = 65.8%, District 2 = 79.3%).

In “The Horned Toad” by Gerald Haslam (1995), the author recounts his childhood experiences with his Spanish-speaking great grandmother, who comes to live with him and his parents in California. Differences in their language and culture cause misunderstandings at first, but over the course of the story, their relationship evolves. Like “The Horned Toad,” Nick C. Vaca’s “Martin” (1967) also tells a story from the author’s childhood where a new person enters his life that also results in a misunderstanding. In Vaca’s case, this was a boy named Martin who has a large belly that Vaca assumes meant that he is rich and well fed—unlike them. His perception of Martin also evolves by the end of the narrative.

The two prompts were counter balanced. Half of the students were randomly assigned to write to each prompt at pretest, and then the prompts were switched at posttest. This means that all students wrote to both prompts. A t-test found no significant differences between the scores received by essays written to each of these two prompts (p = 0.4698), so neither prompt was significantly more difficult.

In the NWP scoring, all essays were scored by two trained readers based upon a 1-6 rubric scale similar to that used by the NAEP (see Appendix B for rubric sample), which has been shown to be a valid and reliable measure of student writing (Bang, 2013). Our rubric for scoring was based on those used to evaluate the essay portion of the California High School Exit Examination (California Department of Education, 2008a), the California STAR 7 Direct Writing Assessment (California Department of Education, 2008b), and the NAEP (ACT, Inc., 2007).

The score given to an essay by each reader was added together for a final score between 2 and 12. For example, if an essay received a 5 from both readers, its final score was 10. If an essay received two scores that differed by 2 or more points—for example, a 3 and a 5—it was scored by a third reader (i.e., if score A = 2, score B = 4, and score C = 3, then the final score of the essay was 6). Reliability of the scoring was assessed through double scoring 10% of the papers. Raters agreed within a single score point for 90% of the papers. Only essays that received the same score from both original readers were included in this study, and this includes both pretests and posttests that satisfy this criterion. The total sample consisted of 461 essays. Of these essays, 50% were from students who received the intervention, and 50% were from the control group.

Of the students who wrote the essays in this sample, 14.32% were ELs, 6.36% were designated initially fluent (IFEP), and 46.59% were re-designated English proficient (RFEP). In addition, 64.55% of the students were Latinx, 15.77% were Asian, 2.27% were Black, 7.27% were Native American, and 9.32% were White. And, 67.61% were on free or reduced-price lunch.

Coding and Analysis

Essays were coded as either having or lacking certain features, such as having a claim or not having a claim (see Table 1). Coding categories were developed based upon three things: (a) the rubric used to score the essays (see Appendix B); (b) the prompt students were given (see Appendix A); and (c) a preliminary study of 50 score-3 and 50 score-4 essays similar to the larger sample examined here, which helped to refine and validate the codes. To establish interrater reliability (α =.8333) a subsample of essays was shared with four graduate students and one university professor trained to code using this coding scheme.

In order to compare the characteristics of passing (˃ 4) and non-passing essays (˂ 3), we focused on essays that received either a 3 (marginal not-pass) or a 4 (adequate-pass) from both readers. There were 174 score-3 essays and 173 score-4 essays. A test of proportions was performed for each coding category to see which elements were statistically different between the two samples. Because commentary was coded as a percentage to examine the proportion of each essay comprising the writer’s thoughts rather than summary, a t-test was performed to compare the average amount of commentary included in not-pass versus adequate-pass essays. Then, a logistic regression was performed to compare the different textual elements to determine which categories had the greatest predictive association with whether essays received a 4 or a 3. Second, to examine potential differences between adequate-pass essays from essays that were strong, score-4 essays were compared with score-5 and -6 essays (114 total), and the same statistical tests were performed.

The coding categories were as follows:

Table 1

 Each category is described in detail below.

 

Clear Introduction and Claim

Having a clear introduction was defined as having an introduction that clearly informed the reader what the essay was going to discuss. Ideally, this would include a thesis or claim, but this did not have to be the case. Essays that began with abbreviated story outlines were also coded as having clear introductions. For example:

In the autobiography The Horned Toad by Gerald Haslam the young boy learns alot from his relationship with his great grandma. (6H_3F1041424)

This introduction clearly states what the essay is going to be about—a boy learning from his great grandmother. However, to be considered as having a claim, it would have had to state what it was he learned because we already know from the prompt that he learns something.

Introductions that included a claim tended to give the essay more direction. For example:

Judging someone by their outer appearance can lead to misunderstandings. In the short story, “Martin” by Nick C. Vaca, it shows how the narrator changes his thoughts on Martin. Martin was small, yet he had an enormous belly. The main character and his cousins envied Martin for having such a belly. They thought that he must have been eating very well. This led to them bullying Martin. Until later they realize Martin does not get to eat much, and that he got bullied for no reason. From this experience people can learn to not judge someone by their outer appearance. (8M_3H0832026)

In this excerpt, the writer gives the reader a clear context for the story and makes a claim about the overall message of the story based upon that context.

It was also possible for an essay to make a claim but not have a clear introduction. For instance:

Nick C. Vaca shows that there is more to a book than its cover and usual things aren't always what you think they are. (8M_3J1032504)

This opening paragraph makes a claim, but it provides no additional information or context to frame the rest of the essay.

Lastly, in order to be marked as having a claim, the essay had to make that claim before the end of the essay. Any claims made only at the end of an essay were not counted because their placement prevented the writer from explaining or supporting the claim.

Clear End

A clear end could be a paragraph where final conclusions were drawn, or it could be the end of a story summary. As long as the writer finished writing the essay and gave some sense of closure, it was considered as having a clear end. For example, the last paragraph in one student’s essay read:

In this narrative the narrator by the end loves, cares, and respects his great-grandmother for all the wisdom she has shown him. She passes away and just like the horned toad she must be buried in her own place. (8H_3Q1240325)

The claim made by this writer was that the great grandmother taught Haslam both not to judge a book by its cover and to value life and family. Although the student did not return to this claim in the conclusion, she gives her essay a sense of completeness by highlighting the narrator’s concluding feelings.

General and Relevant Summary

By summarizing an event or the main ideas of a text, a writer provides context that allows readers unfamiliar with the subject to understand the writer’s claims and comments. However, it is also a tactic that many student writers fall back on when they are attempting to write an analytical essay. Instead of going into depth on the implications of a piece of literature, they may resort instead to summarizing the story’s main events. Because of this, we divided our coding of summary into two categories: general and relevant summary.

Summary can be used in effective ways when it is guided by an overarching purpose. This moves summary away from item-by-item knowledge-telling to more advanced forms of knowledge-telling where writers prioritize information, presenting summary in more strategic ways in order to contextualize the writer’s ideas (Scardamalia & Bereiter, 1987).

Relevant summary was defined as summary that helped situate the reader and put the writer’s quotes and comments into context in such a way that they related to a claim or main idea. This could be an overarching claim made by the essay writer or other statements related to addressing different aspects of the essay prompt.

In contrast, general summary included summary that contained details and events from the story extraneous to the claim being made by the essay or the essay’s attempt to address the prompt. General summary also included summary that was sparse or sketchy in a way that made it unable to provide a clear context. In essays that made no claim, summary was often general because there was no main idea to give it direction.

Irrelevant and Relevant Quotes

Students are often taught to use quotes from the text in order to support the claims that they make in their writing. This is a common technique for giving the writer’s claims credibility and plays an important role in crafting strong essays. The essay prompts ask students to use examples from the text, and this expectation comes up again in the scoring rubric.

Even though many students followed the instructions and included quotes, these quotes did not always serve their intended purpose of supporting the ideas presented in the essay. To take this into account, we separated the use of quotes into the use of relevant and irrelevant quotes, and it was possible for essays to contain both. In addition, single words within quotation marks with no citation were generally not treated as quotes because it was difficult to distinguish between textual references and use of quotation marks for emphasis. An exception was made for Spanish dialogue taken from the text because these were easily identifiable as direct quotations. Also, many writers used quotes as the first line of their essay as a hook; these were also not counted as quotes because their placement implied that they were not meant to support an argument.

The following is an example of a relevant quote:

The size of Martin’s stomach makes the narrator and his cousins very jealous. They have been wanting to enjoy luxiourous snacks as well. They make Martin an outcast to satisfy their own problems. On page 2 it states, “It was his enormous belly that caused us to immediately dislike him.” This line shows us how greatly affected the kids must have been. Without wanting to know more, they immediately stated to bully him. (8M_3H0832026)

In this passage, the writer introduced the idea that the narrator and his friends were jealous of Martin because of his belly and then goes on to provide a quote from the text that specifically states this dislike. Therefore, the quote is relevant to the idea being presented. Relevant quotes did not necessarily have to support the main argument of the essay. Although it was usually necessary for the essay to have a claim in order for a quote to be considered relevant, relevance was also granted when writers made an effort to explain the quote in such a way that it clearly supported surrounding commentary.

In contrast, irrelevant quotes included quotes that did not support the claims being made. For example:

After the kids been judgemental, Martin stared at them and they stared at him, they don't talk, they simply stared at each other. They "stared in a silent eternity, an eternity that was broken when Martin languidly picked up a clod from the ground and threw it" at them. Martin threw a clod to them without reason which suggest that he is or can be aggressive. (8M_3J1032528)

Here, the quote is embedded in a description of an event from the story. Although the writer concludes that Martin was aggressive because of his unprovoked attack, the quote itself does not clearly demonstrate that the attack was, in fact, unprovoked. This kind of embedded quote that seems to serve mainly as an extension of story summary was fairly typical of a large number of irrelevant quotes.

Commentary

Given the small number of both score-3 and score-4 essays containing analysis, essays were coded for inclusion of commentary. Unlike other categories coded either yes or no, commentary was coded as a percentage (how many words out of the total words in each essay were commentary). When students make comments about what they read, they demonstrate that they are engaging with the text. By forming their own opinions or trying to make sense of the text in their own words, they are beginning to use a knowledge-transformation approach to writing.

Commentary could include inferences made by the student, opinions about the characters or story, and speculation about character motives or alternative outcomes if some aspect of the story were to be changed. Consider the following excerpt:

I feel like its wrong of the grandma to start off talking Spanish when she knows that she can speak english. They could of start off on the right foot if she had not spoke Spanish (6H_3I1132113)

In this passage, the writer presents an opinion about the events of the text—noting that she feels what the great grandmother did was wrong. She goes on to speculate that if the great grandmother had chosen to act differently, perhaps she and her grandson would have had a better initial relationship.

Although phrases like “I think,” “I feel,” and “this shows” were often indicators of upcoming commentary, not all sentences that contained such phrases were, in fact, commentary. For example, another student wrote:

I think the narrator’s relationship with his great grandma begins to change when the narrator was surprised that his great grandma could speak english and then the narrator begins to talk to her. (6H_3I1132119)

Although this sentence begins with “I think,” all of the information presented comes directly from the text. It is stated in the story that the incident where the narrator learns that his great grandmother can speak English shocked him. The text also says that this discovery changed everything, and that he began to spend more time with her. In other words, the sentence failed to add any new information to the student’s essay that was not already available in the story. Thus, it was akin to knowledge-telling, not knowledge-transformation. Commentary, then, can be conceived as being on a continuum with general comments on one end and analysis on the other.

Analysis

All analysis was commentary, but only certain kinds of commentary demonstrated analysis. Analysis was defined as specific explanation that links a quote/piece of evidence to a claim, or discussion of how a quote supports a claim using specific references where the writer draws conclusions based on presented textual evidence. The following example is from a score-4 essay.

When Nick Vaca and his playmates first saw Martin for the first time, their initial raction was to automatically hate him, just by his appearance. In Vaca's childhood, almost everyone was very poor and it caused him to judge others if they weren't like them…The boys reacted and stated what they thought of him even before they got to know him. He hated him because for them, "a large belly meant that a person ate well-- too well." They were jealous that he appeared to have meals that fattened him up and completely ignored the rest of his appearance. Vaca didn't bother to get to know him and Figure out what made him the way he looks. (8M_3J1133229)

Here, the writer introduces the idea of the boys disliking Martin for his belly, and then provides direct quotations from the text that illustrate both this dislike and their reasoning—that the large belly made them believe Martin ate “too well.” After this, the writer explains that eating “too well” meant the boys thought he ate so much food that it made him fat, further explaining to the reader why this quote is important in showing the boys’ jealousy.

If the student presented his reasoning clearly, the essay was marked as having analysis even when certain points in the explanation were incorrect. For instance:

“Good luck, Charlie. That old lady’s dynamite,” said the author’s uncle Manuel to his father. In this phrase the uncle used a sense of personification comparing the grandma as a dynamite. This shows how the grandma can be a bit over heated or easily angered knowing that dynamite is an explosive and hard to handle. (8H_3A1231509)

Saying that someone is dynamite is not personification; it is an example of a metaphor. However, the student goes on to clarify that this shows something about grandma because of the characteristics of dynamite. So even though some of what the student writes is incorrect, he does demonstrate some analysis and focuses specifically on the quote, showing the reader why he believes it means what he says it does.

Answering the Prompt

Adherence to the writing prompt was broken down into five different categories according to the main points that writers were asked to address. These included (a) discussion of the characters’ initial perceptions of one another, (b) how these perceptions changed over the course of the story, (c) how the author used figurative language, (d) what lesson the story was trying to convey, and (e) why that lesson was significant. This was done to see whether answering all of the prompt was a major difference between score-3 and score-4 essays and if particular aspects of the prompt were more commonly responded to, perhaps reflecting aspects that were easier or harder for students to answer. To be considered as having addressed any of these five categories, writers had to discuss it at some point after the start of the essay. In other words, it was not enough only to mention these elements in the introductory paragraph. This meant that, if an essay did not manage to move past the introduction, even if it was a very well crafted introduction, it would not have received marks for any of these categories.

Results

The following section includes the coding results by category for each of the three essay groups (not-pass, adequate-pass, and strong-pass). Then, specific comparisons are highlighted to answer each of the two original research questions.

Figure 1

Figure 2

The results of t-tests comparing the percentage of score-3 (not-pass) and score-4 (adequate-pass) essays that displayed each of the textual features shown in Figures 1 and 2 found the difference between essays at these two score levels was statistically significant at p < 0.001 for all features except the use of irrelevant quotes (p = 0.862). Adequate-pass essays also tended to contain more commentary than not-pass essays with an average of 24.1% in score-3 essays and 35.4% in score-4 (p < .001).

Although very few score-4 essays addressed all parts of the prompt, more score-4 essays attempted to discuss each part of the prompt, and 60% made overarching claims about the theme or message of the text. In contrast, only 19% of non-passing score-3 essays made such claims, and none of these essays attempted to address all parts of the prompt (see Figure 2).

In regards to which differences, if any, were the most predictive of an essay receiving a 4 as opposed to a 3, logistic regression results showed having relevant summary, using relevant quotes, and having a clear end to be significant at the p < 0.001 level (see Table 2). In particular, the use of relevant summary was an important distinguishing feature that set apart score-4 essays from score-3 essays (OR of score-4 to score-3 = 32.2)—more than five times the next highest OR, which was for having a clear end at 6.4.

Table 2

Comparing the textual features of essays that were just proficient to essays that were strong, t-test results showed that adequate-pass (score-4) essays looked very different from strong-pass (score-5/6) essays in all categories with p < 0.03 except the use of general summary (p = 0.323), the inclusion of irrelevant quotes (p = 0.127), and addressing the part of the prompt that asked about the characters’ initial perceptions of one another (p = 0.1105; see Figures 1 and 2 for proportions). In regards to which differences, if any, were the most predictive of an essay receiving a 5/6 as opposed to a 4, logistic regression results showed that the greatest distinguishing features (p < 0.001) were having a clear end (OR of strong-pass to adequate-pass = 20.0) and having analysis (OR of strong-pass to adequate-pass = 17.1; see Table 3).

Table 3

Additionally, there was, on average, more commentary in strong-pass essays than in adequate-pass essays (35.4% in score-4 and 56.7% in score-5&6). This difference was statistically significant (p < .001).

Discussion

Prior research on the relationship between textual features and writing proficiency has largely focused on macro and micro level text structures, such as whether students displayed awareness of different text structures or how students used different structural elements like lists in writing expository texts (Englert & Hiebert, 1984; Englert, Stewart, & Hiebert, 1988; Hayes, 2011; MacArthur & Philippakos, 2010). The current study took a slightly different approach to understanding student writing by focusing on a variety of specific textual components and how the inclusion of each element affected the quality of middle and high school students’ text-based, analytical essays. In summary, we found that for such essays (a) using relevant summary, including relevant quotes, and having a clear end were especially significant predictors of whether essays received an adequate-pass rather than a not-pass with relevant summary demonstrating the greatest influence; (b) including analysis and having a clear end were important in determining whether essays were strong rather than just adequate; and (c) not-pass, adequate-pass, and strong-pass essays were similar in their use of irrelevant quotes. The following discussion focuses on these primary distinguishing characteristics and their relationship to generalized skills in the domain of writing as highlighted by writing assessments like the NAEP and educational standards like the CCSS. We will start with the features that set apart essays in the two sets of comparison groups—borderline not-pass to adequate-pass, then adequate-pass to strong-pass. Similarities that emerged across all three essay samples will also be discussed.

Summarizing, Making Claims, and Choosing Key Points

The use of relevant summary was a major distinguishing factor between not-pass and adequate-pass essays. Writing Next, a meta-analysis by Graham and Perin (2007) of writing instruction that works for students from 4th to 12th grade, identified the teaching of explicit strategies for summarizing as an instructional method shown to have significant, positive effects on student writing. These included modeling of good summaries, concept mapping, and specific rules for summarizing such as deleting redundant information and replacing sentences with words or short phrases. However, simply teaching summarization strategies is not enough. Although studies conducted by Bean and Steenwyk (1984), as well as Chang, Sung, and Chen (2002)—both cited in Writing Next—found the teaching of summarization strategies and scaffolding to assist in students’ reading comprehension and summary writing, they focused primarily on whether students were able to write a brief and coherent summary of the main ideas of an expository text. Although this task included deleting redundant ideas or extraneous details, this type of summary task is very different from the kind of purposeful summary needed when students are asked to make their own claims about a text.

Writing is an important skill for success when students enter college not only because it is a means by which we communicate, but because it is a powerful learning tool for students to reflect upon and reconstruct knowledge. Purposeful summary, which involves consciously choosing and making connections between the pieces of information one conveys, is one demonstration of this skill that seems closely linked to students’ development of writing proficiency. Therefore, in order to help students become more proficient writers, we need to work with them on how to summarize in a purposeful way, which includes understanding how to make a claim and encouraging them to include commentary to explain their thinking, as these moves help shape relevant summary. It may be useful, for example, to model for students how the same text might be summarized in different ways to support different arguments. Being a proficient writer with solid writing skills according to such standardized assessments as the NAEP includes being able to communicate with a purpose. Placing too much emphasis on summarizing without also focusing on relevance and including commentary may only result in extensive general summary that fails to convey the student’s ideas and lingers in a knowledge-telling rather than knowledge-transforming framework (Scardamalia & Bereiter, 1987). In fact, for the sample of essays examined in this study, use of general summary actually decreased the probability that an essay had received a passing score.

Explaining One’s Thinking, Commentary, and Analysis

The ability to analyze texts and to write about one’s reasoning is an essential aspect of academic writing at the high school and college level. The CCSS for English Language Arts (2010) states, for instance, that, by the time students graduate from high school, they should be able to both “comprehend and evaluate complex texts,” “cite specific evidence when offering an oral or written interpretation of a text,” and “use relevant evidence when supporting their own points in writing and speaking, making their reasoning clear to the reader or listener” (p. 7). This combination of specific, text-based evidence and clarity of reasoning that set analysis apart from other commentary was a major distinguishing characteristic between strong-pass and adequate-pass essays.

For students to become strong writers and not just proficient ones, they need to learn how to connect to a claim both the comments they make about a text and the evidence they include. Commentary was coded as analysis when it created these specific connections and when it demonstrated close reading of the language used by the author. An example of this can be seen in the way the author’s use of figurative language (prompt part 3) was discussed in essays that received higher scores. For instance, discussion of figurative language in score-3 and score-4 essays—when it was mentioned at all—often consisted of comments about the author using figurative language well or lists of types of figurative language plus examples (i.e., “quote from text” then “this is a metaphor”). Interpretations tended to be shallow and poorly explained and often focused on the general content and meaning rather than what work the language accomplished. In contrast, strong-pass essays analyzed the use of figurative language and went into more detail about the text, making connections between these rhetorical moves and claims made by the essay writer. Strong-pass essays also tended to integrate commentary and analysis throughout the essay rather than devoting one particular paragraph to this purpose.

Although essays at all three levels included commentary, strong essays were much more likely to contain this kind of explicit commentary that actually explained the student’s thinking about how particular pieces of evidence related to their ideas. Rather than simply including a quote and then saying “this shows...”—a common evidence-comment pattern in many of these essays—strong essays also explained “how” and “why.”

The teaching of explicit strategies for thinking about text are a key component of the Pathway Project and has been shown to be an effective means of increasing students’ use of commentary (Olson et al., 2017). These include the use of different colored highlighters to make the proportions of summary, evidence, and commentary more visible in students’ essays and the use of sentence starters to assist students in thinking about a text. Sentence starters or frames help students reflect upon what they read, clarify their understanding of the text, revise meaning, evaluate, and make inferences (see Olson, Land, Anselmi, & AuBuchon, 2010; Olson et al. 2017).

A Clear End and a Sense of Completeness

Having a clear end was among the top two distinguishing characteristics for both comparisons (not-pass versus adequate-pass and adequate-pass versus strong-pass), which is interesting because having a sense of completeness was specifically mentioned in the Smarter Balanced rubrics as a criterion that set apart adequate writing from poor writing. For the current study, this generally meant having some kind of concluding paragraph where students wrapped up their interpretations of the text or recapped their primary claim. Modeling how to write conclusions—like modeling how to write summaries—may be beneficial along with discussion of what conclusions are meant to do for an essay and teaching of common conclusion structures or strategies.

It is also important to note, however, that the essays analyzed for this study were from a timed, on-demand assessment. As such, the importance of a clear end to what score an essay received may better indicate how quickly a student is able to organize and translate his/her thoughts into written words. Therefore, time management strategies in regards to writing may also be extremely helpful.

The Use of Quotes as Evidence

The fact that the use of irrelevant quotes was not statistically different across not-pass, adequate-pass, and strong-pass essays aligns with previous studies, which found the ability to recognize non-relevant information is more challenging than providing relevant details for students at several developmental levels (Englert et al., 1988; Englert & Thomas, 1987). In regards to the use of quoted evidence in text-based, analytical essays, this may also point towards a different understanding of the use of quotes—not necessarily as evidence for a claim as is the usual case in academic papers, but rather as a way of authenticating summary. In other words, by using the words of the author as part of their own story summaries, students may be attempting to increase the credibility or text-based nature of their writing while, at the same time, enhancing the effects of their summaries.

It was common, for instance, for students to quote the Spanish words and phrases spoken by characters in the text while summarizing events even when the speech did not add meaning to their interpretations and usually without translating them for the reader. This kind of quoting seems reminiscent of storytelling techniques where such voicing of characters may be used to enhance a narrative. However, although this can be a powerful tool in narrative writing, it can come across as extraneous and off topic in an academic essay. At the same time, students may believe that including such quotes adds to their credibility and demonstrates their understanding of the text. It may be helpful to discuss with students how quotes are used—and not used—in text-based academic writing.

Limitations

The results from this study were based on essays written for an on-demand assessment in response to a text-based, analytical writing prompt. Findings reflect what these students were able to do during test conditions for a specific type of writing task. Such timed, on-demand writing tasks play a major role in students’ lives and demonstrate what students can do independently. However, these results do not reflect what students might be able to do if given the chance to revise. Writing with multiple drafts is also important, but that is a different kind of writing than that examined in this study.

These results should also not be generalized to other contexts or other types of writing such as narratives where the norms and expectations of what constitutes quality writing may differ. Writing is influenced by many factors including culture, politics, and the specific community in which the writing is produced, and all of these factors affect how writing is evaluated (Graham, 2018).

Lastly, this study only examined the actual texts that students produced. Further research on students’ thinking processes when approaching such writing tasks would be helpful in building our understanding of why students have difficulties in the areas identified above.

Conclusion: Implications for Researchers and Practitioners

When all is said and done, writing is primarily a means of communication, and many of the criteria for writing proficiency outlined by assessments such as the NAEP relate to the communicative efficiency and effectiveness of a piece of written text. Does the piece of writing feel complete? Did the writer make a point and stick to it? Were the details and the words chosen by the writer appropriate to what they were trying to convey? Scoring rubrics highlight such features because they demonstrate the writer’s ability to communicate with readers in a way that involves being clear in one’s reasoning and not confusing one’s intended audience.

Text-based, on-demand writing assessments similar to that examined in this paper are currently the prevailing means by which writing proficiency is determined at the state and national level in the United States. Based on our analysis, it seems that the writing of students just below the level of proficiency as defined by such assessments straddles the line between pure knowledge-telling and the beginnings of knowledge-transformation, trying to move beyond a model of writing that simply involves retrieving and regurgitating information. As students learn to make their own claims and comments about the texts they read, they begin to think about and approach writing in a way that opens the door to making their own meaning out of the text. Developing a communicative purpose and structuring a piece of writing accordingly are all aspects of knowledge-transformation. So, as we consider what truly differentiates proficient from non-proficient writers, making this bridge from knowledge-telling to knowledge-transformation is key.

With the heavy emphasis in the Common Core on text-based analytical writing for students in the upper grades, this study helps elaborate upon the different attributes that characterize this form of writing from students at three different levels—not-pass, adequate-pass, and strong-pass. The results suggest that, although analysis is an important element of strong essays, it may be more beneficial for students struggling to become proficient to first focus on strategies for understanding and writing relevant or purposeful summary, making claims, and including commentary. A solid grasp of these elements, which are closely related to expressing and explaining the writer’s ideas, may then be used to facilitate deeper critical analysis and interpretation of specific text-based evidence. In this way, students can build the skills they need not only to analyze texts but to take control of how they communicate those interpretations.

If as Hayes (2011) suggests, knowledge-telling and knowledge-transformation exist on a continuum, researchers and practitioners need a deep understanding of not only where a student’s written work falls on that continuum, as reflected by the scores on a rubric such as the NAEP, but also how to scaffold instruction to support students in transitioning to the next level. Making the components of high quality essay writing more visible to students by examining the attributes of marginal/not-pass, proficient, and strong-pass essays, and explicitly teaching how to write a coherent introduction, make and defend a claim, connect purposeful summary to relevant evidence, comment upon the significance of that evidence, and analyze why the specific evidence supports the claim are crucial to student success.

Author Notes

Vicky Chen is a doctoral student studying language and literacy in the School of Education at the University of California, Irvine. She has taught English both at home in California and abroad in Taiwan. Her research interests include writing instruction, genre, transfer, and second language writing.

Carol Booth Olson is a Professor Emerita of Education at the University of California, Irvine. She is also the developer of the Pathway to Academic Success intervention. Her research interests are in developing literacy interventions, particularly in support of enhancing students’ academic writing, with an emphasis on improving educational outcomes for English learners.

Huy Q. Chung is the Project Scientist and Director of Research for the UCI Writing Project. He received his Ph.D. in Education from the University of California, Irvine. His research interests include studying literacy interventions, teacher learning from professional development, learning progressions for English learners, and formative assessments.

 

References

ACT, Inc. (2007). Writing framework for the 2011 national assessment of educational progress. Iowa City, IA: Author, the National Assessment Governing Board.

Applebee, A. N. (2007).  Issues in large-scale writing assessment perspectives from the National Assessment of Educational Progress.  Journal of Writing Assessment, 3(2), 81-98.  Retrieved from http://www.journalofwriting assessment.org/archives/3-2.2pdf.

Bang, H. J. (2013). Reliability of National Writing Project’s Analytic Writing Continuum Assessment System. The Journal of Writing Assessment, 6(1). Retrieved from http://journalofwritingassessment.org/article.php?article=67

Bean, T. W., & Steenwyk, F. L. (1984). The effect of three forms of summarization instruction on sixth graders' summary writing and comprehension. Journal of Reading Behavior, 16(4), 297-306.

Berkenkotter, C. (1981). Understanding a writer's awareness of audience. College Composition and Communication, 32(4), 388-399.

California Department of Education. (2008a). English language arts teacher guide, California

High School Exit Examination. Sacramento, CA: Author.

California Department of Education. (2008b). Teacher guide for the 2008 California writing

standards test in Grade 7. Sacramento, CA: Author.

Chang, K. E., Sung, Y. T., & Chen, I. D. (2002). The effect of concept mapping to enhance text comprehension and summarization. The Journal of Experimental Education, 71(1), 5-23.

Common Core State Standards Initiative. (2010). Common Core State Standards for English Language Arts. Retrieved from http://www.corestandards.org/assets/CCSSI_ELA%20Standards.pdf

Dominick, V. L., Stevens, C., & Smith, M. J. (2007). Transition from high school to college. A qualitative study of developmental students. In D. B. Lundell, J. L. Higbee, I. M. Duranzyck, & E. Goff (Eds.) Student standpoints about access programs in higher education (pp. 21-30). Minneapolis, MN: The Center for Research on Developmental Education and Urban Literacy.

Englert, C. S., & Hiebert, E. H. (1984). Children's developing awareness of text structures in expository materials. Journal of Educational Psychology, 76(1), 65.

Englert, C. S., Stewart, S. R., & Hiebert, E. H. (1988). Young writers' use of text structure in expository text generation. Journal of Educational Psychology, 80(2), 143.

Englert, C. S., & Thomas, C. C. (1987). Sensitivity to text structure in reading and writing: A comparison between learning disabled and non-learning disabled students. Learning Disability Quarterly, 10(2), 93-105.

Flower, L. S. (1981). Revising writer-based prose. Journal of Basic Writing, 3(3), 62.

Flower, L., & Hayes, J. R. (1980). The cognition of discovery: Defining a rhetorical problem. College Composition and Communication, 31(1), 21-32.

Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College Composition and Communication, 32(4), 365-387.

Graham, S. (2018). A writer(s)-within-community model of writing. In C. Bazerman et al. (Eds.), The lifespan development of writing (pp. 272-325). Urbana, IL: National Council of Teachers of English.

Graham, S., Burch, J., Fitzgerald, J., Fuedrick, L., Furgeson, J., Greene, K., . . . Smither Walgin, C. (2016). Teaching secondary students to write effectively (NCEE 2017-4002). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Graham, S., & Perin, D. (2007). Writing next: Effective strategies to improve writing of adolescents in middle and high schools. New York: Carnegie Corporation.

Haslam, G. (1995). The horned toad. Petaluma, CA: Thwack, Pow, Productions.

Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing.

Hayes, J. R. (2011). Kinds of knowledge-telling: Modeling early writing development. Journal of Writing Research, 3(2), 73-92.

MacArthur, C. A., & Philippakos, Z. (2010). Instruction in a strategy for compare–contrast writing. Exceptional Children, 76(4), 438-456.

National Assessment of Educational Progress. (2011). Item Maps. Retrieved from NCES website: https://nationsreportcard.gov/itemmaps/?subj=WRI&grade=8&year=2011

National Assessment of Educational Progress. (2016). What does the NAEP writing assessment measure? Retrieved from NCES website: https://nces.ed.gov/nationsreportcard/writing/whatmeasure.aspx

National Assessment of Educational Progress. (2017). NAEP overview. Retrieved from NCES website: https://nces.ed.gov/nationsreportcard/about/

National Assessment Governing Board. (2010).  Writing framework for the 2011 National Assessment of Educational Progress.  Washington, DC:  U.S. Department of Education.

National Commission on Writing. (2004, September). Writing: A ticket to work... or a ticket out: A survey of business leaders. Retrieved from http://www.collegeboard.com/prod_downloads/writingcom/writing-ticket-to-work.pdf

Olson, C. B., Land, R., Anselmi, T., & AuBuchon, C. (2010). Teaching secondary English learners to understand, analyze, and write interpretive essays about theme. Journal of Adolescent & Adult Literacy, 54(4), 245-256.

Olson, C. B., Matuchniak, T., Chung, H. Q., Stumpf, R., & Farkas, G. I. (2017). Reducing achievement gaps in academic writing for Latinos and English learners in grades 7-12. Journal of Educational Psychology, 109(1), 1-21.

Scardamalia, M., & Bereiter, C. (1987). Knowledge telling and knowledge transforming in written composition. Advances in Applied Psycholinguistics, 2, 142–175.

Smarter Balanced Assessment Consortium. (n.d.). Smarter Balanced performance task scoring rubrics. Retrieved from http://www.ode.state.or.us/wma/teachlearn/subjects/science/assessment/smarter-balanced_scoring_rubrics.pdf

Sommers, N. (1980). Revision strategies of student writers and experienced adult writers. College Composition and Communication, 31(4), 378-388.

U.S. Department of Education, National Center for Education Statistics. (2012). The nation's report card:  Writing 2011. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education.

Vaca, N. C. (1967). Martin. El Grito: A Journal of Contemporary Mexican American Thought, 1(1), 25-31.

Wenglinsky, H. (2005). Using technology wisely: The keys to success in schools. NY: Teachers College Press.

Wilcox, K. C. (2014). English language learners. In Writing instruction that works (pp. 130-145). New York, Teachers College Press.

Appendix A

Summarized Writing Prompt for “Martin”

Writing Directions
In “Martin,” Vaca and his playmates react strongly to a newcomer who moves into their neighborhood.

Write an essay in which you make a claim about what Vaca learns as a result of his conflict with Martin.

In the body of your essay:

  • Discuss why Vaca and his playmates first react to Martin the way they do and how these initial perceptions of Martin influence their treatment of him.
  • Explain what enables Vaca to change his impression of Martin and discuss how Vaca’s use of symbolism and imagery reflects his realization.

In your conclusion, describe why Vaca’s experience with Martin made such a lasting impression on him and why the lesson he learned is especially significant.

(Note: The prompt for “The Horned Toad” was structured the same way and asked students to perform similar tasks.)

 

Appendix B

Sample Scoring Rubric (Score-4)

4 Adequate Achievement

  • Writer orients the reader adequately by giving at least some introductory
  • Writer may begin unsteadily but reaches a focus or
  • Overall, writer offers an adequate claim about what the narrator learns as a result of his relationship with his great grandma or conflict with Martin, although this claim may not be in the
  • Writer offers an adequate discussion of what motivates the narrator to react to the arrival of Great Grandma or Martin the way he does based upon how he perceives her/him.
  • Writer adequately describes how and why the narrator subsequently interacts with Great Grandma or Martin the way he
  • Writer may only superficially or indirectly discuss the figurative language (i.e. symbols, imagery, ) the author uses to show the link between the characters in "The Homed Toad" or to describe the setting, the characters, and their actions/interaction in "Martin."
  • Writer adequately explains what enables the narrator change his impression of Great Grandma or
  • Writer adequately discusses why the lesson learned made a lasting impression on the narrator, but this may be addressed
  • Writer weaves a few references from the text into the paper to support his/her
  • Writer interprets less authoritatively than a 5 While the paper has a conclusion, the development of the paper toward that conclusion may be less logically organized.
  • Writer adequately uses some precise, descriptive language, and transition
  • Paper has some errors in the conventions of written English, but none that interfere with the