Automated Writing Technology: Generative AI, ChatGPT

In grad school (2015-2020) I focused my dissertation research on what I termed “automated writing technologies.” I was fascinated by such technology because it represented a nexus of the linguistic and the numeric. These computer programs allowed me to explore my fascination with both alphabetic and quantitative languages. Specifically, my project tested the ability of computers to evaluate human-generated text and asked whether automated writing evaluation could, at the time, provide effective formative feedback on the rhetorical dimension of prose writing at the level of metaphor, irony, humor, and analogy rather than the more rote aspects of grammatical correctness. My dissertation findings, while limited, were not promising.

Large Language Models (LLM) were around during this time, but quite rudimentary. They were about as good as the predictive texting on our smartphones, limited to accurately predicting one or two words at a time. In 2017, however, researchers from Google published Attention Is All You Need, a revolutionary paper about the Transformer architecture that serves as the foundation of generative AI and which represents the “T” in GPT.

By the time I finished grad school in 2020, OpenAI’s second generation LLM, GPT-2, was available to the public. Now, less than three years later, we have OpenAI’s GPT-3 and ChatGPT, as well as other proprietary AI text generators from Google, IBM, Facebook, and more. This area of technology has advanced rapidly, and I am fortunate to have graduated when I did to ride this wave.

While I am not a computer scientist, and don’t pretend to be one despite my attempts to read and understand some of the primary papers from this field, I bring a background in rhetoric and writing assessment to the study of generative AI generally and Large Language Models specifically. My experience in writing pedagogy/assessment and the history of rhetoric prepares me to contribute to the scholastic conversations about these breakthrough technologies in productive ways.

I urge other humanities and liberal arts scholars to join this conversation. I think many of us in the liberal arts have much to contribute to decisions about how to incorporate generative AI into educational and professional institutions, which we need to start thinking about sooner than later. I also think we should make an effort to try to learn some of the more technical aspects of LLMs in order to be conversant with the programmers and companies designing them.

To that end, I will be updating this site mostly about future research regarding LLMs, GPT, and generative AI more broadly, considering both theoretical questions (does AI use rhetoric in the same way we do?) and practical applications (how can we use this technology to better teach writing and thinking?)

I recently gave a talk to my university about what LLM technology is and how we might work with rather than against it in our classrooms. I also published an op-ed on Thursday 23 February 2023 in the Dallas Morning News about the same topic, how generative AI represents an opportunity to re-think and re-define how we teach writing. Finally, I gave an interview to The Texas Standard, a Texas public radio station, on why I’m not as anxious about generative AI’s impact on higher education as some others in my field. The segment aired around 10.30am Wednesday morning on 22 February 2023.

Stay tuned for more updates.

Assessing Purdue’s first year writing program

I’m currently heading up an assessment of Purdue’s first year writing program. We are collecting and analyzing a variety of student writing and beginning to report the results. This might be of interest to you if you teach or are interested in theories and practices of writing assessment. Note that this and other assessments are exploratory pilots, which will provide evidence for which aspects to refine for the full study next year.

Here’s a brief update I prepared on one of the assignments we piloted, the rhetorical analysis: Assessing the rhetorical analysis.

“I can’t teach writing in only one semester”

One complaint I’ve consistently heard (and myself made) during all my years teaching college writing is that one or two semesters is not enough time to teach writing. One finding I’ve repeatedly come across in all my years poring over educational research is that the largest source of variance in student writing performance is the interaction between student and task. In other words, the fewer the number of tasks assigned in your course (assignments, papers, tests, etc.), the higher the variation in student performance, which means the less reliably (consistently) your class measures whatever it is we conceive of as “writing ability.” This seems intuitive to me, and it is consistently found in the literature. On average two different assignments, in the same course and even for the same student, are basically nonpredictive of performance in one by performance in another. A student can write an amazing editorial and absolutely bomb a research paper. Does that make your course an unreliable measurement of writing ability? Maybe.

The logical response to this is simple: just assign many more and wildly different tasks in your course to better cover the vast domain of writing ability and thus reduce performance variance. But, alas, that’s where the time constraint comes in. It’s virtually impossible to assign more than four major assignments a semester. Are four writing assignments enough to reliably capture writing ability? No way. Not given the infinite amount of genres and writing tasks out there and our evolving definition of writing ability.

So then, what if we increased the number of assignments, but made them smaller and spent less time on each? 10 small writing assignments a semester, instead of four major ones? Would 10 assignments more reliably capture writing ability and minimize our measurement error? Statistically, yes. Intuitively, I think yes, too. But I understand the resistance to this idea. There is value, I think, in longer, more in-depth writing assignments. I bet most freshman college students haven’t written a paper longer than 10 pages, and at some point they absolutely should write one. (I think multiple.)

But I wonder if that value can be realized in a freshman writing class. What is the realistic purpose of a one semester, freshman college writing class after all? If our time, and thus our measurement instrument, is narrowed to one semester, maybe we should break up the cognitive trait we intend to measure into smaller chunks, since the whole construct can never be reliably captured in a semester with four major assignments. It’s like we’re trying to measure several miles with four yardsticks. If we’re only going to get one (or two at most) semesters(s), maybe we should adjust the use of our narrowed instrument accordingly, by using it on more, smaller, varied tasks. Then instead of measuring miles with yardsticks we’ll at least be measuring yards with rulers.