AI and Middle East Studies, Redux
Yesterday, we began a decision meeting for a POMEPS workshop excited by the unusually large number of proposals, the on-point topics, and the diversity of scholars – many of them from the global South. We ended up deflated and deeply troubled. The first abstract on our long list read like AI to me, so on the spot we ran it through Pangram. 100% AI. The next abstract on our long list: 100% AI. The next one? 100% AI. By the end, out of 65 proposals, only 10 of them were at least 50% human written. 10 out of 65.
As it happens, our "hit" rate was pretty high – we had already selected 9 out of those 10 human written abstracts to the short list before checking them. We had already rejected 40 to 45 of the abstracts which turned out to have been AI-written because they weren't good. But the troubling area is the ten or so papers which we had ranked highly based on the topic and proposed research and had wanted to accept before checking. These each looked like exciting research, with well-formed questions and intriguing findings, directly related to the workshop theme. When they came back 100% AI-written, we really struggled with how to process them: reject on principle, disregard the use of AI and only judge the content, or conditionally accept based on a conversation with the authors about what they did?
POMEPS established an AI policy a few months ago, which I'm pretty happy with. We recognized that AI can be a useful tool for many tasks, and indeed has become standard for use in many areas. Our policy emphasizes the centrality of authorship – not just taking responsibility for all the facts, claims, quotes, citations, and so forth, but actually being responsible for generating the arguments, designing the research, interpreting the findings. (I liked Tom Pepinsky's take on this from the perspective of readers, too.) And we require a full declaration of AI use, as do most academic journals these days. Not a single one of the 55 submissions which were 100% AI declared AI usage.
There are some real issues that we as a scholarly community need to grapple with – especially a scholarly community like POMEPS that is organically connected with and actively tries to engage and support scholars from the Arab world and the broader global South. It's easy to be a purist and reject all AI use, but what would it mean for scholars doing Middle East research to not take advantage of powerful tools that their political science peers competing for the same jobs and opportunities are using freely? Pangram doesn't tell us what exactly were these scholars doing with AI. If they were generating literature reviews and ideas that weren't theirs, no problem - that's plagiarism and it should be punished. But what if it was only being used to improve their English language presentation? Would disallowing such use disproportionately harm scholars from the region? Was it being used to refine and interpret real research which had been done on the ground – and would disallowing such usage result in submerging important findings generated from those areas? And if we gave the benefit of the doubt and accepted the abstracts, and explained our policy on AI use, would they be able to produce the promised research within those parameters?
I think the debates over AI use really do have to think about these potential inequities. It extends beyond AI, or course: I was surprised to learn from Paul Musgrave's take on Gary King's controversial paper on AI reviewing that Harvard has a program called RapidPeer for its scholars to submit their papers for peer review before submission to journals, which seems like an unfair advantage accruing to the best-resourced institutions (POMEPS has long prioritized scholars from less advantaged institutions and from the region in our own virtual research workshops). If global south scholars are able to use tools such as letting Claude review their papers before submission to level the playing field, should we be punishing them for it or working with them to guide their AI usage into permissible bounds?
I have no answers to these questions. I was just disheartened and even shocked by the prevalence of AI use in these workshop proposals. It's made me even more reluctant to take on journal review assignments. And it's made me really have to wonder what we're doing here. Much more to come, I'm sure.
**
Speaking of Tom Pepinsky, he and his co-authors Jordan Gans-Morse and Daniel Gingerich have just published a terrific exploration of trends in area studies publications in political science in Perspectives on Politics:
"Drawing on an original survey of American Political Science Association members and an analysis of roughly 4,500 comparative politics articles in leading journals, we identify four shifts in the discipline: from a theoretical to an empirical orientation, from cross-national datasets to country- and region-specific studies, from macro- to micro-level analyses, and from descriptive to causal inference. We also document evolving patterns in language training, fieldwork, methods use, and data collection." "Comparative politics is more closely aligned with area studies than it has been for decades. Moreover, comparative research in contemporary political science is highly compatible with area studies, and comparativists of all generations express strong support for area studies training, regionally specific course offerings, and the hiring of scholars with area expertise."
It's a really important contribution, and engages themes I discussed in my Cambridge Element What is the Middle East?
**
A new issue of the APSA's MENA Politics section journal just came out. It's another rich set of roundtables and essays. Two essays look at the connections between the Middle East and the Horn of Africa, followed by a four essay section on Iran and the war. There's also a roundtable on Arang Keshavarzian's excellent book Making Space for the Gulf (which I reviewed here). Check out the free PDF download here.
**
I also wanted to draw attention to an outstanding new article in Middle East Law and Governance by Melani Cammett and Christiana Parreira on the challenges facing quantitative analysis of conflicts in the MENA region. It's part of a symposium on quantitative research, and I'm looking forward to the full collection. Cammett and Parreira point out a range of ethical and practical problems facing quantitative conflict researchers, and the potentially serious data problems that might be skewing findings in all kinds of ways. In their words, "we look specifically at challenges to original quantitative data collection and analysis – either the collection of on-site, already-existing data (e.g., census and electoral data) or of original quantitative data (e.g., surveys). We argue that a series of challenges shape whether such data can be collected in the first place, how it is collected, and the findings that can be drawn from it." Highly recommended, and sure to find its place in Middle East politics syllabi everywhere.
**
I've released two podcast episodes since the last time I posted – one with Sharan Grewal, the other with Idriss Jebari. I want to give each a full writeup, so just consider this a placeholder and and invitation to listen to them!
**
Finally, join me this weekend at the MESA annual conference in Boston. While there isn't a POMEPS reception this year, sadly, I'll be at the MERIP reception Saturday night. I have panels on the Arab Barometer, the decline of American primacy, and the challenges of teaching Middle East politics today. If you have a book idea for my Columbia series, or if you just want to meet up, get in touch soon as my calendar is rapidly filling up. Hope to see many of you there!