Gaming Metrics: Innovation & Surveillance in Academic Misconduct

UC Davis, February 4-5, 2016

The event is open to the public.

Follow on Twitter at #GamingMetrics.

This conference explores a recent evolution of scholarly misconduct connected to the increasing reliance of metrics in the evaluation of individual faculty, departments, and universities.  Misconduct has traditionally been tied to the pressures of “publish or perish” and, more recently, to the broadened opportunities enabled by electronic publications. The conference takes the next step and asks whether the modalities of misconduct have changed in time to adapt themselves not just to the general demands of “publish or perish” but to the specific features and techniques of the modern processes of academic evaluation variously connected to the notion of “metrics.” Have we moved from “publish or perish” to “impact or perish”? If so, are metrics of evaluation now creating new incentives for misconduct? Are metrics also helping the evolution of forms of misconduct in specific and innovative directions? And, crucially, can we reliably draw a clear separation between gaming the metrics game and engaging in misconduct?  Traditional discourses and policies of misconduct were rooted in oppositions between truth and falsehood, right and wrong, honest mistake and fabrication, but new metrics-based misconduct seem to be defined less by opposition than by degree — the amount of gaming involved.  In sum, are new metrics-based forms of misconduct asking us to rethink what misconduct means?

DAY 1 (Vanderhoef Studio Theatre, Mondavi Center)

9:00-9:15 Welcoming remarks (Ralph Hexter, Provost, UC Davis)

9:15-9:30 “FROM PUBLISH OR PERISH TO IMPACT OR PERISH” (Mario Biagioli, STS & Law, UC Davis)
A brief discussion of the conference themes and working hypothesis concerning the relation between academic metrics and misconduct. Current scenarios exemplify a vast increase of kinds of misconduct compared to traditional definitions (fabrication, falsification, plagiarism), but also point to a shift in the very goals of misconduct.  Initially driven by “publish or perish,” misconduct has become geared toward maximizing more complex metrics of academic credit encapsulated in a new imperative: “have impact or perish.”

9:30-10:30 “GAMING THE GAME ACROSS THE BOARD”
This session is meant to provide a baseline for the conference’s subsequent discussions by casting a wider net on metrics-gaming well beyond the specific field academic publishing, looking at how different communities and professions construe the line between acceptable and unacceptable gaming.  Mapping a wide range of gaming scenarios will then allow us to contextualize the specific forms of academic misconduct involving metrics gaming concerning academic credit.

• Timothy Lenoir (UC Davis, Cinema and Digital Media & Science and Technology Studies) (Chair)
• Alex Csiszar (Harvard University, History of Science) “(Gaming) Metrics Before the Game”
• Paul Wouters (Leiden University, Science and Technology Studies) “The Mismeasurement of Quality and Impact”
• Karen Levy (NYU, Media, Culture, and Communication) “Networks of Resistance in Trucking”

10:45-11:00 BREAK

11:00-12:00 “UNIVERSITY RANKINGS: GAMING OR COOKING?”
As university rankings are gaining increasing importance across the globe, they have been praised as agents of democratization against traditional academic “brands” living off reputational rent, but also criticized for the substantial ranking distortions that their easy gaming allows for.  When can these practices be treated as ranking gaming, and when do they cross over into institutional misconduct?

• Martin Kenney (UC Davis, Human Ecology) (Chair)
• Barbara Kehm (University of Glasgow, School of Education Robert Owen Centre for Education Change) “Global University Rankings: Impacts and Applications”
• Lior Pachter (UC Berkeley, Mathematics) “How King Abdulaziz University Became a ‘Better’ University than MIT in Mathematics”
• Daniele Fanelli (Stanford University, METRICS) “Institutional Pressures to Publish: What effects do we see?”

12:00-1:30 LUNCH

1:30-3:00  “PERSONAL V. INDUSTRIAL CHEATING”
One conspicuous difference from the days of “traditional” misconduct is the shift between misconduct as the work of individual scientists and scholars to scenarios in which misconduct is a more “collaborative” endeavor, as in the case of citation rings among journals to maximize their impact factors. (The production of fake alternative impact factors may be another example).  In addition to these novel conspiracies (which typically involve editors and publishers rather than traditional individual cheats like scientists and scholars), modern misconduct also involves businesses and organizations providing tools, platforms, and opportunities to academics interested in misconducting themselves.  These include so-called “predatory” journals, fake conferences, fake prizes, etc., that is, tools that enable and entice academics to meet the demands of their institutions’ evaluation metrics by gaming/cheating them.  Also, while these activities concern publications, they are not limited to the production of a fraudulent text (as “traditional” misconduct typically was), but aim at facilitating its publication.  They may be perhaps termed “postproduction” misconduct.

• MacKenzie Smith (UC Davis, University Librarian) (Chair)
• Finn Brunton (NYU, Media, Culture, and Communication) “Making People and Influencing Friends: Citation Networks and the Appearance of Significance”
• Sarah de Rijcke (Leiden University, Science and Technology Studies) “System Identity: Predatory publishing as socio-technical disruption”
• Jeffrey Beall (University of Colorado, Denver, Information Science) “Fake Impact Factors and the Abuse of Bibliometrics”
• Dan Morgan (University of California Press, Collabra Project) “Cui Bono? Judging Intentions (and Outcomes) of Personal and Industrial Cheating”

3:00-3:15 BREAK

3:15-5:00 “META GAMING, META CHEATING”
This session has a double goal.  First, to analyze the kind of gaming that involve not the manipulation of a metric but the construction or adoption of a metrics – not gaming an established game, but the gaming that goes into defining the game itself. Is the competitive market of academic metrics (from faculty performance to university rankings) a form of gaming the game itself?  And where/when/how can it become misconduct?  Second, this session aims at engaging with Goodhart’s law, which is taken to show not only that the introduction of any kind of metric creat, es a market for gaming it, but that by so doing it invalidates the significance of that metrics.  If so, one could argue that any metrics will create the possibility of misconduct, but that the articulation of forms of misconduct specific to that metric will eventually “crowd” that market, thus creating an incentive to change the metrics, which in turn will usher in the next generation of innovative misconduct.  Or can we argue, against Goodhart, that it is possible to find a metrics of academic evaluation that can break the nexus with gaming/misconduct?

• Anupam Chander (UC Davis, Law) (Chair)
• Johan Bollen (Indiana University, School of Informatics and Computing) “From Bibliometric Metrics to Crowd-Sourced Science Funding Systems”
• Carl T. Bergstrom (University of Washington, Biology) “It’s All a Game: The twin fallacies of epistemic purity and a scholarly invisible hand”
• Jennifer Lin (Crossref) “Trust through Transparency: O brave new world/ That has such data in’t!”
• Michael Power (London School of Economics, Accounting) “Research Impact and the Logic of Auditability: Solicited testimony as a case of meta-gaming”
• James Griesemer (UC Davis, Philosophy) “Taking Goodhart’s Law Meta: Gaming, Meta-Gaming, and Hacking Academic Performance Metrics”

————————————————————————————————————
DAY 2 (Kalmanovitz Appellate Courtroom, King Hall)

9:15-9:30 Welcoming Remarks (Kevin Johnson, Dean, UC Davis School of Law)

9:30-10:30 “MISCONDUCT WATCHDOGS (I)”
The emergence and pervasiveness of new forms of misconduct exceed the
reach, resources, and conceptual framework of traditional governmental watchdog organizations typically connected to funding agencies like, in the US, the ORI.  This has spawned a new generation and new figures of misconduct surveillance, detection, and prosecution. Among these is a new breed of “watchdogs” — new actors who are often institutionally unaffiliated. These “watchdogs” have assumed an important role and a credible voice, often by creating new “ecologies of support” for themselves — websites, blogs, wikis, social media, etc.  Does their somewhat unique role indicate something about the specific nature of modern academic misconduct?  Does it suggest that the “battlefield” of misconduct is moving away from governmental agencies (acting according to traditional and possibly outdated definitions of misconduct) and toward journals and the watchdogs who monitor their publications?

• Jonathan Eisen (UC Davis, Genome Center) (Chair)
• Ivan Oransky (Retraction Watch & NYU) “Retraction Watch: What We’ve Learned Since 2010”
• John Bohannon (Science Magazine) “Grey Hat Hacking for Science”
• Elizabeth Wager (Sideview) “Why Do We Need a Committee on Publication Ethics and What Should It Do?”

10:30-10:45 BREAK

10:45-12:00 “MISCONDUCT WATCHDOGS (II)”

• Jonathan Eisen (UC Davis, Genome Center) (Chair)
• Darren Taichman (Executive Deputy Editor, Annals of Internal Medicine Vice President, American College of Physicians) “A False Sense of Security?”
• Debora Weber-Wulff (University of Applied Sciences Berlin, HTW, Media and Computing & VroniPlag Wiki) “Documenting Plagiarism in Doctoral Theses: The Work of the VroniPlag Wiki Academic Community in Germany”
• Brandon Stell (The PubPeer Foundation & CNRS) “Introducing PubPeer”
• Emmanuel Didier and Catherine Guaspare (EPiDaPo, UCLA) “The Voinnet Affair: New Norms in High-Pressured Science”

12:00-1:30 LUNCH

1:30-2:30  “COUNTERFEITING BRANDS V. FAKING PRODUCTS”
This session looks at a specific form of fakery rooted in “brand appropriation.”  While the preceding session considers generally fake journals, conferences, etc., here we want to look more specifically at imaginary journals whose titles (as well as the look and feel of their websites) are made to resemble those of well-known and respectable journals.  One could perhaps add to this list certain “academic” conferences that take place in prestigious locations (say, Oxford) but are not actually affiliated with the university, or the appropriation of the names of respected academics that are then listed (without authorization) on editorial boards of fake journals or organizing committees of fake conferences.  Similarly, fake universities who sell degrees without any attempt at educating their students (not even online) tend to assume names with an Ivy League ring to them.  The common denominator here is an attempt at the mimicry of a “brand” rather than just the copying/pirating of a product.

• Madhavi Sunder (UC Davis, Law) (Chair)
• Marie-Andree Jacob (Keele University, Law) “Template, Creativity and Publication Ethics”
• Alessandro Delfanti (University of Toronto, Institute of Communication, Culture, Information and Technology) “ArXiv or viXra? Physics and the quest for the true archive”
• Sergio Sismondo (Queen’s University, Philosophy) “Leveraging Academic Value in the Pharmaceutical Industry”

2:30-3:30  CARNIVALESQUE RESPONSES
While misconduct “watchdogs” (discussed in a previous session) expose through public communication and denunciation, this session focuses on other actors who reveal misconduct and poor oversight through a carnivalesque approach.  Humor and absurdity—submitting profane papers and computer-generated gibberish articles that “sound” academic, or whistleblowers using clever anagrams as aliases–become a mode of critique and unmasking. Neither clearly “predatory” journals, “fake” conferences nor “legitimate” journals are immune to being the subject of a joke–a joke that, in some cases, may be more powerful than punishment. In a way, carnivalesque responses to misconduct continue the logic of an older history of art forgery-as-prank in which the forgery reveals through a kind of satire.  Are these cases telling us, perhaps, that satire is the best approach to both metrics and the gaming they elicit?

• Alexandra Lippman (UC Davis, Innovating Communication in Scholarship Project) (Chair)
• Cyril Labbé (Joseph Fourier University – Grenoble I) “Ike Antkare, His Publications and Those of His Disciples”
• Burkhard Morgenstern (Universität Göttingen, Bioinformatics) “Virtual Editors Can Significantly Improve the Visibility of Junk Journals – A case study”
• Paul Brookes (University of Rochester, Medicine) “Crossing the Line – Pseudonyms & Snark in Post-Pub Peer Review”

LOCATION: Please note that the conference will be held at two different locations on the UC Davis campus.  On Thursday, February 4 we will convene at the Vanderhoef Studio Theatre, Mondavi Center.  On Friday, February 5 our proceedings will take place in the Kalmanovitz Appellate Courtroom at King Hall (UC Davis Law School) (located here).

CAMPUS MAP: Can be found at http://campusmap.ucdavis.edu/

ADDITIONAL INFORMATION: Please email Alexandra Lippman (alippman at ucdavis dot edu)

INNOVATING COMMUNICATION IN SCHOLARSHIP — a UC Davis IFHA Project
CENTER FOR SCIENCE AND INNOVATION STUDIES