Scientific misconduct allegations: tell me, what would you do?
What is the appropriate way to investigate allegations of misconduct? At PLOS Biology we can attest anecdotally to an increase (which others have commented on) in the number and type of allegations all journals are receiving. Very occasionally issues arise during peer review or as a result of our pre-publication checks of accepted manuscripts. Other concerns come anonymously from readers of published papers, or from current or former colleagues of published authors. Some arise within the ‘comments’ section of a published article. In keeping with the general PLOS policy around comments on published papers we do not allow actual or apparent accusations of misconduct or unfounded assertions to remain within the comments section of a paper, but we nevertheless assiduously investigate any concerns that are raised there. Regardless of how a concern or accusation relating to misconduct is presented to the editorial team, each one is responded to and followed up assiduously. At any time, the PLOS journals that I am involved with (PLOS Biology, PLOS Genetics and PLOS Computational Biology) are between them actively investigating tens of accusations – and I want to be clear that in all the points I make in this post I am not referring to any particular current or past case.
In considering all cases we aim to follow both the letter and the spirit of the advice offered by the Committee on Publication Ethics, of which we are members. We first check that we understand the allegation and whether the complainant is happy to be identified to the authors, if appropriate. We then approach the authors in a non-accusatory way to ask for an explanation, and allow them a week or two to respond. We assess their response, and sometimes there is a simple remedy. For example, in a situation where the authors need to acknowledge in a comment on a published article that they did indeed use the same marker lane for two halves of a gel that ended up in different figures, no wrong has been done, and a little extra clarification will help future readers.
But sometimes the issue is much more complicated. The accusation may be much more serious, and/or the experiments done a long time ago. The person who did the study may have left the lab, for example, leaving the lab head grappling with old records and files to understand the situation in full. And once the situation is fully understood, what if it appears there has indeed been wrongdoing? It is very important to understand who did what and when, and to ensure a fair hearing for all parties. And once an issue is identified the process of investigating it lies with the authors’ institutions and their funders, and sometimes with regulatory oversight bodies, not with journals. Some investigations span several institutions and many years of work. In general, we expect investigations to take weeks or months, and very occasionally they take years. This is frustrating, and is not within our control.
What should happen while an investigation is ongoing? Our view in light of legal concerns and guidance from the Committee on Publication Ethics is that it is unreasonable to publicise an allegation before any wrongdoing is proven. Indeed, on occasion publishers and journals are threatened with legal action for simply mentioning the existence of an investigation, let alone predicting its outcome. This means that, while an investigation may be ongoing, we may not be able even to confirm its existence, and we cannot routinely issue some kind of ‘advance warning’ to readers that a particular paper may in the long term prove to be unsafe.
I hope it is evident to readers of this post that the staff of PLOS Biology take issues of misconduct and mistakes within the literature very seriously. We want what is published to be as ‘right’ as it can be, and we want to correct errors that occur. Everyone would agree that science benefits from discussion and debate on published papers; even papers that were accurate at the time of publication may subsequently be superseded. We need better mechanisms for highlighting errors that are identified and for correcting them, and this is an area in which we at PLOS are actively seeking technological solutions.
But a scientific paper is generally prepared by many, and any misconduct investigation and its outcome could impact all of their careers in profound ways, so we take equally seriously the rights of all parties to have a fair and unbiased hearing. That is why it is particularly troubling to me that we are seeing a proliferation of websites devoted to anonymous and/or public allegations of misconduct. I am also personally troubled when, as has happened recently, accusers appear to suggest that this journal’s staff are dishonest, lazy, incompetent or otherwise delinquent in our approach to handling these issues, simply because we will not publicly comment on proceedings that, quite rightly, happen in private. My personal request would be that those who consider Twitter or an anonymous blog post the best forum for accusations that may terminate someone’s scientific career instead rethink and try to be patient while investigations take place at an appropriate pace.
I would also like us to continue to have a (public) debate about some of the most difficult issues here, some of which have been touched on by helpful commenters on Twitter and elsewhere. How do we express concern about a paper without doing damage to innocent parties en route? It is not straightforward to highlight potential misconduct within a scientific publication without also implicating the authors or appearing to accuse them. How should we balance the wish of many to live their lives in public and with immediacy, via social media, with situations that necessarily deserve a time-consuming and confidential due process? And how do we create a blame-free mechanism of correction that allows authors to amend their papers as they learn more about their studies? We are delighted to see these issues coming to the fore in recent discussions and look forward to hearing more of your views on them. Please comment here, or tweet or e-mail us, and help guide us as we continue to try to ensure that the scientific literature is primarily home to truthful accounts of work that has been done as described and interpreted with care.
————
Update – 13 November
Many thanks to everyone for your helpful contributions, both here and on Twitter. I take home these main points from the discussion so far:
- Where we are right now on the issue of scientific publications and misconduct requires some balance between the “rights” of authors and of readers. Some people would like us to be in a world of complete openness about all things, and maybe we’ll get there at some point but for now there are issues that remain confidential. PLOS Biology shares with readers a wish to be able to have an open discussion without making accusations. There may be times when it is hard to phrase a concern without being accusatory, but we should keep trying to achieve this goal.
- Journals could be doing more to ensure data integrity. At PLOS Biology we screen all images in accepted articles for obvious signs of manipulation and if we see any cause for concern we ask for original files. Some commenters have suggested we should require – and publish – the raw image files for every composite figure, and this is an idea we’ll discuss internally and with our editorial board. Please do let us know if you have strong views on the desirability of this, as we balance the burden on authors with the potential benefit to readers.
- PLOS Biology will continue to respond to every email and other message we receive about potential misconduct – although not always instantaneously – and will try harder to ameliorate the frustration complainants feel when we are not able to share with them the progress of an investigation. There will, inevitably, be occasions when it is not an option for us to say what is going on in an investigation, and we currently cannot flag manuscripts or warn readers, however frustrating this may be.
- Much as we love Twitter for its brevity, immediacy and ease of use, there will continue to be times when it is not the best medium for discussion of complex issues.
Depends by what you mean by “a long time ago”. Labs are bound to keep original data and at least in the UK, we have for a while been required by funders to have a “Data Management Plan” whereby we agree to make original data available for a minimum of 5 years. I keep to a 10 year rule as a minimum and have a set of old computers to read legacy software – when they die, so do the data. Indeed, I recently put in the recycling bin printouts from a gamma counter from 25 years ago – illegible! I also used some 20 year old data to answer a referee’s critique in 2012. So there is little excuse for a paper that is 5-10 years old, other than poor practice, in which case the lesson is learned at the school of hard knocks.
If a paper is 5 or less years old, but the authors cannot source the data, then the paper should be pulled, unless the authors can demonstrate a force majeure, e.g., their lab burned down.
If a paper is 5-10 years old, the authors really should be able to source the data, though one or two items may be a bit moth eaten. If the authors cannot source any of the data, then I think it is the school of hard knocks again.
After all, a lab must have not just the data for the paper, but a load of ancillary data too, some of which must relate to training of new PhD students. So though original data may be a bit thin, there must be so much more that a sufficient case can be presented to demonstrate the veracity of the observations. These data may not be as pretty, but they will overlap the paper completely.
A paper is a statement to the community that you have observed some phenomenon in the natural (or synthetic in the case of chemistry) world that is some interest and which as far as you can tell is real. That means that your measurements are reproducible – a standard if you want. It is obvious then that you have to be able to pull that standard out on request. At present I would say that you should be able to produce your data for at least 10 years after publication. If you cannot back up what you published 10 years down the line, then your data management is so poor you shouldn’t be in research.
Thanks for this response to recent events. Two things catch my eye…
1) “I am also personally troubled when, as has happened recently, accusers appear to suggest that this journal’s staff are dishonest, lazy, incompetent or otherwise delinquent in our approach to handling these issues, simply because we will not publicly comment on proceedings that, quite rightly, happen in private.”
There is a difference between publicly commenting, versus keeping the original correspondent informed, which I hope you can see.
There is a difference between refusing to comment (and saying so, even in a form email), versus simply not responding at all to emails spaced several weeks apart, even 2 line emails requesting a response for the sole purpose of indicating the lights are on. Hopefully you can tell the difference?
There’s also taking a simple accusation of poor communication and falsely claiming that person is calling you lazy and incompetent. Your words, not mine, but since you chose them, let’s agree that failing to keep in touch (even a 1 line email taking 30s of your time) is both lazy and incompetent. Taking 4 months to verify the data in a western blot, is not what one typically expects from a $23m/yr operation. What’s your excuse?
2) “My personal request would be that those who consider Twitter or an anonymous blog post the best forum for accusations that may terminate someone’s scientific career instead rethink and try to be patient while investigations take place at an appropriate pace.”
The original comment on PLoS used my real name. The PubPeer comment linked to that comment, therefore indirectly was linkable to me. I actually posted it in my real name originally but on the advice of the site it was anonymized, even though it linked (for 24hrs) to the PLoS comment in my real name. I also Tweeted to the PubPeer comment using my real name (I only have one Twitter account). The tweet to NYTimes was my own name. The Pubmed Commons comment was me using my real name. The blog post by Detek Lowe was a real person, and was commented on by me using my real name. The final straw was my own blog post on my own lab website. Where do you get off calling this “anonymous”?
While it is true that I used to be the proprietor of http://www.science-fraud.org, and am a proud supporter of PubPeer and their calls for anonymity, that has nothing to do with this case. For this paper, at every level I played by the rules using my real name. The implication that anything here was done in an underhanded or anonymous manner, is ridiculous.
I’ll grant you one good point… “Try to be patient”. That’s great, but everyone’s patience wears thin, especially when their emails go unanswered for months. You can’t have it both ways. You’re allowed to say “be patient”, if you present concrete evidence of progress behind the scenes.
What you can’t do is say “be patient” and then when people ARE patient (waiting weeks for an email response is patient!), you just ignore them.
Excellent post Theo!
It is indeed important to tread carefully as such allegations are serious and as you write, potentially career-destroying.
If indeed it is impossible to even have a very general ‘flag’ on a paper under investigation (which is, of course, very understandable), then at the very least, all involved parties must be under complete and full disclosure as to the progress of the investigation. It cannot happen that either a party starting or contributing to the investigation is left out of the loop, nor any of the authors/reviewers. Journals need to be proactive, as such proceedings can be lengthy, for all the reasons you mention and more. Thus, if nothing has happened for a month, the involved parties need to be informed that things are going more slowly at such a point and reasons disclosed.
From my perspective, it is unacceptable that any of the involved parties are left uniformed at any point during such a serious process. Not getting a reply from a journal after misconduct allegations is like the police not answering your phonecalls after you told them your neighbor is beating his wife. At the very least, there needs to be a “we’re working on it and here are the reasons the process is in the state it’s currently in”. This has to happen every time an involved party contacts the journal for information, every day if need be.
Of course, it should not be public, but the involved parties need to know where the process is, at all times.
On the PubPeer thread Peer 4 wonders if it is possible to point out the problems:
“without accusing anybody in particular of any specific misdeed.”
https://pubpeer.com/publications/D2A46528724F9B59FD58693CA41560#fb3707
One small step would be to silently pull it from online access until the matter has been resolved.
Much of the problem related to scientific misconduct accusations stems from the continued practice to only publish narrative summaries of the performed research (a.k.a. papers), rather than the whole workflows (including lab notes, code and data).
If we had our workflows public by default – and a public explanation in cases where some piece cannot made public, e.g. for reasons of patient privacy – asking questions about individual items in a particular workflow would be so natural that
(a) the potential threat to the careers of those addressed by an individual question would be much less than what it may appear today;
(b) many of the typical cases of scientific misconduct (fabricating data, leaving out existing data points) would be easier to detect, probably even in early stages;
(c) potential fraudsters would be more conscious of the possibility of public exposure of their misconduct.
As I took pains to say, Paul, my post is not about any one of the many accusations under investigation at the moment or in the past, so no sentence of it should be read as being about, or directed at, you or anyone else.
I also stand by my statement that we respond to all emails. If you have a complaint about non-response on a particular issue, please could we discuss this somewhere else? Feel free to e-mail me – see http://www.plosbiology.org/static/contact.action. Thanks
This is a really good point Dave. We do need a universally agreed duration for when labs should keep data in good order, and institutions and funders have a role here in agreeing what that should be.
Thanks for this thoughtful comment, Björn. I agree with you to a point: all parties should definitely be kept informed and responded to, but I’m afraid they cannot always be given the details they might like. For example, issues under investigation at some institutions, or at a body such as the ORI (http://ori.dhhs.gov/), may have a confidentiality blanket that means we cannot report on an investigation that is underway, even to interested parties. This is why complainants get the first part of the message (“we’re working on it”), but can’t always receive details of why things aren’t moving as fast as one might hope.
I guess by your analogy, the same is true of many criminal investigations: all appears to “go quiet” while an investigation is under way or a court case being prepared, and even the victim of the crime might not be kept as informed as they would like about what is happening, and the witnesses may hear nothing at all.
I think there is an interesting discussion to be had about what form of wording or alert would be a reasonable way to say, on a paper itself, that some readers are concerned about the interpretation of some of the results in this paper, without making an accusation against the authors.
I’m afraid I don’t agree with this suggestion, but even if I did, it isn’t possible: under our open access policies we deposit full copies of our articles in various places, including PubMed Central (http://www.ncbi.nlm.nih.gov/pmc/), so pulling it from the journal website wouldn’t remove it from circulation.
As you know, Daniel, I entirely agree with you that the future should look like this. It might also help with another issue related to your point (a): it would be entirely clear which author did what part of each article.
Jonathan Eisen (@phylogenomics) made some interesting points on Twitter that I repeat here for ease of integration with the discussion.
“Well @Theo_Bloom I appreciate but don’t agree w/ your concerns nor w/ PLOS policy as applied here (pt1)
Pt2 – I think comments on content of papers should be given broad leeway as long as they do not stray outside of content
Pt3 if content of paper has flaws, whether accidental or not, it is important for those to be revealed ASAP”
I completely agree that these are sensitive cases that deserve time to be investigated properly. Consequences for the career of the authors are likely to be severe, so they must be given appropriate time to be able to respond to allegations. However, the potential damage of incorrect data has also be taken into account. How many PhDs and postdocs waste their chance of an academic career by embarking on projects that are based on false data in the literature? They have every right to know if a paper comes under scrutiny. Why should their career prospects be any less important than the ones of the authors?
I think much of the problem stems from the fact that we view papers as the end point of scientific work, rather than being part of it. If publications were less static and would be constantly discussed and modified, correcting data could become normal. I understand that PLOS is taking steps into that direction and the possibility to comment on articles is an important one. But what good does a commenting function do, if it does not actually allow to raise critical questions about the data presented in the article?
You ask an important question of how to find balance with public forums/social media and situations that may require time and confidentiality. It is time for science to fully adapt. We are able to immediately publish thoughts and opinions about someone’s work, thus it is time to publish data with the expectation that the very data will be evaluated immediately. It is the author’s responsibility to be prepared for questions and/or concerns from their online audience.
Theo,
Do I get your point correctly?
A comment that point out potential flaws in a paper is a great contribution to scientific debate, but if those flaws can only be explained by misconduct, then such comments (even carefully worded to avoid accusations of misconduct) cannot be published until a full investigation has taken place? I think that this is unreasonable. The data are here, in the public domain, for everybody to see, and just highlighting surprising aspects of the data cannot be forbidden.
Raphael
I’ve taken a look at the paper that PubPeer refers to (and that presumably started this discussion), and also at PubPeer’s fairly inflammatory original post, which can be found at the top of the chain that PubPeer refers to in a post above. I don’t see the same things being identical that PubPeer sees, but in such a case it would be appropriate for PLOS to ask the authors for the original western blots. However, I do agree with Theo that it is not appropriate for such allegations to be posted in the comments section, especially as written. I strongly suggest that all posters to this chain take a look at the original paper and at PubPeer’s allegations and decide for themselves before posting.
That said, this particular problem could easily have been avoided if the journal were to require deposition of original data (i.e. the whole western blot), either as supplements or in Dryad, as other journals are beginning to do. That way, an interested reader who was skeptical of the trimmed western panels could go check for him/herself. Requiring such a deposition would serve as an excellent check on authors who were thinking of “improving” their data.
I do agree with Jonathan Eisen that comments on paper contents should be given broad leeway, but two points here:
1. Authors should be given the opportunity to respond to comments before they are posted. As the recipient of such a comment myself (see http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.0050324) I appreciated the opportunity to respond to a comment on the validity of our findings in a temperate way that allowed the reader to decide for him/herself whether or not claims of invalidity are correct.
2. Twitter, with it’s character limit and emphasis on impulsivity, is in my opinion an inappropriate as a venue for a reasoned discussion about complicated issues.
OK, so how far are we from PLOS journals finally requiring that data and code be deposited somewhere public at submission?
I think both Paula and Fillip make good points that somewhat echo those of Daniel Mietchen and Jonathan Eisen: how do we make the shift to being able to view a research article as one snapshot of an ongoing endeavour, and to seeing it as normal to have all types of discussion and review happening in the open? I think it will take a number of steps to get us to that end point, and it is worth taking each of them as an when we can.
It should certainly be possible to flag potential problems without accusing anyone of misconduct. We have a “How to” on PubPeer:
https://pubpeer.com/howto
and
https://pubpeer.com/misconduct
It is quite simple to point to similar images as well as draw attention to more complicated issues without claiming any intent by the authors:
https://pubpeer.com/publications/54AECF24E96162E3A563AED08BE0B3#fb2952
Thanks Michael. Although I genuinely wasn’t writing about one particular case (sadly there are many), I agree with all you say. As others have noted above and elsewhere, we need to find a form of courteous discourse that encourages discussions and corrections and doesn’t automatically impugn the authors – and probably twitter isn’t quite right for that.
Raphael, I think it is a little more subtle than you suggest. For example (not a real case), if you say “I don’t believe the phylogenetic tree in Figure 5 can be right because my own work has shown dogs and cats to be much more closely related”, that is fine. If you say “The phylogenetic tree in Figure 5 is obviously fabricated and could only have been arrived at by the authors making up numbers,” that isn’t OK. But clearly there are cases in between, and that is where the grey area lies, around what is useful discussion of the article’s content and its interpretation, and what is an accusation of misconduct by the authors. In the second example, in my view it would be entirely reasonable to say “I downloaded all the sequences the authors used to build their Figure and when I build the tree it looks different, with cats and dogs switched, so could they please explain further how they arrived at theirs, ideally providing all steps along the way”.
In short, I see this discussion as being about courteous discourse and a non-accusatory approach, as well as about how we make articles into more living entities that can adapt to new results, as others have noted.
Thanks, your guidance notes are really thoughtful and a helpful guide in this thorny area. I also really like your example of a complex case that is well explained and non-accusatory.
One clarification: We have never consulted a lawyer about any of that. What we wrote just seemed reasonable to us. A larger enterprise like PLoS might want to make sure of those issues…
Try the Publication Integrity & Ethics
http://www.integrity-ethics.com/
[…] Note, there is nothing less “formal” about the process we are proposing — we just want to make the formal process something that uses the technology we now have, and which also addresses the “social” problems we see that lead to corrections or retractions not happening quickly enough. Editors frequently experience situations where a published article needs to be corrected quickly but is delayed due to procedural reasons. […]
[…] Note, there is nothing less “formal” about the process we are proposing — we just want to make the formal process something that uses the technology we now have, and which also addresses the “social” problems we see that lead to corrections or retractions not happening quickly enough. Editors frequently experience situations where a published article needs to be corrected quickly but is delayed due to procedural reasons. […]