I want to stick my neck out to say that this has the potential of being very bad for science.
Imagine saying "no" to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you. And this is not just me feeling this way - quoting myself from a previous comment, and according to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents" and "support for public review inversely correlated with reviewing experience".
A measure that women ~~and inexperienced researchers~~[2] do not support is a measure that favors only those who are already part of the club.
[2] This part has been correctly pointed out as being wrong.
JumpCrisscross 10 hours ago [-]
> Imagine saying "no" to a researcher with a big social media profile
"The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now."
(Also "support for public review [being] inversely correlated with reviewing experience" means inexperienced reviewers are more likely to support it. Not less.)
probably_wrong 9 hours ago [-]
You are correct about the second point - I'll strike it through once I find out how.
As for the anonymous part, that's why I wrote "with style-detection and deanonymization tools". If the Internet could find Shia Labeouf's flag in a day [1], could they find a reviewer based on their writing?
The difference is that as a scientific reviewer you are not hiding a physical location and what you need is plausible deniability, which would still exist. In addition to this, actively attempting to deanonymise your reviewers is on the level of scientific misconduct that your employer and professional organisation should consider taking disciplinary action against you. I am not arguing that this makes it entirely safe to publish anonymised reviews and that we will not affect reviewer behaviour (maybe for the better in some cases, as "one-sentence reviews" will be something in the public record), but it is in stark contrast to the example that you bring up.
herewulf 9 hours ago [-]
Translate into $RANDOM_LANG and then back to English. The perfect prose obfuscator.
10 hours ago [-]
y1zhou 6 hours ago [-]
Has there been recent developments in the style detection and deanonymization tools you mentioned? I would assume many would not work well given the high usage of LLMs nowadays.
9 hours ago [-]
b59831 10 hours ago [-]
What is the alternative?
Or are you just for creating classes of people that just can't be critiqued in any circumstance?
This kind of sounds like, 'Wont anyone think of the grifters?!'
probably_wrong 9 hours ago [-]
I know every area is different but the "grifters" in the area of Computational Linguistics (the ACL) are "any volunteer[1] whose paper has been accepted at least once", meaning anyone from PhD students to professors and industry researchers.
Not all academia is Elsevier.
[1] This policy has been altered recently, though, and now submitting a paper comes with reviewing duties.
ninjin 7 hours ago [-]
We are struggling badly with review quality in natural language processing though. Most likely due to the unprecedented expansion of the field over the last ten or so years. Reviewers are suffering with review loads far exceeding what one reasonably can manage mentally (used to be two to three papers per reviewer and now five would be considered rather generous). Authors and area chairs suffer from worse quality reviews due to reviewer inexperience and overload, not to mention how good reviewer/author correspondence with author and area chair comments frequently being ignored by the reviewers. To me, the last holdout of good peer review in the field is Transactions of the Association for Computational Linguistics (TACL), but there the acceptance bar is sky high compared to ACL Rolling Review (ARR) for better or worse.
The ACL leadership and senior members of the field are very much aware of this and are trying their best (ARR being an attempt to improve the situation, but I am unsure how much better it really is compared to the old system of conference reviewing now that we are a few years in). But there appear to be no easy fixes for a complicated, distributed system such as peer review. Every discussion I have with said leadership and other senior members always ends with us agreeing on the problems and likewise agreeing that despite considerable mental effort we are failing to come up with solutions.
Returning to the main topic. Nature is worthy of praise for making their peer review transparent and I say this as a massive Nature critic. It is a move I loved seeing from NeurIPS (then NIPS) and ICLR over a decade ago, as it helps younger researchers see what good (and bad) communication looks like and that even papers they know now are greatly appreciated received a fair amount of criticism (sometimes unwarranted). I have argued for ACL to introduce the same thing for nearly a decade at this point, but we still do not and I have never heard a solid argument as to why not (best argument was the technological effort, but OpenReview, with all its flaws, makes this even easier than with Softconf; not that it would have been that hard with Softconf either).
aDyslecticCrow 12 hours ago [-]
This is amazing. The trust of peer review as a stamp of quality among academics is dwindling, and distrust of science among the population is growing (within the increasing politicisation of some areas of research). Transparent peer review raises the bar for both academics and enhances the potential trust in the process.
This is desperately needed as AI could further deteriorate the quality of science if the publishing process is not made more strict. This represents a significant step forward in rigorous science. I hope other publications follow suit.
jfengel 10 hours ago [-]
Of the people who distrust science, how many of them have ever read a scientific paper?
I suspect the number is low. If that's the case, they're unlikely to be more convinced by the presence of published peer review, either.
aDyslecticCrow 9 hours ago [-]
I think there are a few groups and reasons of distrust. Some more or less valid.
Those that distrust authority as a whole and lean into conspiracies cannot be saved with this kind of thing.
But i think news and science are having similar perception issues recently.
Distrust for news growing among the average population (for some good reasons). People are loosing faith in the objectivity of established media organisations. Most people are exposed to science through these traditional news.
So adding back some sense of confidence and authority to scientific institutions is very valuable to non-academics. Even if they themselves would not read the papers or revews.
Dig1t 7 hours ago [-]
The fact remains that distrust of science continues to grow. Up to now the establishment’s response has been one of condescension. Your comment echoes that attitude.
Ignoring the problem is not going to fix it, and in fact continuing to regard these people as beneath you is only going to accelerate the downfall of this system.
vkou 7 hours ago [-]
> The fact remains that distrust of science continues to grow.
Of course it does, billion-dollar interests who have a vested interest in attacking it have a well-funded propaganda arm, and as we've been discovering over the past century - angry, loud, and frequently repeated bullshit with a profit motive floods any signal out of the room.
What's really sad is people who have legitimate concerns or desires for improvement hitching their horse-cart to the former.
No amount of peer review or replication is going to convince someone whose fortune is built on peddling snake oil.
> Ignoring the problem is not going to fix it,
Here's a solution. Hold the people making these attacks to the same level of rigor as what they are attacking. Stop giving proven liars a loudspeaker. The day the press will do that is the day some meaningful progress to fix that damage may be made.
Until you do that, we're going to continue getting shit outcomes.
NoMoreNicksLeft 10 hours ago [-]
>Of the people who distrust science,
Why should anyone trust science? Skepticism should always be the default position. Putting it on a pedestal to be worshipped is what led us all into this mess. If science needs trust to work, then whatever it is doing is something I'd like to see fail.
JumpCrisscross 10 hours ago [-]
> Why should anyone trust science?
When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it.
mikeyouse 9 hours ago [-]
Right. RFK supporters aren’t carefully weighing evidence to arrive at their conclusions, they are blindly following someone who is a mindless contrarian with a 30yr vendetta against vaccines. No amount or quality of peer review is going to change any of their minds.
JumpCrisscross 9 hours ago [-]
> RFK supporters aren’t carefully weighing evidence to arrive at their conclusions, they are blindly following someone
I'd actually be curious what fraction of his supporters share his views on vaccines. It strikes me as more likely that they like one of his random, idiosyncratic views and are willing to excuse the anti-vaccine nonsense to get those policy outcomes.
fc417fc802 9 hours ago [-]
That, but also people who take the blanket view that none of the work produced by the scientific establishment is trustworthy. An ironic stance considering that it's the inverse of the blind worship that they accuse others of.
NoMoreNicksLeft 3 hours ago [-]
>none of the work produced by the scientific establishment is trustworthy.
There's that word again. No works are trustworthy. You shouldn't trust, instead verify. Be skeptical. I'm more concerned with which of their works can be verified.
>An ironic stance considering that it's the inverse of the blind worship
I'm not sure there's any difference worth mentioning between blind worship and cuddly warm trusting that you want to do. One may seem a little less zealous, but it's the same pig just with extra lipstick.
kergonath 9 hours ago [-]
> That, but also people who take the blanket view that none of the work produced by the scientific establishment is trustworthy.
That’s a red herring. No scientist actually says that. What we say is that on some subjects the evidence is overwhelming and to overcome the current understanding you need compelling evidence and theories, not screeches about bias and liberal elites.
No. We are not going to take seriously someone’s pet theory about a new perpetual motion machine, or cold fusion, or lack of global warming, or the ineffectiveness of vaccines, or anything that is contrary to massive amounts of accumulated evidence.
fc417fc802 8 hours ago [-]
I think you misread my comment? I'm talking about the non-scientist naysayers.
kergonath 8 hours ago [-]
I just think we do not disagree.
kergonath 9 hours ago [-]
> Why should anyone trust science? Skepticism should always be the default position.
Skepticism is not anti-scientific. Hell, distrusting results and theories is not anti-scientific. Distrusting the scientific method is. There is a difference.
> Putting it on a pedestal to be worshipped is what led us all into this mess.
Scientists did not ask for this. Amongst all high profile politicians, those who whine about science becoming political are those who made it so, by taking contrarian positions to rile up their voter base. Most people who make this point are not being honest. If you are, you should make specific arguments rather than rehash propaganda.
NoMoreNicksLeft 3 hours ago [-]
>Skepticism is not anti-scientific.
One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
>Distrusting the scientific method is.
Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn. And maybe when it's finally proven that, maybe I go on distrusting it... and make it prove it some more. If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven". But so little do I ever hear of that sentiment. Like in this very thread.
>Scientists did not ask for this.
Probably not. I'd be skeptical of the theory that they asked for it. But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask. And liking it, they started behaving in ways that encouraged it just the same. And those scientists who enjoyed it the most elbowed their way to the front to egg it on even more. And so that's where we are now. I mean, I'm a little skeptical that it happened that way, and I'd welcome evidence that disproves it.
>Most people who make this point are not being honest.
It's really sort of hilarious how it's only my side that can ever be honest. Not the other guys'. And this is true no matter who "me" is. None of your opponents, for instance, can ever be genuine... they're always trying to cheat. And if your side tries to cheat, well, it's with the best of intentions because the stakes are so high. Not that you would cheat, only the other side ever does that.
>f you are, you should make specific arguments rather than rehash propaganda.
If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
kergonath 1 hours ago [-]
> One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
As a matter of fact, it is already too late for some of the consequences. The permafrost is not going to re-freeze, for example, and wildfires are already significantly on the rise.
The thing is, I don’t believe this because science, I know this for a fact because it was proven. It’s like nihilistic edgelords giving cynicism a bad name. Skepticism is not the rejection of facts because they are inconvenient. It is not stubbornness in the face of evidence, either.
> Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn.
Right, but then to prove itself it needs to be evaluated and assessed. You need a kind of method to do so, including a standard of proof and the basics of reasoning and logic. This framework is itself the scientific method. If you start from the axiom that it does not work, then the only conclusion is that it does not work, because how would you prove otherwise?
> If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven".
Well, yeah. That’s what we do. Though the standards shift over time in the face of overwhelming evidence. We can not start from first principles every time, it just does not work and there are theories we can rely on. But when holes are found in these theories we try to figure out what the problem is and how to build better ones. The scientific method itself has been refined over centuries and the standards now are far from what Newton used. A common example is the evolution of what is considered to be a proof in Physics.
> But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask.
I am sure some of them do, and not necessarily the best ones. But most of us are not attracted by public engagement and just want to do our science in our lab. We did not ask to become pawns in sectarian struggles. We rang the alarm because, honestly, when the house is on fire you have to do something and not doing it would mean some degree of responsibility in the outcome.
> It's really sort of hilarious how it's only my side that can ever be honest.
I am not following. I am not talking about sides. Regrettable attitudes towards science are not confined in one political or social tribe and there are examples in history of both right- and left-leaning ideologies becoming anti-science. But yeah, making a topic political in order to then claim that it is political is blatantly dishonest. It’s a cheap rhetorical trick with lasting consequences. What is political is how you address the situation, not whether reality is real.
> If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
You are reading way too much. First, I am not attacking you. Then, a good starting point is facts we can observe and things we can infer from them instead of the a priori position that reality is not real and the other side is just making stuff up.
oerpli 10 hours ago [-]
I would go further: Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.
Your posting doesn't give me the impression that you're very familiar with "science".
iorrus 10 hours ago [-]
Absolutely, the quickest way to lose faith in "The Science" is actually to do Science in a formal research institute....
probably_wrong 9 hours ago [-]
Isn't that true for everything, though? If I weren't a software developer I wouldn't know that I have to worry about questions like "has this plane been rebooted in the last 51 days?" [1] or "does this bank offer anything other than SMS as second factor?".
Maybe structural engineers feel safer after their Master's when they traverse a bridge, but I bet that's more the exception than the rule.
Seeing how the sausage gets made makes you realize ALL the downsides and things you'd rather not have known. That doesn't mean you can think of a way to fix science. Let alone get the required funding to make an actual attempt.
kergonath 9 hours ago [-]
> Anyone who has published a "scientific paper" in the last decade or so either "distrusts science" or is more likely than not a mid-wit at best.
That is very unhelpful, to say the least. The amount of noise has increased, but it does not mean that the scientists who know their subject disappeared. They are still around and not any less bright than their predecessors were 30 years ago.
kergonath 9 hours ago [-]
> The trust of peer review as a stamp of quality among academics is dwindling
The thing is, peer review is not a stamp of quality, and never was. It is just the basic level of due diligence. The referees cannot reproduce the results most of the time for a lot of very good reasons. They are here as a sanity check, to ensure that the work avoided common pitfalls and actually makes sense.
What most people do not understand is that articles are not good because they are peer-reviewed; it’s the lack of peer review that is a red flag. Amongst reviewed articles, a lot of them will turn out to be wrong or flawed in ways that are impossible for the reviewers to find out.
thfuran 11 hours ago [-]
>and enhances the potential trust in the process.
I highly doubt it's a meaningful factor in public trust.
Ar-Curunir 11 hours ago [-]
The general public does not distrust science because peer reviews are not public.
kergonath 9 hours ago [-]
Right. The general public cannot understand the points that are being made during peer review.
jostmey 12 hours ago [-]
This only addresses a small part of the problem with peer-review. The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers. If the paper doesn’t fit what is expected, it will often be rejected. This can also lead to self-reinforcing views that ignore contrarian data. Also, the data can be made up, and if it makes sense to the reviewers, it is generally not questioned
yummypaint 10 hours ago [-]
Have you personally reviewed a non-zero number of papers? What is this statement based on? For a thread ostensibly about science, the comments are disappointingly lacking in evidence and heavy on vibes.
Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.
Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
jostmey 9 hours ago [-]
I’ve only peer reviewed a few papers. I’ve submitted my share of papers for review
> The real problem is that peer reviewers can’t possibly replicate the study, and so are forced to look for inconsistencies in the papers.
This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.
I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.
The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.
aDyslecticCrow 11 hours ago [-]
Bad quality work is a much bigger problem than dishonest work. Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.
Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
fc417fc802 9 hours ago [-]
> Systematically well-done research with fake results or data is much rarer than just... lazy bad science.
Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
jxjnskkzxxhx 12 hours ago [-]
Reading your comment makes me think that you believe that the point of peer-review is to ensure that a paper is correct, or at least that specific aspects of it are correct. Is that the case? What do you think the point of peer-review is?
yupitsme123 11 hours ago [-]
I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."
The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
thomasfedb 11 hours ago [-]
Nothing is ever “proven”. There is simply more or less support for a theory or proposition.
Replication and meta-analysis are an important part of this.
Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.
blix 8 hours ago [-]
I wish this comment was more representative of my personal experience in science.
Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
jxjnskkzxxhx 7 hours ago [-]
What field, may I ask?
blix 5 hours ago [-]
Applied physics. I'd prefer not to get too specific. Most of my peers are working for the US DoD or DoE now.
jxjnskkzxxhx 10 hours ago [-]
I approve of this comment.
kergonath 8 hours ago [-]
> I'm not the person you replied to, but I think that in the lay world, people do indeed think that peer review is as you've described. If it's not, then maybe it should be?
It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.
Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.
JumpCrisscross 10 hours ago [-]
> public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus
The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)
That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.
That's certainly true but I think there's also a very real issue as described by GP. Research gets cited in a political setting as a rhetorical tool. That cycle does a lot to erode trust in the establishment because it incentivizes non-scientists (who are otherwise uninvolved) to behave as though the process is a partisan effort to be interfered with for the benefit of one's "team" rather than an objective pursuit to be funded at arms length for the betterment of society at large.
Obviously reproducing results as part of peer review is not a workable (or even coherent) solution. I don't pretend to have any idea what a solution might be. The obvious issue is that academic publications were never intended as political tools and should not be made into that.
On several occasions I've had interactions with laymen where I found myself thinking "if only you hadn't had access to pubmed and way too much motivation we'd both be better off right now" yet I firmly believe that free and open access to information is a huge net benefit to society on the whole.
eig 12 hours ago [-]
A good step forward. Reading the referee reports and rebuttals from papers that previously opted in to transparent reviewing was incredibly useful to my own paper publishing process. In many ways Nature is ahead of Science on this.
I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor.
trelane 8 hours ago [-]
> You have full access to this article via your institution.
I would suggest to Nature that this phrase hints at much larger a problem than showing authors arguing with Reviewer 2.
dr_dshiv 12 hours ago [-]
We need more scientific societies. Modern peer review is super modern. Go back to the origins of science and it was all about a real community—setting high standards and having just a few people decide what to publish. It wasn’t trying to be “objective” — it was just high standards, determined invisibly by the members of the society. “Should we publish this?” asked the society.
Alas, scale ruins everything. Nevertheless, I strongly believe “science is friendship”
aDyslecticCrow 11 hours ago [-]
That model had some major issues. Too many opposing theories to the "established norm" were dismissed, among things we now know were wrong all along.
Although it could raise the standard of the process itself (methodology, writing) to very high levels, it restricted innovative ideas or unpopular outcomes.
10 hours ago [-]
HPsquared 10 hours ago [-]
The trouble is, scientific research is now a livelihood and career for must published researchers. It's not an aristocratic hobby anymore. The incentives are all different.
striking 11 hours ago [-]
Sounds like a recipe for serious inequity to me.
JumpCrisscross 11 hours ago [-]
> Sounds like a recipe for serious inequity to me
If it works better, it works better. The problem with the society method is it didn't work better than a decentralised scientific system.
aspenmayer 11 hours ago [-]
One of the weaknesses to such a system is human nature. We want to believe, which leads to farces like Piltdown man, which was a farce by a man called Charles Dawson (not Darwin, not the natural selection guy, though I did a double take at first), against the Geological Society and society at large. The farce wasn’t definitively disproven until 41 years later, which is quite the downside risk of the gentlemen’s club research group gatekeeping scholarly research, but the more things change, the more they stay the same. This is a good thing which seems like it’s in response to grumblings about irreproducibility and bias, but those issues will likely always be with us and must be considered anew each time an experiment is designed, and each time a print run is cut.
I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?
10 hours ago [-]
lenerdenator 11 hours ago [-]
I only got as far as undergraduate research assistant in the academia racket, so maybe I'm not "with it" in the way serious researchers are, but is this to suggest that this wasn't being done, by default, on everything already?
fc417fc802 8 hours ago [-]
Any reputable journal had a reasonably robust peer review process (exact details vary of course) but historically it was strictly confidential. Only over the past perhaps 2 decades has a process where reviews are made public begun to gain mainstream support.
WhatsTheBigIdea 1 hours ago [-]
A positive step! Bravo Nature!
To little, to late? Maybe.
Surely very little, surprisingly late.
There are so many great reasons to distrust "The Science"... perhaps the greatest is that without powerful and persistent skepticism, science simply isn't science. Questioning and skepticism is profoundly fundamental to the scientific process. "Trust the science" or worse yet "Believe the science" are statements that are about as destructive to the foundation of science as one can get.
The whole point of science was that everyone has access to the true nature of the universe which we can discover through theorizing and experimentation. Empirical experience is the great equalizer that puts the Arch Duke of Where-and-What or the Highest of High Priests the on the same footing as, say, Joseph Preistly or Benjamin Franklyn. Questioning and skepticism must be accepted from all quarters. To say that only the most select, distinguished and credentialed can be admitted to the discussion smacks of priesthood and aristocracy.
The argument that we need to re-persuade the people to trust in science is missing the point by such a wide margin as to be a symptom of the problem.
It is more or less the same as fretting that people aren't going to church so much any more.
The science should not be trusted. The science should be tested. Those who say "Trust the science" have completely lost the thread.
Imagine saying "no" to a researcher with a big social media profile. Imagine 4chan coming at you with style-detection and deanonymization tools simply because their favorite racist or antivaxer got their nonsense rejected and sent their followers after you. And this is not just me feeling this way - quoting myself from a previous comment, and according to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents" and "support for public review inversely correlated with reviewing experience".
A measure that women ~~and inexperienced researchers~~[2] do not support is a measure that favors only those who are already part of the club.
[1] Original here (currently offline): http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019..., summary here: https://www.aclweb.org/adminwiki/images/f/f5/ACL_Reviewing_S...
[2] This part has been correctly pointed out as being wrong.
"The identity of the reviewers will remain anonymous, unless they choose otherwise — as happens now."
(Also "support for public review [being] inversely correlated with reviewing experience" means inexperienced reviewers are more likely to support it. Not less.)
As for the anonymous part, that's why I wrote "with style-detection and deanonymization tools". If the Internet could find Shia Labeouf's flag in a day [1], could they find a reviewer based on their writing?
[1] https://www.dailydot.com/unclick/4chan-shia-labeouf-secret-l...
Or are you just for creating classes of people that just can't be critiqued in any circumstance?
This kind of sounds like, 'Wont anyone think of the grifters?!'
Not all academia is Elsevier.
[1] This policy has been altered recently, though, and now submitting a paper comes with reviewing duties.
The ACL leadership and senior members of the field are very much aware of this and are trying their best (ARR being an attempt to improve the situation, but I am unsure how much better it really is compared to the old system of conference reviewing now that we are a few years in). But there appear to be no easy fixes for a complicated, distributed system such as peer review. Every discussion I have with said leadership and other senior members always ends with us agreeing on the problems and likewise agreeing that despite considerable mental effort we are failing to come up with solutions.
Returning to the main topic. Nature is worthy of praise for making their peer review transparent and I say this as a massive Nature critic. It is a move I loved seeing from NeurIPS (then NIPS) and ICLR over a decade ago, as it helps younger researchers see what good (and bad) communication looks like and that even papers they know now are greatly appreciated received a fair amount of criticism (sometimes unwarranted). I have argued for ACL to introduce the same thing for nearly a decade at this point, but we still do not and I have never heard a solid argument as to why not (best argument was the technological effort, but OpenReview, with all its flaws, makes this even easier than with Softconf; not that it would have been that hard with Softconf either).
This is desperately needed as AI could further deteriorate the quality of science if the publishing process is not made more strict. This represents a significant step forward in rigorous science. I hope other publications follow suit.
I suspect the number is low. If that's the case, they're unlikely to be more convinced by the presence of published peer review, either.
Those that distrust authority as a whole and lean into conspiracies cannot be saved with this kind of thing.
But i think news and science are having similar perception issues recently.
Distrust for news growing among the average population (for some good reasons). People are loosing faith in the objectivity of established media organisations. Most people are exposed to science through these traditional news.
So adding back some sense of confidence and authority to scientific institutions is very valuable to non-academics. Even if they themselves would not read the papers or revews.
Ignoring the problem is not going to fix it, and in fact continuing to regard these people as beneath you is only going to accelerate the downfall of this system.
Of course it does, billion-dollar interests who have a vested interest in attacking it have a well-funded propaganda arm, and as we've been discovering over the past century - angry, loud, and frequently repeated bullshit with a profit motive floods any signal out of the room.
What's really sad is people who have legitimate concerns or desires for improvement hitching their horse-cart to the former.
No amount of peer review or replication is going to convince someone whose fortune is built on peddling snake oil.
> Ignoring the problem is not going to fix it,
Here's a solution. Hold the people making these attacks to the same level of rigor as what they are attacking. Stop giving proven liars a loudspeaker. The day the press will do that is the day some meaningful progress to fix that damage may be made.
Until you do that, we're going to continue getting shit outcomes.
Why should anyone trust science? Skepticism should always be the default position. Putting it on a pedestal to be worshipped is what led us all into this mess. If science needs trust to work, then whatever it is doing is something I'd like to see fail.
When people talk about those who distrust science, they aren't referring to the carefully sceptical. They're talking about people who come to a conclusion and then reject any evidence against it.
I'd actually be curious what fraction of his supporters share his views on vaccines. It strikes me as more likely that they like one of his random, idiosyncratic views and are willing to excuse the anti-vaccine nonsense to get those policy outcomes.
There's that word again. No works are trustworthy. You shouldn't trust, instead verify. Be skeptical. I'm more concerned with which of their works can be verified.
>An ironic stance considering that it's the inverse of the blind worship
I'm not sure there's any difference worth mentioning between blind worship and cuddly warm trusting that you want to do. One may seem a little less zealous, but it's the same pig just with extra lipstick.
That’s a red herring. No scientist actually says that. What we say is that on some subjects the evidence is overwhelming and to overcome the current understanding you need compelling evidence and theories, not screeches about bias and liberal elites.
No. We are not going to take seriously someone’s pet theory about a new perpetual motion machine, or cold fusion, or lack of global warming, or the ineffectiveness of vaccines, or anything that is contrary to massive amounts of accumulated evidence.
Skepticism is not anti-scientific. Hell, distrusting results and theories is not anti-scientific. Distrusting the scientific method is. There is a difference.
> Putting it on a pedestal to be worshipped is what led us all into this mess.
Scientists did not ask for this. Amongst all high profile politicians, those who whine about science becoming political are those who made it so, by taking contrarian positions to rile up their voter base. Most people who make this point are not being honest. If you are, you should make specific arguments rather than rehash propaganda.
One might think that to be the case, but science is in a hurry to save our planet, and it has no patience for your skepticism. It may already be too late.
>Distrusting the scientific method is.
Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn. And maybe when it's finally proven that, maybe I go on distrusting it... and make it prove it some more. If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven". But so little do I ever hear of that sentiment. Like in this very thread.
>Scientists did not ask for this.
Probably not. I'd be skeptical of the theory that they asked for it. But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask. And liking it, they started behaving in ways that encouraged it just the same. And those scientists who enjoyed it the most elbowed their way to the front to egg it on even more. And so that's where we are now. I mean, I'm a little skeptical that it happened that way, and I'd welcome evidence that disproves it.
>Most people who make this point are not being honest.
It's really sort of hilarious how it's only my side that can ever be honest. Not the other guys'. And this is true no matter who "me" is. None of your opponents, for instance, can ever be genuine... they're always trying to cheat. And if your side tries to cheat, well, it's with the best of intentions because the stakes are so high. Not that you would cheat, only the other side ever does that.
>f you are, you should make specific arguments rather than rehash propaganda.
If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
As a matter of fact, it is already too late for some of the consequences. The permafrost is not going to re-freeze, for example, and wildfires are already significantly on the rise.
The thing is, I don’t believe this because science, I know this for a fact because it was proven. It’s like nihilistic edgelords giving cynicism a bad name. Skepticism is not the rejection of facts because they are inconvenient. It is not stubbornness in the face of evidence, either.
> Maybe I don't trust it either. Maybe because I'm distrustful, the scientific method should prove, over and over again, that it's worth a damn.
Right, but then to prove itself it needs to be evaluated and assessed. You need a kind of method to do so, including a standard of proof and the basics of reasoning and logic. This framework is itself the scientific method. If you start from the axiom that it does not work, then the only conclusion is that it does not work, because how would you prove otherwise?
> If one truly though the scientific method worthy of any respect, such a person might say "that's ok, because it will go on re-proving what has already been proven".
Well, yeah. That’s what we do. Though the standards shift over time in the face of overwhelming evidence. We can not start from first principles every time, it just does not work and there are theories we can rely on. But when holes are found in these theories we try to figure out what the problem is and how to build better ones. The scientific method itself has been refined over centuries and the standards now are far from what Newton used. A common example is the evolution of what is considered to be a proof in Physics.
> But it doesn't seem altogether implausible that they might have enjoyed it once it happened even though they didn't ask.
I am sure some of them do, and not necessarily the best ones. But most of us are not attracted by public engagement and just want to do our science in our lab. We did not ask to become pawns in sectarian struggles. We rang the alarm because, honestly, when the house is on fire you have to do something and not doing it would mean some degree of responsibility in the outcome.
> It's really sort of hilarious how it's only my side that can ever be honest.
I am not following. I am not talking about sides. Regrettable attitudes towards science are not confined in one political or social tribe and there are examples in history of both right- and left-leaning ideologies becoming anti-science. But yeah, making a topic political in order to then claim that it is political is blatantly dishonest. It’s a cheap rhetorical trick with lasting consequences. What is political is how you address the situation, not whether reality is real.
> If we can't start here, then there can be no real conversation. Basically you're just asking me to surrender and admit that I'm the bad guy trying to cheat and swindle, everything I say is a lie, and you're the innocent victims. Haha.
You are reading way too much. First, I am not attacking you. Then, a good starting point is facts we can observe and things we can infer from them instead of the a priori position that reality is not real and the other side is just making stuff up.
Your posting doesn't give me the impression that you're very familiar with "science".
Maybe structural engineers feel safer after their Master's when they traverse a bridge, but I bet that's more the exception than the rule.
[1] https://www.theregister.com/2020/04/02/boeing_787_power_cycl...
That is very unhelpful, to say the least. The amount of noise has increased, but it does not mean that the scientists who know their subject disappeared. They are still around and not any less bright than their predecessors were 30 years ago.
The thing is, peer review is not a stamp of quality, and never was. It is just the basic level of due diligence. The referees cannot reproduce the results most of the time for a lot of very good reasons. They are here as a sanity check, to ensure that the work avoided common pitfalls and actually makes sense.
What most people do not understand is that articles are not good because they are peer-reviewed; it’s the lack of peer review that is a red flag. Amongst reviewed articles, a lot of them will turn out to be wrong or flawed in ways that are impossible for the reviewers to find out.
I highly doubt it's a meaningful factor in public trust.
Maybe people could learn about what peer review is before posting their strong feelings about it? The purpose certainly isn't to replicate people's experiments, that happens after publication and not by referees. One of a reviewer's duties is to look at whether the study could be replicated given the included information. That is a very different thing.
Also, just because something has made it past peer review also doesn't mean it isn't controversial in the field.
This is my academic profile https://scholar.google.com/citations?user=bxn78bkAAAAJ&hl=en
This is a misunderstanding of the role of peer review. The point is not to prove that a paper is correct, the point is to ensure a minimum level of quality. You are entirely right that most reviewers cannot hope to reproduce the results presented, and very often for very good reasons. If I write 3 proposals over the course of a year to get some beam line on a neutron source, it is completely unrealistic to expect a referee to have the same level of commitment.
I think this hints at a more profound problem, which is that a lot of studies are not replicated. This is where the robustness of a scientific result comes from: anybody can make the same observation and reach the same conclusions under the same conditions. This is the real test, not whether you convinced 3 or 5 referees.
The real value of an article is not in whether it was peer reviewed (though the absence of peer review is a red flag). Instead, it is in whether different people confirmed its main results over the years that follow publication.
There is a massive incentive to publish. Inflate the value, inflate the results, and stretch out projects to multiple smaller papers, fake results to make it seem important. This is lazy and fast, and can be caught by a stricter review and scrutiny.
Papers that are properly done all the way through, but with faked data meant to push an agenda, can be disproven by counter research.
Genuine mistakes, logical errors, and other oversights are even more common than that. For all the issues it has, peer review is quite good at catching the things that it's intended to catch.
Research gets cited constantly in public debates and is used for policy decisions, so the public should be able to quickly separate the good from the bad, the "maybe this is true" from the "this is empirically proven."
The public has lost a lot of trust in Science because research papers have been used to push political agendas, which can then never be questioned because doing so means arguing with a supposed peer-reviewed scientific consensus.
Replication and meta-analysis are an important part of this.
Most scientists are in fact very conservative with how they claim their results - less so university PR departments and “study shows” clickbaiters.
Instead I got PIs happy to say that weak evidence "proved" their theory and to try suppress evidence that negatively impacted "fundablity". The most successful scientists I worked with were the ones who always talked like a PR puff piece.
It is not, and it cannot be. It is unrealistic to expect a referee working in their free time to confirm studies that often cost millions of dollars. This is a fundamental misunderstanding of what peer review is and why it is useful in popular or heavy vulgarised science.
Politicians, journalists, and university press offices are guilty of this, and they are those abusing peer review to give some studies more weight than they deserve.
The public has lost trust in science because 10 to 30% of it is scientifically illiterate [1]. (Tens of millions of American adults are literally illiterate [2].)
That's what lets activists and politicians cherrypick bad science that supports their position or cast a scientific consensus as unquestionable.
[1] https://carsey.unh.edu/publication/conspiracy-vs-science-sur...
[2] https://nces.ed.gov/pubs2019/2019179/index.asp
Obviously reproducing results as part of peer review is not a workable (or even coherent) solution. I don't pretend to have any idea what a solution might be. The obvious issue is that academic publications were never intended as political tools and should not be made into that.
On several occasions I've had interactions with laymen where I found myself thinking "if only you hadn't had access to pubmed and way too much motivation we'd both be better off right now" yet I firmly believe that free and open access to information is a huge net benefit to society on the whole.
I hope that making things transparent will help reduce the situation where big labs have an easier time getting their work into high impact journals through relationships with the editor.
I would suggest to Nature that this phrase hints at much larger a problem than showing authors arguing with Reviewer 2.
Alas, scale ruins everything. Nevertheless, I strongly believe “science is friendship”
Although it could raise the standard of the process itself (methodology, writing) to very high levels, it restricted innovative ideas or unpopular outcomes.
If it works better, it works better. The problem with the society method is it didn't work better than a decentralised scientific system.
I’m having trouble finding a downside besides vote buying or voting rings, now that which way one has voted is now attributable. Can you think of any risks under the new system?
To little, to late? Maybe.
Surely very little, surprisingly late.
There are so many great reasons to distrust "The Science"... perhaps the greatest is that without powerful and persistent skepticism, science simply isn't science. Questioning and skepticism is profoundly fundamental to the scientific process. "Trust the science" or worse yet "Believe the science" are statements that are about as destructive to the foundation of science as one can get.
The whole point of science was that everyone has access to the true nature of the universe which we can discover through theorizing and experimentation. Empirical experience is the great equalizer that puts the Arch Duke of Where-and-What or the Highest of High Priests the on the same footing as, say, Joseph Preistly or Benjamin Franklyn. Questioning and skepticism must be accepted from all quarters. To say that only the most select, distinguished and credentialed can be admitted to the discussion smacks of priesthood and aristocracy.
The argument that we need to re-persuade the people to trust in science is missing the point by such a wide margin as to be a symptom of the problem.
It is more or less the same as fretting that people aren't going to church so much any more.
The science should not be trusted. The science should be tested. Those who say "Trust the science" have completely lost the thread.