Spotting bad science - guide

[Altair]

Spotting bad science.

Fullsize image: https://s3-us-west-1.amazonaws.com/h...e_fullsize.jpg



Previously:

"Flat Earth" photo, and why it's so hard to debunk pseudoscience
NASA renaming cosmic objects for woke movement - image
Physicist risking life to demonstrate laws of physics - GIF
Richard Feynman on the uncertainty of knowledge - video
Climate activist cements hand to pavement - GIF

[mklotz]

Good stuff but it might be easier to just assume

Anything in the media that alludes in some way to science is worthless filler written by journalism majors under the influence of mind altering drugs.

A worthwhile corollary technique is to assume that any media advertisement is a lie.

[Jon]

A couple of quick points:

-Peer review is overrated. The difference between a peer reviewed paper and a paper that is not peer reviewed is often just a few scientists giving their opinion. These days you can take the url of a paper from a preprint server and copy-paste it into Twitter search, and you will find plenty of no-holds-barred review on it from scientists in its field.



-Top tier medical journals are overrated. Especially these days, they have political positions that greatly influence their choices of papers to publish. Large journals often have a history of various scandalous false science that they published and was accepted by the public based on their authority.

-Review studies are overrated, although they are usually a better indicator of good science than peer review or publication in a top journal. There are so many studies on any topic that you can find all of the poorly run ones, and publish a review of them that falsely lends generalized credibility to bad science.

-Mass media uses the concepts of peer review, top tier journals, and review studies as a form of the logical fallacy Appeal to authority. Peer review and journals did not exist at the foundations of the scientific method. At its core, science just means that anyone can arrive at any conclusion that they like, and that they publish the steps that led them to that conclusion. If other people follow those same or similar steps and arrive at the same conclusion, then that science becomes more widely accepted. Instead of calling it "science", it could just be called "repeatability".

-What mklotz said.



To add some nuance to the bit about media and science, here's a clip from a popular US television show from today. Here they discuss the connection between COVID-19 infection and cardiac events. To be fair, this connection is reasonably well documented at this point. And yes, COVID vaccines also cause cardiac problems, but at much lower prevalence and in the case of myocarditis, the existing data we have suggests that it resolves quicker than COVID-induced myocarditis.

Anyway, here's the news tidbit:



-This is based on a study that came out in September of 2022. There's nothing wrong with this study. It's not the only one to reach these conclusions, and long-term effects of COVID have been formally published for years now. The media pretends that this is hot news that they just discovered: "doctors and researchers are only now beginning to chart the longer term health effects of COVID-19".

-One of the co-authors on the study is featured for only 26 seconds of the 7-minute clip. She is far more circumspect than the remainder of the report, using phrasing like "seems to", "appears to" and "increase the likelihood of".

-Black females are very trendy in America these days, so this report focuses on one of them who suffered cardiac problems. Of course she is also a basketball player. More time in this clip is spent on showing her playing basketball, talking about basketball, and posing with a basketball, than speaking with the study co-author.

-The report is loaded with visually catchy EKG lines and emergency vehicles.

-A TV-friendly "Dr. John" (a real doctor, but no connection to this study) approves the science.

This is what the media does with a study that is actually reasonable!

The darker side of media and science, and the support for ignoring all mention of science in media, is found in something called VNRs or Video News Releases. Corporations (often drug companies) film video press releases. Just like newspapers slightly modify press releases to create articles, TV stations take these video news releases and slightly modify them before placing them on news shows. This saves the TV stations money, because most of the work is already done. A good example of this with side-by-side videos of a produced VNR and the news version is here: https://www.prwatch.org/fakenews/vnr18

[CharlesWaugh]

Don't forget:
Don't forget:
• 'Science by Press Release'
• Moving the endpoints of a study.
• Odd inclusion criteria
• Odd exclusion criteria
• Testing againt placebo (not other, existing drug)
• Running multiple trials and burying all but a few (only reporting prositive results)
• Journals selectively publishing SHOCKING results, not basic science
• Journals getting paid (that is, charging) outrageously for reprints of even 'yawn' results
(because they are positive results and who actually reads those reprints that get handed out to docs?)
• Replication studies lacking (or nonexistent) in every discipline, esp. the 'soft sciences'
• Replication studies not getting pulished because they aren't 'cutting edge'
• FDA, USDA, etc. regulators moving directly into industry after their careers and getting fat consulting or 'research advisor' contracts
• Journals with high impact scores Articles written by CRO's (Contract Research Organizations) and rubber-stamped by academic 'authorities' (in pharma research)

And, the FDA's budget for Fiscal Year 2021 is [was]:
approximately $6.1 billion. provided by federal budget authorization (54%, or $3.3 billion)
and user industry fees (46%, or $2.8 billion). (almost HALF from those they are supposed to regulate)
https://www.fda.gov/about-fda/fda-basics/fact-sheet-fda-glance#:~:text=FDA's%20budget%20for%20Fiscal%20Year,%25%2C%20or%20%242.8%20 billion).

[IntheGroove]

But if it's on the internet it must be true...

[mklotz]

Kills 97% of germs on surface == I doubt that any germ count on any surface can be done with 1% accuracy.

5 horsepower shop vacuum that plugs into 120 volt, 15 amp standard home electrical outlet.

Makes-you-feel-good cloth shopping bags filled with supermarket products almost all of which are wrapped in plastic.

Copper infused clothing that relieves pain endorsed by numerous sports stars that have only recently been taught to speak.

Every month a "new" age-defying miracle drug with a scientific-sounding made-up name is discovered in Europe and promptly incorporated in a small jar that sells for more than the same amount of heroin.

It's an established fact that no children's toys are made that are not "educational".

Me cynical? You betcha!

[IntheGroove]

My favorite is "Virtually"...

[mklotz]

Call "Cheatam and Howe" law firm now to discover what your case is WORTH.

[Jon]

It's ironic that the importance of a "free press" is drilled into society by - guess who - the press. While the ability to speak freely is crucial, what the press does with their freedom suggests that they don't deserve it. Or at very least they certainly don't deserve extra protections for journalists that allow them greater leeway than ordinary citizens.

The new AI trend has an interesting faux science bent. Very faux, actually. ChatGPT has this nasty habit of completely fabricating references to scientific papers. In artificial language terminology, this is known as "hallucinating": https://en.wikipedia.org/wiki/Halluc..._intelligence)

More details on this in a Hacker News discussion here: https://news.ycombinator.com/item?id=33841672

And here is a Twitter thread in which a physicist queries ChatGPT about a topic on which she wrote her doctoral dissertation: https://twitter.com/paniterka_ch/sta...93718214901760 . ChatGPT fabricated convincing-looking scientific papers out of thin air, complete with fake DOIs.

[neilbourjaily]

The Journal of Irreproducible Results