AI Tackles Yale/Stanford Junk Science |634|
AI stumbles on data analysis but eventually gets it right.
Podcast: Play in new window | Download
Subscribe: RSS
Ready for another AI truth engine experiment? Last time, we tackled AI logic, and the Pi (the LLM from Inflection) demonstrated how AI could apply logic in instances where even trusted experts fall short.
In Skeptiko 634, I tasked AI with some mildly complex data analysis of a published scientific paper. AI stumbled mightily but eventually show the potential we’re all hoping for.
The Experiment
My methodology was straightforward:
- I selected a published scientific paper: the Bangladesh mask study from the journal Science.
- I copied and pasted the results section into various AI language models (LLMs).
- I asked the AIs to interpret the results and, through inductive reasoning, generate numbers not immediately available in the abstract but calculable with a scientific background.
The Results
The performance of the AI models varied significantly:
- Gemini: Not surprisingly, was the worst performer. Google even accused me of being dishonest. A claim it later withdrew after its “mistake” was revealed.
- Perplexity: The star of the show, generating correct answers with minimal prompting.
- Other LLMs: Most struggled initially but showed potential for improvement.
Interestingly, while the numerical analysis proved challenging for many models, all AIs demonstrated an ability to identify the “Junk Science” issues in the study’s methodology. They correctly pointed out that the study’s conclusions rested on a difference of just 16 individuals in a population of 340,000 – a detail that raises questions about the strength of the findings.
What This Means
This experiment highlights several key points:
- AI’s potential in scientific data analysis is promising but not yet fully realized.
- Different AI models have vastly different capabilities, even when tackling the same task.
- AI can be a valuable tool in identifying potential weaknesses in scientific methodologies.
As we continue to develop and refine AI technologies, experiments like this one offer valuable insights into their current capabilities. It’s clear that AI has the potential to revolutionize how we analyze and interpret scientific data.
Youtube: https://youtu.be/fO9Idq5gLLY
Rumble: https://rumble.com/v59crxe-ai-tackles-yalestanford-junk-science-634.html
[box]
More From Skeptiko
Andrew Paquette: Rigged! Mathematical Patterns Reveal Election Database Manipulation |646|
Painstaking analysis of algorithms designed to manage and obscure elections. In Skeptiko episode 646 Dr...Toby Walsh: AI Ethics and the Quest for Unbiased Truth |645|
The tension between AI safety and truth-seeking isn’t what you think it is. In Skeptiko..Alex Gomez-Marin: Science Has Died, Can We Resurrect it? |644|
Consciousness, precognition, near-death experience, and the future of science Dr. Alex Gomez-Marin is a world-class..Bernardo Kastrup on AI Consciousness |643|
Consciousness, AI, and the future of science: A spirited debate If we really are on..The Spiritual Journey of Compromise and Doubt |642|
Insights from Howard Storm In the realm of near-death experiences (NDEs) and Christianity, few voices..Why Humans Suck at AI? |641|
Craig Smith from the Eye on AI Podcast Human bias and illogical thinking allows AI to..How AI is Humanizing Work |640|
Dan Tuchin uses AI to enrich the workplace. How AI is Humanizing Work Forget about..Christof Koch, Damn White Crows! |639|
Renowned neuroscientist tackled by NDE science. Artificial General Intelligence (AGI) is sidestepping the consciousness elephant..AI Ethics is About Truth… Or Maybe Not |638|
Ben Byford, Machine Ethics Podcast Another week in AI and more droning on about how..