Researchers at Rutgers University have found a major flaw in the way that algorithms designed to detect « fake news » evaluate the credibility of online news stories.
Researchers at Rutgers University have found a major flaw in the way that algorithms designed to detect « fake news » evaluate the credibility of online news stories.
Most of these algorithms rely on a credibility score for the « source » of the article, rather than assessing the credibility of each individual article, the researchers said.
« It is not the case that all news articles published by sources labeled ‘credible’ (e.g., The New York Times) are accurate, nor is it the case that every article published by sources labeled ‘non-credible’ publications are ‘fake news,' » said Vivek K. Singh, an associate professor at the Rutgers School of Communication and Information and co-author of the study « Misinformation Detection Algorithms and Fairness Across Political Ideologies: The Impact of Article Level Labeling, » published on OSFHome.
« Our analysis shows that labeling articles for misinformation based on the source is as bad an idea as just flipping a coin and assigning true/false labels to news stories, » added Lauren Feldman, an associate professor of journalism and media studies at the School of Communication and Information, who is another co-author of the paper.
The researchers found using source-level labels for credibility isn’t a reliable method, with article-level labels matching 51% of the time.
