6.4 Evaluating Sources

Evaluating Sources

Sometimes, the first source that appears in our search results is great. It’s relevant and recent, and it’s written by someone who really knows what they are talking about. Much more often, however, the first source is not actually useful or reliable. And sometimes it can be hard to tell. Learning to evaluate your sources is an important part of being a critical thinker.

So how can you tell the difference between a good source and a not-so-good source?

You may have heard of the CRAAP test for evaluating sources: currency, reliability, authority, accuracy, and purpose. This system, developed by Molly Beestrum, asks you to consider how recent your source is (and how much that matters), whether the source is overly biased or lacking citations of its own, the credentials of the author, and what the source is trying to accomplish. These are all excellent areas for investigation.

This system is really useful, but we follow Patricia Lynne’s suggestion to rearrange the order. She views “authority” as the most important criterion for evaluating the credibility of a source. If the source doesn’t have authority, she suggests moving on to another source. If the information is sound, another source with more authority will also discuss those ideas. And if you only find the ideas in one source, consider that a red flag.

Authority

The concept of having authority on a topic has a problematic history. In academia, this has primarily meant that an author has an academic background and their work has undergone peer review. However, as we know from discussing the different types of knowledge, cultural, spiritual, and experiential knowledge can often be as valuable as mainstream and institutional knowledge. Moreover, an increasing number of scholars are sharing their work in spaces beyond peer-reviewed publications, and there are journalists, documentarians, and social media content creators who employ rigorous methods to create and disseminate knowledge. That said, there is still a need to ensure that whoever is providing a source is genuinely authoritative on a topic and uses methods that hold up under scrutiny. Here is a list of Authority red flags to keep in mind as you evaluate sources:

• You can’t find an author and the organization is questionable.
• The author has credentials, but their credentials aren’t relevant to the material they are writing about.
• The author claims to have experience on a topic or be a member of the community affected by the issue, but you cannot find credible evidence of that experience or membership in that community.

Since we all do so much of our research online, it makes sense to use websites as an example of how to evaluate authority. However, the tips in this example can be useful for evaluating sources in other formats. And remember, the more you take these steps, the faster it goes because always examining your sources becomes second nature.

A Source’s Neighborhood

To understand this concept and begin to use it, imagine that sources are part of communities. Considering the neighborhood in which a source is located can help you determine whether the source is credible and suitable for your purpose.

For a website, you can determine its neighborhood by checking pages labeled About Us, About This Site, Mission, Site Index, and Site Map, if available. (If such pages or similarly labeled ones don’t exist, it may be a sign that the site may be less trustworthy.) Ask yourself these questions to gather clues that will help you decide what neighborhood you’re in:

• Is the site selling products and/or services (even if there are articles and other useful information, too)? Perhaps it’s a retail, service center, or corporate site.
• Are there membership applications and requests for contributions of money or time anywhere on the site? They’re usually a sign that you’re on a site that promotes particular ideas or behavior – in other words, they’re in the advocacy neighborhood.
• Do postings, articles, reports, and/or policy papers give a one-sided view or multiple views on issues, people, and events? If they’re one-sided, the site is likely a commercial site or part of an advocacy group. If the information is balanced and includes different perspectives on an issue, the site is more likely to be on the side of the library, museum, school, or mainstream U.S. news. Sites there usually provide information designed to educate rather than persuade. Newspapers, whether online or in print, usually have editorial pages, but labeling opinions as such helps keep mainline U.S. news sources in the newsstand and out of advocacy.

To determine the neighborhood of other sources, you’ll follow similar methods. You can look at things like the introduction and conclusion, the table of contents, information about the publisher or periodical, and the author’s background, to discern if the author intended to use the source to inform/educate, persuade, sell, or entertain. Is the author’s purpose suitable for your purpose? For instance, does the fact that a resource was intended to persuade mean it can’t help you answer your research question?

If your source does not name the author, it might still be reliable, especially if you have confidence in the publisher. However, most credible and reliable publications tell readers who wrote the articles they contain. It’s important to ask yourself:

What are the author’s qualifications?

• Do they seem to be an expert in the field?
• Have they written about this topic before?
• Are there other experiences that seem to uniquely qualify them as a reliable and credible source on this topic?

Many academic publications provide detailed information about their authors, including their degrees and academic training, the institution where they work (if they are a college professor or instructor), and other publications they have authored in the past. Popular sources tend to include less information about their writers, though they too will often indicate in a byline (where the writer’s name is listed in a magazine or newspaper article) if the writer is a reporter, contributing editor, or editor for a particular subject, and the byline may link to more information about the author.

Credible internet sources will also describe the qualifications of the source’s author or authors. If you can find an author’s name on a website but you can’t find anything about their qualifications on their research subject, you should be suspicious about what that research has to say.

Where did you come across the author?

After you have conducted a bit of research on your topic, you might find yourself coming across the same authors writing similar articles in different publications. You may also find various publications referencing the author or their work, which suggests that the author is indeed reliable and credible in their field. After all, if other articles and writers refer positively to a particular writer or their articles again and again, then it seems likely that the often-referred-to writer is credible.

Are you familiar with the publication?

If you are a new researcher in a particular field of study, this can be a challenging question to answer, as you may not be familiar with some of the more well-known and credible publications in that field. But once you get to know the field better (which will inevitably be the case as you conduct more research on your topic), chances are you will begin to realize certain publications are seen by experts in the field as more credible than others.

Currency

Although not an exclusive metric, research conducted more recently is often considered more reliable because it is presumably up to date on current issues in the field. That said, you don’t need to automatically eliminate research that isn’t recent; if multiple sources cite a specific older source, that suggests it is important to the field. When reviewing for currency, you can look out for these red flags:

• With static publications, the source was published before recent events relevant to your topic.
• With dynamic media (e.g., websites, news, and social media channels), the source has not been updated recently or appears to be unmaintained and/or out-of-date.
• The sources referenced are not current.

When was it written?

The date of publication can dramatically affect the credibility of your research. Obviously, this is especially important for date-sensitive research topics. If you were writing a research project about the internet, chances are any research older than about 1990 or so would be of limited use since the internet as we understand it did not exist before 1990.

But other potentially less obvious topics of research have date-sensitive components. For example, if you were doing research on cigarette smoking or drunk driving, you would have to be careful about evaluating the credibility of research from the 1970s or 1960s or earlier since cultural “norms” in the United States for both smoking and drinking have changed a great deal.

Knowing (or rather, not knowing) the date of publication of a piece of research is another factor to consider when evaluating the credibility of sources. For example, many websites do not include any information about the date of publication or the date when the page was last updated. This means that you have no way of knowing when the information on that dateless page was published. Again, just because the date of publication or update is missing from a website does not automatically discount it as a credible source; however, it should raise your suspicions.

Relevance and Intended Audience

It should be fairly obvious that sources should be relevant to your topic or your intended audience. However, many students think that every source needs to be about their exact topic. That is not the case—a source’s relevance should be evaluated for its usefulness for your project as a whole, including things like context/background, methodology or theoretical approach, or application of your results/conclusions. Even so, there are still some Relevance red flags to watch for; just use your best judgment as you use them to evaluate your sources:

• The information isn’t really related to your topic or the way you plan to discuss your topic, even if the information itself seems good.
• The information is presented in a simplistic manner or ignores complexities that you are aware of.
• The source is written for an audience different from your intended audience.

Evaluating for Relevancy

Relevant sources are those that support your answer to your research question. Instead of thinking you have to read all of every source in order to figure out whether it is relevant, read or skim only parts of each source. If you’re looking at the right parts, that should provide you with enough information to make an informed decision about relevancy.

But what should you look for as you read and skim? One way to figure this out is to first parse your research question so that you can identify its main concepts. (This is like identifying main concepts in your research question in order to search precisely.)

For instance, suppose your research question is: How does having diverse members in a group increase the critical thinking of the group? What are this question’s main concepts? Our answer is: group diversity and critical thinking. So when trying to judge which sources are relevant to these main concepts, you would assess whether each source you’ve found pertains to at least one of these concepts.

Accuracy

Knowledge is typically constructed by aggregating information from multiple sources. Your sources should have turned to other sources to develop their information, so it’s important to verify that they are using sources and that they are accurately representing the ideas from those sources. Watch out for these red flags:

• You can only find the information in this one source or, if it is corroborated elsewhere, it has just been copied and pasted.
• The source lacks citations.
• The source has links, but these links only direct users to pages on the same site or a few select sites.
• When you check the citations, they don’t seem to align with what the source claims.

Purpose and Objectivity

Sources are composed for a variety of reasons and with different degrees of objectivity. The author taking a position or trying to persuade the reader, however, is not what makes a source unreliable. Instead, we need to examine how an author grounds their position in the evidence, situates their argument within the existing conversation on the topic, and argues their position to create a response in their audience. Therefore, you should watch out for these purpose and objectivity red flags:

• The source has an identifiable conflict of interest. In particular, examine any advertising that might be present or funding behind the research. Would the revenue stream interfere with objectivity and balance in the presentation of information?
• The source is trying to make people angry or arouse other strong emotions.
• The source provides arguments on only one side of an issue or presents opposing arguments unfairly or in too simplistic a fashion.
• The source uses fallacious reasoning when presenting arguments.

What is the writer’s motivation?

Is the writer identified with a particular organization or group that might have a specific interest in the subject of the writing?

This can often be the source of conscious or unconscious bias. An obvious example: a writer who is identified as a member of the National Rifle Association, which represents a variety of Americans particularly interested in protecting the right to own guns, will certainly have a different view on gun ownership than a member of The Center to Prevent Handgun Violence, an organization working to enact gun control legislation.

You need to be particularly careful with internet-based sources of research, especially social media, when considering the writer’s affiliation with different groups or organizations. There have been numerous incidents where websites or social media posts falsely claimed they were affiliated with particular groups or causes.

Does the writer have a political bias?

Considering a creator’s politics is particularly important when thinking about the credibility of an internet-based or social media source. Besides the ease with which a writer can misrepresent themselves or others, the low cost and wide reach of the internet and social media has also made it an attractive forum for hate groups, terrorists, and other “fringe” political movements. This doesn’t automatically mean the information you find on reactionary or radical sites or accounts is wrong; however, writers with particularly strong and extreme politics may present information that is biased to the point of inaccuracy.

Of course, while it is important to consider why a writer wrote about their subject and to think about how their motivations influence their writing, having a particular bias or motivation doesn’t automatically lead to a lack of credibility or reliability.