Consensus and Credibility: What makes a legitimate source?
Intro
When you’re trying to communicate information, I find it pretty much comes down to:
- Who is your audience?
- What are you trying to say?
As I officially step into the role of social science researcher, the ability to tailor my words and approach becomes an increasingly important skill. This skill comes up whether I am submitting a proposal for committee review or standing around the kitchen island shooting the breeze with a cousin-once-removed. In both scenarios, I don’t want to undermine my legitimacy, but I want to present information that they understand, which is also compelling and relevant.
I want to make a point, I want them to agree, and I want it to be true.
Evaluating Source Legitimacy
I would describe my general taste in podcasts as “Wikipedia article.”
Ologies is a reputable science communication podcast that spotlights researchers. In a semi-structured interview with the scientists, Alie Ward discusses the state of the field, recent developments, and everyday applicability; in the latter half of the episodes, there are featured listener questions for the researcher to answer. Alie will occasionally have asides where she provides context, which is where she mentioned a factoid about Chicago.
In an episode about Conversational Psychology, the host dropped that Chicago is in the top 10 cities in the world for friendliness. Born-and-raised, I was thrilled to find Chicago superiority had scientific backing. Smugly, I went to track down the article so I could parade it, spreading pro-Chicago propaganda to the masses. (Yes, I may be biased.)
The first warning bell went off when I saw the casual tabloid-esque presentation of the article. But I was quickly reassured by the 18,500 participant sample. This reassurance waned when I started running the numbers in my head. Assuming the “top 20” listed cities are the only cities they polled, that’s 925 respondents per city on average. Not quite as great as I thought, but perfectly legitimate. Then another sentence implies that there were at least 50 cities polled, so this drops the average to 370 respondents… that’s suddenly a lot less convincing. I don’t like that the sample size is not stated accurately upfront. It feels like they’re using their total N as a shield against the reality of their numbers.
My primary methodological gripe is that they were interviewing locals. I know that doesn’t invalidate the entire premise, but it kind of did for me. My intention was to have some kind of back-up when I tell outsiders (e.g. people I meet abroad) about how friendly Chicago is, but what use is it to them if it’s only local opinions? Ironically, it’s knowing that I’m a “local opinion” that made me want to cite a source, since I knew my perspective was skewed. What if locals are friendly among themselves, but not so much towards others?
For example, multiple cities from the United Arab Emirates (UAE) are on the top 20 according to this list. It is illegal to be gay in the UAE, so friendliest cities for whom? If there can be legal repercussions for simply living your life, then it seems misleading to label the city as “friendly and welcoming.” I’m not sure I can in good conscience cite a list that puts Brisbane, AUS below Abu Dhabi, UAE.
Obviously my own confirmation bias led me down this rabbit-hole. I’m self aware enough to not blindly accept and spread statistics I find personally pleasing. However, I had to go through several layers of analysis before I concluded that the statement was pretty much trash.
Discussion
More than a random fact, the assertion about Chicago’s top-ranking friendliness was couched in apparent reputation and legitimacy. A well-known and usually accurate podcast, featuring an expert, had stated it and cited their source on their website. How many people are repeating things from places they trust, who may have decided to be a bit lax in their sources for that one sentence? And how many people actually care about the scientific rigor behind their statement?
In casual conversation, I prefer to delay fact checking for as long as possible. I think it’s good for socialization to speculate without being overly concerned with accuracy. There’s an aspect of this present in creative processes and study design, where you need to come up with your ideas and opinions then realize them (or debunk them) through your medium.
Additionally, why do I need a source to say my hometown is among the best? What is this impulse to back up my words with an external source? In casual conversation, is there really any more weight to my voice just because I can find someone else who agrees with me?
Source-citing is a helpful and useful instinct for academia, but not so much socially. In an academic setting, a contrarian nature or compulsion to fact-check is really beneficial, but it kind of makes you socially unbearable if you aren’t careful. People don’t want to debate structure vs. agency or talk about P-values when they haven’t seen you in months. Honestly, sometimes I don’t either.
I think there was an aspect of this in my attitude that prompted this article. A lot of the people I am meeting are either experts or upcoming experts, by nature of my vocation. It feels lacking to make unsupported assertions, which ignores the usefulness of anecdotes and sometimes misses the point of human connection.
Especially in an all-encompassing programme like a PhD, where you live and breathe your research for years, it can be easy to lose that perspective. Even with my Masters project, I considered my analyses basic and wished I had done more, but my friends* have expressed that it is very technical. Your own standards and purpose can look very different from those that are acceptable.
*(I’m very grateful to my friends who have offered to read my dissertation. It means a lot that they are so curious! Shout-outs to Charlie, Cam, Jenna, and Eliana.)
Takeaway
There’s not much of a conclusion to be had. I would like to explore this topic as a series, since it comes up in so many different ways between politics, community, and academic writing.
In the meantime, be careful about what you state as fact. But it’s still fine (and sometimes even good!) to enjoy things that aren’t perfectly accurate.
Leave a Reply