How do journalists gauge the impact of their work? More importantly, does their work have a positive impact on people’s perceptions of important societal issues such as climate change? Well people have to read the article for a start but let’s assume people do move past the title for now. You can look at comments, page views, discussions on a topic to assess a given article’s impact. However, it is very hard to say whether an article, or group of articles, were responsible for those changes observed. This is where an interesting new study by Dr Gary King at the University of Harvard comes in. Published in Science, the study attempted something rather impressive: a randomised controlled trial of journalism.
Randomised controlled trials (or RCTs) are typically run in medicine to ascertain whether a particular treatment has an effect on a studied illness. They have the advantage of being less biased than if researchers were to assign people to either the treatment or control group based on a whim. The randomisation process reduces the impact the experimenter, or clinician, may have on the efficacy of the treatment in question. This is exactly the approach this study took to understand whether writing on one of 11 key policy topics (race, climate, abortion, etc.) stimulated conversations on these topics. However, instead of a treatment they looked at whether the impact of writing an article on a societal issue (treatment) could lead to an increase in conversations about this topic on social media.
The researchers recruited 48 media outlets, of generally small size, to take part in their study to assess the impact of journalism on public conversations. To deal with journalists trying to ‘scoop’ one another and get the best story out there the quickest they used a process known as pack journalism. This also had the advantage of maximising the chance people would be speaking about their policy area on social media. Pack journalism is where multiple media outlets will collaborate and share sources and even quotes to ensure a story is reported and, importantly for political stories, reported in the same way. This is the approach the experimenters took, and they organised their recruited media outlets into packs of 2-5. Each ‘pack’ wrote about the same policy area that was assigned based on their expertise and confidence in the chosen area.
The researchers then took two consecutively quiet news weeks. They randomised each pack to write about the specific policy area on one week and to produce their normal content on the other. This allowed the researchers to compare Twitter activity surrounding the chosen policy area (e.g. climate change) on both weeks. This was not a quick experiment to set up. As the authors state in the paper, it took almost five years to source, organise and oversee the news outlets involved.
When all packs had been randomised and completed their two-week experiment, the researchers assessed the number of Twitter posts across the two weeks. This, broadly speaking, allowed them to assess the impact of the policy-relevant article. Upon examining all of the packs together, it was found that in the first day after publication of the policy article the number of Twitter posts on this area increased by about 20% compared to the control condition. Over the course of the following week the increase in posts on the policy area was just over 10%. These effects are not big but, unlike previous estimates of the impact of journalism, we can be more convinced by their providence. It’s important to note that the media outlets included were small. Therefore, it shouldn’t be too surprising that the effects on Twitter would also be small as presumably these outlets have limited impact on global conversations. It’s also important to note that particularly quiet news weeks were chosen and, although this was practical, it likely influenced the effect.
To assess what the impact of a heavy-hitting media outlet, the researchers examined the change in Twitter posting following a story by the New York Times about fracking influencing drinking water. In this instance, although considerably less controlled, they found that the first day following the article publication, there was a 300% increase in Twitter posts concerning water quality and related topics. This suggests that with larger institutions the impact would be more impressive.
One question you may still be asking is whether this ‘intervention’ had an impact on people’s opinions. When the researchers compared the views expressed when the article was published, compared to service as usual, they found that there was a 2.3% shift in opinions to those expressed in the article. Now this is tiny, but it would be interesting to see whether larger effects could be found in heavy-hitting media outlets. Whether such a study is even possible remains to be seen.
The most exciting thing about this study is that it was possible to carry out an experimental, and relatively well-controlled, study to assess the impact of media reporting on social media conversations. The effects were small, and it was far from perfect (although impressive given how difficult it must have been to set-up), but it was an excellent proof of concept. Personally, science journalism can feel like it is speaking out to the converted but if research can validate that such articles are getting people talking, discussing, and (heaven forbid) changing their opinions on key area such as climate science then I’m sure many journalists would sleep much sounder.
Hey, look at that, I did manage to fit sleep into this article. Go me!
King, G., Schneer, B., & White, A. (2017). How the news media activate public expression and influence national agendas. Science, 358(6364), 776-780.