MC02 - attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Media Cloud: Measuring Attention\n",
    "================================\n",
    "\n",
    "At this point you should be ready to query Media Cloud for data. You can use boolean query syntax - [read our query guide](https://mediacloud.org/support/query-guide) for more details about the exact syntax (it runs an [ElasticSearch search](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html) under the hood). **This notebook demonstrates how to quickly measure attention paid to an issue by the media**.\n",
    "\n",
    "Studying media attention is critical for understanding how much readers are exposed to an issue, and has a long tradition. Media Cloud supports investigating attention within individual sources we track, or within collections or sources. We have wide global coverage with both national-level and regional/state-level collections for most countries. You can [browse our geographic collections](https://search.mediacloud.org/collections/news/geographic) to see more.\n",
    "\n",
    "Our Python API exposes two methods that are particularly helpful for studying attention: \n",
    "\n",
    "* `story_count`: return the total number of stories in our database matching your query\n",
    "* `story_count_over_time`: return the total number of stories in our database matching your query _by day_\n",
    "* `story_list`: page through the actual stories that match your query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up your API key and import needed things\n",
    "import os, mediacloud.api\n",
    "from importlib.metadata import version\n",
    "from dotenv import load_dotenv\n",
    "import datetime as dt\n",
    "from IPython.display import JSON\n",
    "import bokeh.io\n",
    "bokeh.io.reset_output()\n",
    "bokeh.io.output_notebook()\n",
    "MC_API_KEY = 'MY_API_KEY'\n",
    "search_api = mediacloud.api.SearchApi(MC_API_KEY)\n",
    "f'Using Media Cloud python client v{version(\"mediacloud\")}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention from a Single Media Source\n",
    "You can start by looking at attention from a single media source to a topic you are interested in. We have almost a million media sources in our system, but only about 100,000 of them are ones that we regularly collect stories from, via RSS feeds or more recently from their sitemaps. You can get the internal id number for any source by searching for it in our [Directory](https://search.mediacloud.org/directory) and noting the ID number just under the large title on that page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check how many stories include the phrase \"climate change\" in the Washington Post (media id #2)\n",
    "my_query = '\"climate change\"' # note the double quotes used to indicate use of the whole phrase\n",
    "start_date = dt.date(2023, 11, 1)\n",
    "end_date = dt.date(2023, 12,1)\n",
    "sources = [2]\n",
    "search_api.story_count(my_query, start_date, end_date, source_ids=sources)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can see this count by day as well\n",
    "results = search_api.story_count_over_time(my_query, start_date, end_date, source_ids=sources)\n",
    "JSON(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and you can chart attention over time with some simple notebook work (using Bokeh here)\n",
    "import pandas as pd\n",
    "from bokeh.plotting import figure, show\n",
    "from bokeh.models import ColumnDataSource\n",
    "df = pd.DataFrame.from_dict(results)\n",
    "df['date']= pd.to_datetime(df['date'])\n",
    "source = ColumnDataSource(df)\n",
    "p = figure(x_axis_type=\"datetime\", width=900, height=250)\n",
    "p.line(x='date', y='count', line_width=2, source=source)  # your could use `ratio` instead of `count` to see normalized attention\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing within a Source\n",
    "\n",
    "Looking at absolute attention at the story level is intriguing, but you probably want to normalize this in some way to support comparisons between sources. To do this, we typically compare attention to the total number of stories we have from a source within that same timespan. That's why the `story_count` endpoint returns two numbers. `relevant` is the number of stories that matched your query.\n",
    " * `relevant` is the number that matched your search with all the conditions\n",
    " * `total` is the number that matched your search without the query terms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = search_api.story_count(my_query, start_date, end_date)\n",
    "source_ratio = results['relevant'] / results['total']\n",
    "'{:.2%} of the Washington Post stories were about \"climate change\"'.format(source_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Research Within a Country - using collections\n",
    "\n",
    "[We have wide global coverage](https://sources.mediacloud.org/#/collections/country-and-state), with sources published in a country grouped into collections. For many of these countries we also have collections of media sources published in the various states and provinces. Lets compare the source-level attention to country-level attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check in our collection of country-level US National media sources\n",
    "my_query = '\"climate change\"'\n",
    "US_NATIONAL_COLLECTION = 34412234\n",
    "results = search_api.story_count(my_query, start_date, end_date, collection_ids=[US_NATIONAL_COLLECTION])\n",
    "us_country_ratio = results['relevant'] / results['total']\n",
    "'{:.2%} of stories from national-level US media sources mentioneded \"climate change\"'.format(us_country_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we can compare this to the source-level coverage\n",
    "coverage_ratio = 1 / (source_ratio / us_country_ratio)\n",
    "'\"climate change\" received {:.2} times less coverage in WashPo than you might expect based on other US national papers'.format(coverage_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or compare to another country (India in this case)\n",
    "INDIA_NATIONAL = 34412118\n",
    "results = search_api.story_count('\"climate change\"', start_date, end_date, collection_ids=[INDIA_NATIONAL])\n",
    "india_country_ratio = results['relevant'] / results['total']\n",
    "'{:.2%} of stories from national-level Indian media sources in 2019 mentioned \"climate change\"'.format(india_country_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coverage_ratio =  1 / (india_country_ratio / us_country_ratio)\n",
    "'at the national level \"climate change\" is covered {:.2} times less in India than the US'.format(coverage_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing Stories\n",
    "\n",
    "Story counts are fine, but often what you really want is the story themselves. Note that **we cannot provide story content** due to copyright restrictions. However, you can get a list of all the URLs and then fetch them yourself. We can also return word counts down to the story level (see the \"language\" notebook for more info on that)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# grab the most recent stories about this issue\n",
    "stories, _ = search_api.story_list(my_query, start_date, end_date)\n",
    "stories[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to list ALL the stories matching, you need to page through the results. This is accomplished via the `pagination_token` parameter. This code snippet pages through all the stories in a query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's fetch all the stories matching our query on one day\n",
    "all_stories = []\n",
    "more_stories = True\n",
    "pagination_token = None\n",
    "while more_stories:\n",
    "    page, pagination_token = search_api.story_list(my_query, dt.date(2023,11,29), dt.date(2023,11,30),\n",
    "                                                   collection_ids=[US_NATIONAL_COLLECTION],\n",
    "                                                   pagination_token=pagination_token)\n",
    "    all_stories += page\n",
    "    more_stories = pagination_token is not None\n",
    "len(all_stories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you may have noted, this can take a while for long time periods. If you look closely you'll notice that it can't be easily parallelized, because it requires content in the results to make the next call. A workaround is to divide you query up by time and query in parallel for something like each day. This can speed up the response. Also **just contact us directly if you are trying to do larger data dumps, or hit up against your API quota**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a CSV of Story Data\n",
    "\n",
    "What you probably want is a csv of all this story data. Here's a quick exmaple of dumping that data to a CSV (like our Search tool does)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "fieldnames = ['id', 'publish_date', 'title', 'url', 'language', 'media_name', 'media_url', 'indexed_date']\n",
    "with open('story-list.csv', 'w', newline='') as csvfile:\n",
    "    writer = csv.DictWriter(csvfile, fieldnames=fieldnames, extrasaction='ignore')\n",
    "    writer.writeheader()\n",
    "    for s in all_stories:\n",
    "        writer.writerow(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and let's make sure it worked by checking out by loading it up as a pandas DataFrame\n",
    "import pandas\n",
    "df = pandas.read_csv('story-list.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Top Media Sources\n",
    "\n",
    "Attention within a collection is useful to know, and to compare across collections. We also offer the ability to see the\n",
    "media that had the most stories matching your search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List media producing the most stories matching the search\n",
    "INDIA_NATIONAL = 34412118\n",
    "results = search_api.sources('\"climate change\"', start_date, end_date, collection_ids=[INDIA_NATIONAL])\n",
    "JSON(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}