Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "partial-firmware",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:22:07.413950Z",
     "start_time": "2021-03-09T14:22:06.819093Z"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import base64\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import gzip\n",
    "from tqdm.auto import tqdm\n",
    "from datetime import datetime\n",
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "pharmaceutical-graham",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:22:08.291856Z",
     "start_time": "2021-03-09T14:22:08.287079Z"
    }
   },
   "outputs": [],
   "source": [
    "def p_date(x: str) -> datetime:\n",
    "    \"\"\"Short hand for datetime.fromisoformat(x)\"\"\"\n",
    "    if isinstance(x, datetime):\n",
    "        return  x\n",
    "    return datetime.fromisoformat(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seven-newspaper",
   "metadata": {},
   "source": [
    "## Events log format\n",
    "Each log object is a dictionary with multiple fields. The ones that are really important and used in this analysis are the following. Anything not docummented is not used for analyisis, and is probably an artifact of how SearchX treats the data, and can be safely ignored. (Feel free to explore, tough)\n",
    "- `event`: Type of the event recorded. Related to clicks and hovers on parts and widgets of the SERP.\n",
    "- `userId`: UniqueID of a user. On the user first interaction, this can be different to the final, due to the behaviour of Prolific user IDs.\n",
    "- `sessionId`: User session ID. basically unused.\n",
    "- `task`: JSON with the details of the user task, under `data` (Topics picked for the pre-test, topic picked with description and others):\n",
    "    - `topics`: List with topics picked for the pre-test. `terms` are the terms used for the VKS test.\n",
    "    - `scores`: VKS score on the pre-test for each topic\n",
    "    - `topic`: Topic picked for the user\n",
    "    - `subtopic` : List of subtopics for the user topic.\n",
    "        - `terms`: Nested hierarchical outline for the topic (base64-formatted)\n",
    "- `meta`: Data specific to the event type:\n",
    "    - `url`: URL related to the event\n",
    "    - `query`: User last issued query\n",
    "    - `serpID`: Unique ID for this SERP\n",
    "    - `session`: Type of experiment the user is currently on (control, qe(AQE_sc), visual(CURATED_sc) or progress(FEEDBACK_sc) )\n",
    "    - `position`: Integer with the position of url (i.e. rank position)\n",
    "- `date`: event timestamp\n",
    "\n",
    "## Query log format\n",
    "Each item in the `all_queries` object is a single submitted query\n",
    "- `userId`: UniqueID of a user. On the user first interaction, this can be different to the final, due to the behaviour of Prolific user IDs.\n",
    "- `sessionId`: User session ID. basically unused.\n",
    "- `query`: User submitted query\n",
    "- `created`: creation timestamp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "invalid-scope",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:22:18.479347Z",
     "start_time": "2021-03-09T14:22:09.230420Z"
    }
   },
   "outputs": [],
   "source": [
    "with gzip.open(\"data/all_logs_clean.json.gz\", 'r') as inf:\n",
    "    all_logs = json.loads(inf.read().decode(\"utf-8\"))\n",
    "with gzip.open(\"data/all_queries_clean.json.gz\", 'r') as inf:\n",
    "    all_queries = json.loads(inf.read().decode(\"utf-8\"))\n",
    "\n",
    "vocab = json.load(open(\"data/vocab.json\")) #VKS vocabulary dictionary\n",
    "users = set([x[\"userId\"] for x in all_logs])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rising-methodology",
   "metadata": {},
   "source": [
    "## Pre_tests\n",
    "Pre testes contains the information of the pre-tests from users (questions asked before user starts searching).\n",
    "\n",
    "These questions are:\n",
    "- degree: What is the highest degree completed?\n",
    "    - 0: High School or lower\n",
    "    - 1: Associate's degree(s) (e.g., AA AE, AFA, AS, ASN)\n",
    "    - 2: Bachelor's degree(s) (e.g., BA, BBA, BFA, BS)\n",
    "    - 3: Master's degree(s) (e.g., MA, MBA, MFA, MS, MSW)\n",
    "    - 4: Specialist degree(s) (e.g., EdS)\n",
    "    - 5: Applied or professional doctorate degree(s) (e.g., MD, DDC, DDS, JD, PharmD)\n",
    "    - 6: Doctorate degree(s) (e.g., EdD, PhD)\n",
    "    - 7: Other\n",
    "- background_degree: For which subject areas do you have a \\<degree answer\\>?\n",
    "- english: Are you an English native speaker?\n",
    "- english-level: What is your level of English?\n",
    "    - 0: Beginner\n",
    "    - 1: Elementary\n",
    "    - 2: Intermediate\n",
    "    - 3: Upper-intermediate\n",
    "    - 4: Advanced\n",
    "    - 5: Proficiency\n",
    "- web-previous: How often do you learn about a scientific topic (see the examples above) by searching the web?\n",
    "- web-information-need: Think about the most recent time you learned about a scientific topic by searching the web. Describe what you were trying to learn.\n",
    "- web-online: What are your preferred online resources (like Wikipedia, Coursera, Youtube etc.) to learn about a scientific topic?\n",
    "- web-offline: What are your preferred offline resources (can be books, people, institutions) to learn about a scientific topic?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "representative-lover",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:23:02.390332Z",
     "start_time": "2021-03-09T14:23:02.265642Z"
    }
   },
   "outputs": [],
   "source": [
    "pre_tests = {\n",
    "    **{\n",
    "        x[\"sessionId\"]: x[\"meta\"]\n",
    "        for x in all_logs if x['event'] == \"SURVEY_REGISTER_RESULTS\"\n",
    "    },\n",
    "    **{\n",
    "        x[\"userId\"]: x[\"meta\"]\n",
    "        for x in all_logs if x['event'] == \"SURVEY_REGISTER_RESULTS\"\n",
    "    },\n",
    "    **{\n",
    "        x[\"meta\"][\"data\"][\"userId\"]: x[\"meta\"]\n",
    "        for x in all_logs if x['event'] == \"SURVEY_REGISTER_RESULTS\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "racial-census",
   "metadata": {},
   "source": [
    "## Load data from Prolific\n",
    "This data comes from Prolific system. We are just interested in their status, if they have returned (finished unsuccessfully) and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "centered-desire",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:23:03.708811Z",
     "start_time": "2021-03-09T14:23:03.699406Z"
    }
   },
   "outputs": [],
   "source": [
    "prolific_path = [f\"data/{x}\" for x in [\"prolific_export_5e4d0dea60c37d14cb329934.csv\", \"prolific_export_5e4e4f7be482212a2429a366.csv\", \"prolific_export_5f46799646aa3002de1afa25.csv\"]]\n",
    "prolific_last_time = dict()\n",
    "session_length_prolific = dict()\n",
    "prolific_status = dict()\n",
    "prolific_users = set()\n",
    "returned_code = dict()\n",
    "found = 0\n",
    "for p in prolific_path:\n",
    "    for line in open(p):\n",
    "        l = line.split(\",\")\n",
    "        if l[0] == \"session_id\":\n",
    "            continue\n",
    "        participant_id = l[1] \n",
    "        prolific_users.add(participant_id)\n",
    "        duration = float(l[5])\n",
    "        status = l[2]\n",
    "        returned_code[participant_id] = l[11]\n",
    "        prolific_status[participant_id] = status\n",
    "        prolific_last_time[participant_id] = l[4]\n",
    "        session_length_prolific[participant_id] = duration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cosmetic-cooperation",
   "metadata": {},
   "source": [
    "## Metrics\n",
    "These functions compute specifict metrics for analysis. They may be inter-dependent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "every-edmonton",
   "metadata": {},
   "source": [
    "### Learning Gain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "alpine-connectivity",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:23:08.379294Z",
     "start_time": "2021-03-09T14:23:08.371374Z"
    }
   },
   "outputs": [],
   "source": [
    "def ALG(user_logs: list[dict], topic: str) -> float:\n",
    "    \"\"\"Computes Absolute Learning Gains:\n",
    "    Args:\n",
    "        user_logs: sorted (by time) list with all of one user's logs\n",
    "        topic: string with user's topic id\n",
    "    \"\"\"\n",
    "    score = 0\n",
    "    topic_terms = get_topic_terms(topic)\n",
    "    pretest_results = [\n",
    "        r for r in user_logs if r[\"event\"] == \"SURVEY_PRE_TEST_RESULTS\"\n",
    "    ][0]\n",
    "    posttest_results = [\n",
    "        r for r in user_logs if r[\"event\"] == \"SURVEY_POST_TEST_RESULTS\"\n",
    "    ][0]\n",
    "    score_mapping = {1: 0, 2: 0, 3: 1, 4: 2}\n",
    "    for question in topic_terms:\n",
    "        qid = \"Q-\" + str(topic) + \"-\" + str(question)\n",
    "        pre_score = score_mapping[int(pretest_results[\"meta\"][\"data\"][qid])]\n",
    "        post_score = score_mapping[int(posttest_results[\"meta\"][\"data\"][qid])]\n",
    "        score += max(0, post_score - pre_score)\n",
    "    return score / 10\n",
    "\n",
    "\n",
    "def MLG(user_logs: list[dict], topic: str) -> float:\n",
    "    \"\"\"Computes Maximum possible learning potential\n",
    "    Args:\n",
    "        user_logs: sorted (by time) list with all of one user's logs\n",
    "        topic: string with users' topic id\n",
    "    \"\"\"\n",
    "    topic_terms = get_topic_terms(topic)\n",
    "    pretest_results = [\n",
    "        r for r in user_logs if r[\"event\"] == \"SURVEY_PRE_TEST_RESULTS\"\n",
    "    ][0]\n",
    "    score = 0\n",
    "    score_mapping = {1: 0, 2: 0, 3: 1, 4: 2}\n",
    "    for question in topic_terms:\n",
    "        qid = \"Q-\" + str(topic) + \"-\" + str(question)\n",
    "        pre_score = score_mapping[int(pretest_results[\"meta\"][\"data\"][qid])]\n",
    "        score += 2 - pre_score\n",
    "    return score / 10\n",
    "\n",
    "\n",
    "def RPL(user_logs: list, topic: str) -> float:\n",
    "    \"\"\"Computes user's Realized Potential Learning:\n",
    "    Args:\n",
    "        user_logs: List of one user's logs\n",
    "        topic: string with user's topic id \n",
    "    \"\"\"\n",
    "    sorted_logs = sorted(user_logs, key=lambda x: x[\"date\"], reverse=True)\n",
    "    _ALG = ALG(sorted_logs, topic)\n",
    "    _MLG = MLG(sorted_logs, topic)\n",
    "    return _ALG / _MLG"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "potential-forest",
   "metadata": {},
   "source": [
    "### Other metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "equipped-campbell",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:23:10.144115Z",
     "start_time": "2021-03-09T14:23:10.115257Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_all_logs(user: str) -> list[dict]:\n",
    "    \"\"\"Get all logs from one user\"\"\"\n",
    "    user_logs = [x for x in all_logs if x[\"userId\"] == user]\n",
    "    return user_logs\n",
    "\n",
    "\n",
    "def get_topic_title(user_logs: list[dict]) -> str:\n",
    "    \"\"\"Gets the title of the topic assigned to user\"\"\"\n",
    "    return [x for x in user_logs if x[\"event\"] == \"SURVEY_POST_TEST_RESULTS\"\n",
    "            ][0]['task']['data']['topic']['title']\n",
    "\n",
    "\n",
    "def get_topic_id(user_logs: list[dict]) -> str:\n",
    "    \"\"\"Gets the id of the topic the user was assigned to\"\"\"\n",
    "    for x in user_logs:\n",
    "        if x[\"event\"] == \"SEARCHRESULT_VIEW_URL\":\n",
    "            return x[\"task\"][\"data\"][\"topic\"][\"id\"]\n",
    "\n",
    "\n",
    "def get_topic_terms(p: str) -> list[str]:\n",
    "    \"\"\"Gets a list of the terms for the VKS for a given topic id\"\"\"\n",
    "    return vocab[p][\"terms\"]\n",
    "\n",
    "\n",
    "def get_both_topics(user_log: list[dict]) -> list[str]:\n",
    "    \"\"\"Gets both topics assigned to user during pre-test\"\"\"\n",
    "    return [\n",
    "        x[\"title\"] for x in [\n",
    "            x for x in user_log if x[\"event\"] == \"SURVEY_PRE_TEST_RESULTS\"\n",
    "        ][0][\"task\"][\"data\"][\"topics\"] if x[\"title\"] != \"Sports\"\n",
    "    ]\n",
    "\n",
    "\n",
    "def get_user_education(user_log):\n",
    "    sessionId = user_log[0][\"sessionId\"]\n",
    "    userId = user_log[0][\"userId\"]\n",
    "    if userId in pre_tests:\n",
    "        value = pre_tests[userId][\"data\"][\"degree\"]\n",
    "        mapping = {\n",
    "            0: \"High School\",\n",
    "            1: \"Associate's degree(s)\",\n",
    "            2: \"Bachelor's degree(s)\",\n",
    "            3: \"Master's degree(s)\",\n",
    "            4: \"Specialist degree(s)\",\n",
    "            5: \"Applied or professional doctorate degree(s)\",\n",
    "            6: \"Doctorate degree\",\n",
    "            7: \"Other\"\n",
    "        }\n",
    "        return mapping[value]\n",
    "    return \"Unknown\"\n",
    "\n",
    "\n",
    "def get_queries(user: str) -> list[str]:\n",
    "    \"\"\"Get all queries a user submitted\"\"\"\n",
    "    u_queries = [x[\"query\"] for x in all_queries if x[\"userId\"] == user]\n",
    "    return u_queries\n",
    "\n",
    "\n",
    "def get_session_duration(user_logs: list[dict]) -> int:\n",
    "    \"\"\"Gets how long, in seconds, a user session lasted, from start to end\"\"\"\n",
    "    sorted_logs = sorted(user_logs, key=lambda x: x[\"date\"], reverse=False)\n",
    "    end_time = p_date(sorted_logs[-1][\"date\"])\n",
    "    start_time = p_date(sorted_logs[0][\"date\"])\n",
    "    return (end_time - start_time).seconds\n",
    "\n",
    "\n",
    "def get_user_average_dwell_time(user_log: list[dict]) -> int:\n",
    "    \"\"\"Gets the user average dwell time in seconds (How long they spend one one document)\"\"\"\n",
    "    documents_events = sorted([\n",
    "        x\n",
    "        for x in user_log if x[\"event\"] in [\"DOCUMENT_CLOSE\", \"DOCUMENT_LOAD\"]\n",
    "    ],\n",
    "                              key=lambda x: x[\"date\"],\n",
    "                              reverse=False)\n",
    "    urls = {x[\"meta\"][\"url\"] for x in documents_events}\n",
    "    dwell_times = []\n",
    "    for url in urls:\n",
    "        doc_events = [x for x in documents_events if x[\"meta\"][\"url\"] == url]\n",
    "        for i in range(0, len(doc_events), 2):\n",
    "            if i + 1 == len(doc_events):\n",
    "                continue\n",
    "            if doc_events[i][\"event\"] == \"DOCUMENT_LOAD\" and doc_events[\n",
    "                    i + 1][\"event\"] == \"DOCUMENT_CLOSE\":\n",
    "                start_time = p_date(doc_events[i + 1][\"date\"])\n",
    "                end_time = p_date(doc_events[i][\"date\"])\n",
    "                dwell_times.append(start_time - end_time)\n",
    "    if len(dwell_times) == 0:\n",
    "        return 0\n",
    "    return np.mean(dwell_times).seconds\n",
    "\n",
    "\n",
    "def get_session(user_logs: list[dict]) -> str:\n",
    "    \"\"\"Get what type of experiment the user was in\"\"\"\n",
    "    event = [x for x in user_logs if x[\"event\"] == \"SEARCHRESULT_VIEW_URL\"][0]\n",
    "    return event[\"meta\"][\"session\"]\n",
    "\n",
    "\n",
    "def get_views(user_logs: list[dict]) -> list[dict]:\n",
    "    \"\"\"Gets all snippets the user saw\"\"\"\n",
    "    return [x for x in user_logs if x[\"event\"] == \"SEARCHRESULT_VIEW_URL\"]\n",
    "\n",
    "\n",
    "def get_clicks(user_logs: list[dict]) -> list[dict]:\n",
    "    \"\"\"Gets all document click events\"\"\"\n",
    "    return [x for x in user_logs if x[\"event\"] == \"SEARCHRESULT_CLICK_URL\"]\n",
    "\n",
    "\n",
    "def get_unique_docs(user_clicks: list[dict]) -> list[dict]:\n",
    "    \"\"\"Gets all unique docs ther user clicked\n",
    "    Args:\n",
    "        user_clicks: List of all SEARCHRESULT_CLICK_URL events (call get_clicks() for this)\n",
    "    \"\"\"\n",
    "    return len(set([x['meta']['url'] for x in user_clicks]))\n",
    "\n",
    "\n",
    "def get_search_session_time(user_log: list[dict]) -> int:\n",
    "    \"\"\"Gets how long, in seconds, between first document open and last document close\"\"\"\n",
    "    li = [x['event'] for x in user_log]\n",
    "    for i in range(len(li)):\n",
    "        if li[i] == \"SEARCH_QUERY\":\n",
    "            index = i\n",
    "            first_query_time = p_date(user_log[index][\"date\"])\n",
    "            break\n",
    "    #get last document_close\n",
    "    for i in range(len(li) - 1, -1, -1):\n",
    "        if li[i] == 'DOCUMENT_CLOSE':\n",
    "            last_doc_close = p_date(user_log[i][\"date\"])\n",
    "            break\n",
    "    delta = last_doc_close - first_query_time\n",
    "    return delta.seconds\n",
    "\n",
    "\n",
    "def get_answer_changes(user_logs: list[dict]) -> tuple[int]:\n",
    "    \"\"\"Gets how many changes of user knowledge happenned on the post-test\n",
    "    Args:\n",
    "        user_logs: List with all user interactions\n",
    "    Returns:\n",
    "        one_to_four: Change from knowledge level 1 to 4\n",
    "        one_to_three: Change from knowledge level 1 to 3\n",
    "        three_to_four: Change from knowledge level 3 to 4\n",
    "        one_remaining: Stayed on level 1 (no knowledge improvement)\n",
    "        other: Other changes (knowledge decreased [??])\n",
    "        valids: valids questions (sanity check)\n",
    "    \"\"\"\n",
    "    topic = get_topic_id(user_logs)\n",
    "    terms = get_topic_terms(topic)\n",
    "    one_to_four = 0\n",
    "    one_to_three = 0\n",
    "    three_to_four = 0\n",
    "    one_remaining = 0\n",
    "    other = 0\n",
    "    valids = 0\n",
    "    for r in user_logs:\n",
    "        if r[\"event\"] == \"SURVEY_PRE_TEST_RESULTS\":\n",
    "            pretest_results = r\n",
    "            break\n",
    "    for r in user_logs:\n",
    "        if r[\"event\"] == \"SURVEY_POST_TEST_RESULTS\":\n",
    "            posttest_results = r\n",
    "            break\n",
    "    for question in terms:\n",
    "        qid = \"Q-\" + str(topic) + \"-\" + str(question)\n",
    "        pre = int(pretest_results[\"meta\"][\"data\"][qid])\n",
    "        post = int(posttest_results[\"meta\"][\"data\"][qid])\n",
    "        if (pre == 1 or pre == 2) and post == 4:\n",
    "            one_to_four += 1\n",
    "        elif (pre == 1 or pre == 2) and post == 3:\n",
    "            one_to_three += 1\n",
    "        elif pre == 3 and post == 4:\n",
    "            three_to_four += 1\n",
    "        elif (pre == 1 or pre == 2):\n",
    "            one_remaining += 1\n",
    "            valids -= 1\n",
    "        else:\n",
    "            other += 1\n",
    "    return (one_to_four, one_to_three, three_to_four, one_remaining, other)\n",
    "\n",
    "\n",
    "def get_average_query_term_coverage(user_queries: list[str],\n",
    "                                    user_log: list[dict]) -> float:\n",
    "    \"\"\"Gets how many query terms, on average, the user used\"\"\"\n",
    "    topic_terms = get_topic_subtopic_terms(user_log)\n",
    "    user_queries_terms = [set(x.lower().split()) for x in user_queries]\n",
    "    query_coverage = []\n",
    "    for query in user_queries_terms:\n",
    "        query_coverage.append(\n",
    "            len(query.intersection(topic_terms)) / len(query))\n",
    "    return np.mean(query_coverage)\n",
    "\n",
    "\n",
    "def get_subtopics(user_log: list[dict]) -> str:\n",
    "    \"\"\"Gets all subtopics for a given user topic\"\"\"\n",
    "    data = [x for x in user_log if x[\"event\"] == \"SURVEY_POST_TEST_RESULTS\"\n",
    "            ][0][\"task\"][\"data\"]['subtopic'][\"terms\"]\n",
    "    return base64.b64decode(data).decode(\"utf-8\")\n",
    "\n",
    "\n",
    "def get_subtopic_terms(user_log: list[dict]) -> list[str]:\n",
    "    \"\"\"Gets all terms for subtopics\"\"\"\n",
    "    subtopics = [\n",
    "        x for x in user_log if x[\"event\"] == \"SURVEY_POST_TEST_RESULTS\"\n",
    "    ][0][\"task\"][\"data\"]['subtopic'][\"terms\"]\n",
    "    return subtopics\n",
    "\n",
    "\n",
    "def get_topic_subtopic_terms(user_log: list[dict]) -> set[str]:\n",
    "    \"\"\"Gets all topic and subtopic titles, and made then into a set\"\"\"\n",
    "    subtopics = eval(get_subtopics(user_log))\n",
    "    topic = get_topic_title(user_log)\n",
    "    terms = \" \".join(subtopics.keys()).lower().split()\n",
    "    for i in subtopics.values():\n",
    "        for j in i:\n",
    "            terms += j.lower().split()\n",
    "    return set(terms)\n",
    "\n",
    "\n",
    "def get_number_of_bookmars(user_log: list[dict]) -> int:\n",
    "    \"\"\"`Gets how many bookmarks the user created\"\"\"\n",
    "    return len([x for x in user_log if x[\"event\"] == \"BOOKMARK_ACTION\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "olympic-falls",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:23:18.081286Z",
     "start_time": "2021-03-09T14:23:11.849646Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fdd750c0028e4aa8859c7f208f7d188f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/127 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(127, 20)\n"
     ]
    }
   ],
   "source": [
    "dataset = []\n",
    "\n",
    "for count, user in tqdm(enumerate(users), total=len(users)):\n",
    "    user_log = get_all_logs(user) # All user events\n",
    "\n",
    "    user_queries = get_queries(user) # All user queries\n",
    "    user_session = get_session(user_log) # What experiment the user was assigned to\n",
    "    topic = get_topic_title(user_log) # User topic\n",
    "    topic_id = get_topic_id(user_log) # User topic id\n",
    "    education_level = get_user_education(user_log) # User education level\n",
    "    \n",
    "    session_duration_in_minutes = get_session_duration(user_log)//60 \n",
    "    realized_learning_gain = RPL(user_log, topic_id)\n",
    "    absolute_learning_gain = ALG(user_log, topic_id)\n",
    "    user_url_views = get_views(user_log) # All snippets seen\n",
    "    number_of_views = len(user_url_views) # Number of snippets seen\n",
    "\n",
    "    user_url_clicks = get_clicks(user_log) # All documents clicked\n",
    "    number_of_clicks = len(user_url_clicks) # Number of documents clicked\n",
    "    unique_docs = get_unique_docs(user_url_clicks) # Number of unique documents clicked\n",
    "    \n",
    "    number_of_queries = len(user_queries) # Number of issued queries\n",
    "    avg_query_len = np.mean([len(x.split()) for x in user_queries])\n",
    "\n",
    "    search_time = get_search_session_time(user_log) # How long from first document open to last document close\n",
    "    session_duration_in_minutes = get_session_duration(user_log) # How long from start to end of experiment\n",
    "    average_dwell_time = get_user_average_dwell_time(user_log) # Average dwell time on document\n",
    "    \n",
    "    # In-depth analysis\n",
    "    (one_to_four, one_to_three, three_to_four, one_remaining, other) = get_answer_changes(user_log) # Changes from one level to another on post-test\n",
    "    query_term_coverage = get_average_query_term_coverage(user_queries, user_log) # Overlap between query terms and scaffolding terms\n",
    "\n",
    "    number_of_bookmars = get_number_of_bookmars(user_log) # Number of bookmark events\n",
    "    dataset.append({\"user\":user, \n",
    "                    \"absolute_learning_gain\": absolute_learning_gain, \n",
    "                    \"realized_learning_gain\": realized_learning_gain,\n",
    "                    \"type\": user_session, \n",
    "                    \"average_query_len\":avg_query_len,\n",
    "                    \"number_of_queries\": number_of_queries,\n",
    "                    \"number_of_views\": number_of_views,\n",
    "                    \"number_of_clicks\": number_of_clicks,\n",
    "                    \"session_duration\":session_duration_in_minutes,\n",
    "                    \"one_two_to_four\": one_to_four,\n",
    "                    \"one_two_to_three\": one_to_three,\n",
    "                    \"three_to_four\": three_to_four,\n",
    "                    \"one_two_no_change\": one_remaining,\n",
    "                    \"other\": other,\n",
    "                    \"topic\": topic,\n",
    "                    \"average_dwell_time\" : average_dwell_time,\n",
    "                    \"unique_docs\" : unique_docs,\n",
    "                    \"number_of_bookmars\": number_of_bookmars,\n",
    "                    \"education_level\": education_level,\n",
    "                    \"search_time\": search_time\n",
    "                    })\n",
    "df = pd.DataFrame(dataset)\n",
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "restricted-affair",
   "metadata": {},
   "source": [
    "## Plots\n",
    "These are auxiliary functions to plot boxplots for the metrics computed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "quiet-picking",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:34:09.812541Z",
     "start_time": "2021-03-09T14:34:09.808713Z"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.stats import f_oneway, kruskal, ttest_ind, pearsonr\n",
    "from matplotlib.pyplot import figure\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "contrary-stereo",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:35:52.570220Z",
     "start_time": "2021-03-09T14:35:52.563081Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['progress', 'visual', 'control', 'qe']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "order = list(df.groupby(\"type\", sort=False)[\"realized_learning_gain\"].mean().index)\n",
    "order"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "residential-shakespeare",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-09T14:39:52.762884Z",
     "start_time": "2021-03-09T14:39:52.498190Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P-Value control vs qe:\t\t0.36835, Not singnificant\n",
      "P-Value control vs visual:\t0.23638, Not singnificant\n",
      "P-Value control vs progress:\t0.81375, Not singnificant\n",
      "P-Value qe vs visual:\t\t0.71019, Not singnificant\n",
      "P-Value visual vs progress:\t0.19322, Not singnificant\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxEAAAHpCAYAAAD9ObE7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABD8UlEQVR4nO3deZxddX3/8ddnJvsesgAhhIAJm4IoARRRg8oStdXyQ0WpRkWpbSWt1dal1qLSKi5V41L3CtZ9Q0AooLIpIgRZZM8AAcISsofsy3x+f5w7k5thtpPcyb2ZeT0fj3nMPed8zzmfO3Mzue/7Pd/zjcxEkiRJknqrqd4FSJIkSdqzGCIkSZIklWKIkCRJklSKIUKSJElSKYYISZIkSaUYIiRJkiSVMqjeBfSFiRMn5vTp0+tdhiRJkrRHu+WWW5Zl5qSO6/tliJg+fToLFiyodxmSJEnSHi0iHu5svZczSZIkSSrFECFJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJIkSSrFECFJkiSplLqGiIj4dkQ8FRF3drE9ImJ+RLRExB0R8fzdXaMkSZIax7JlyzjnnHNYvnx5vUsZ0OrdE/Ed4NRuts8BZla+zgb+ezfUJEmSpAZ1wQUXcMcdd3DBBRfUu5QBra4hIjOvA1Z00+Q1wIVZuBEYFxH77p7qJEmS1EiWLVvG5ZdfTmZy+eWX2xtRR/XuiejJfsCjVcuLK+skSZI0wFxwwQVkJgCtra32RtRRo4eI6GRddtow4uyIWBARC5YuXdrHZUmSJGl3u+qqq9iyZQsAW7Zs4corr6xzRQNXo4eIxcD+VctTgcc7a5iZX8/MWZk5a9KkSbulOEmSJO0+J510EoMHDwZg8ODBnHzyyXWuaOBq9BBxMfCWyl2aXgCszswn6l2UJEmSdr+5c+cSUVyo0tTUxNy5c+tc0cBV71u8/gD4A3BIRCyOiLMi4l0R8a5Kk8uAB4EW4BvA39WpVEmSJNXZxIkTmTNnDhHBnDlzmDBhQr1LGrAG1fPkmfnGHrYn8Pe7qRxJkiQ1uLlz57Jo0SJ7IeqsriFCkiRJKmPixIl88YtfrHcZA16jj4mQJEmS1GAMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJIkSSrFECFJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJIkSSrFECFJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJLUjy1btoxzzjmH5cuX17sUSf2IIUKSpH7sggsu4I477uCCCy6odymS+hFDhCRJ/dSyZcu4/PLLyUwuv/xyeyMk1YwhQpKkfuqCCy4gMwFobW21N0JSzRgiJEnqp6666iq2bNkCwJYtW7jyyivrXJGk/sIQIUlSP3XSSScxePBgAAYPHszJJ59c54ok9ReGCEmS+qm5c+cSEQA0NTUxd+7cOlckqb8wREiS1E9NnDiROXPmEBHMmTOHCRMm1LskSf3EoHoXIEmS+s7cuXNZtGiRvRCSasoQIUlSPzZx4kS++MUv1rsMSf2MlzNppzkLqiRJ0sBkiNBOcxZUSZKkgckQoZ3iLKiSJEkDlyFCO8VZUCVJkgYuQ4R2irOgSpIkDVyGCO0UZ0GVJEkauAwR2inOgipJkjRwGSK0U5wFVZIkaeBysjntNGdBlSRJGpgMEdppzoIqSZI0MHk5kyRJkqRSDBGSJEmSSjFESJIkSSrFECFJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJIkSSql7iEiIk6NiPsioiUiPtDJ9rERcUlE3B4Rd0XE2+pRpyRJkurv/vvvZ86cObS0tNS7lAGtriEiIpqBLwNzgMOBN0bE4R2a/T1wd2Y+F5gNfDYihuzWQiVJktQQzjvvPNatW8fHPvaxepcyoNW7J+JYoCUzH8zMzcAPgdd0aJPA6IgIYBSwAti6e8uUJElSvd1///0sWrQIgEWLFtkbUUf1DhH7AY9WLS+urKv2JeAw4HHgz8A/ZGbr7ilPkiRJjeK8887bYdneiPqpd4iITtZlh+VTgNuAKcBRwJciYswzDhRxdkQsiIgFS5curXWdkiRJqrO2XoiulrX71DtELAb2r1qeStHjUO1twM+z0AI8BBza8UCZ+fXMnJWZsyZNmtRnBUuSJKk+pk+f3u2ydp96h4ibgZkRcWBlsPQZwMUd2jwCvBwgIvYGDgEe3K1VSpIkqe4+/OEP77D8kY98pE6VqK4hIjO3Au8GrgDuAX6cmXdFxLsi4l2VZh8Hjo+IPwO/Ad6fmcvqU7EkSZLq5eCDD27vfZg+fTozZsyob0EDWGR2HIKw55s1a1YuWLCg3mVIkiSpxu6//37+4R/+gS9+8YuGiN0gIm7JzFkd1w+qRzGSJEnSzjj44IO5/PLL613GgFcqRETEXsDbKeZ3GA80d9IsM/PlNahNkiRJUgPqdYiIiEOBa4BJdH5r1jb97/ooSZIkSe3KDKz+DDAZOB84CBicmU2dfHXWOyFJkiSpnyhzOdOLgV9l5of6qhhJkiRJja9MT0QAd/dVIZIkSZL2DGVCxC0UE71JkiRJGsDKhIiPAa+MiNl9U4okSZKkPUGZMRH7A78EroyIH1D0TKzqrGFmXrjrpUmSJElqRGVCxHcobt8awJsrXx1v5xqVdYYISZIkqZ8qEyLe1mdVSJIkSdpj9DpEZOYFfVmIJEmSpD1DmYHVkiRJkmSIkCRJklROl5czRcSDFIOkX5GZD1WWeyMz81k1qU6SJElSw+luTEQTO959qeNyV2KXKpIkSZLU0LoMEZk5vbtlSZIkSQOTYyIkSZIklWKIkCRJklRKmcnmAIiIocAxwH7A0M7aZKYzVkuSJEn9VKkQERFvBz4FjO+qCcXga0OEJEmS1E/1+nKmiDgV+CbwBPA+isDwS+Bfgasqyz8B3l77MiVJkiQ1ijJjIt4LLAeOz8zPVdbdlpmfzMxTgXcCpwEP1LhGSZIkSQ2kTIh4PnBJZj7d2f6Z+S3g9xQ9E5IkSZL6qTIhYiTFpUxtNgJjOrRZABy3q0VJkiRJalxlQsSTwKSq5SeAQzq0GQs072pRkiRJkhpXmRBxFzuGhuuBl0fEiwEi4jnA6yvtJEmSJPVTZULE5cCLImJKZflTwDbgmohYCtwOjAbOq22JkiRJkhpJmRDxNYoJ5pYBZObdwMspwsUy4EpgTmZeVusiJUmSJDWOXk82l5lbgCUd1t0IvLrWRUmSJElqXGV6IiRJkiSp9z0R1SJiGjC9k02PZuZDu1SRJEmSpIbWbYiIiEEUE8itBU7KzNbKprcBH+lklwcj4rDM3FrbMiVJkiQ1ip56Ik4HjgFOrwoQbQL4XtXyWOBVwF8BP6lZhZIkSZIaSk8h4i8pJpn7RSfbMjPf3LYQEQE8ApyGIUKSJEnqt3oaWH00cG1mZk8HqrS5urKPJEmSpH6qpxAxhaJ3oaPVXax/Eth3V4uSJEmS1Lh6ChGDgY5jIcjMz2fmgZ2031bZR5IkSVI/1VOIWAHsX+J40yr7SJIkSeqnegoRtwMvq9zqtVuVNicCd9SiMEmSJEmNqadw8CvgFOC9wPk9tP0nYG/gkhrUJUnSHmf+/Pm0tLTUu4wdLF68GICpU6fWuZLtZsyYwbx58+pdhqRd0FNPxLeBx4DzIuJjETG6Y4OIGBURHwX+A3i8so8kSWoAGzZsYMOGDfUuQ1I/Ez3dvTUijgcuB0YBG4BbKIJFAvsBs4DhwDrg5My8sS8L7o1Zs2blggUL6l2GJEl11/aJ//z58+tciaQ9UUTckpmzOq7vcaxDZt5QCRJfAl4KvLiTZtcA52TmXbtaqCRJkqTG1mOIAKiEgxMj4iDgRcA+QABPAL/PzAf7rkRJkiRJjaRXIaJNJSwYGCRJkqQBrKeB1ZIkSZK0A0OEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkrp9TwRETGtF81agTWZuWbnS5I0UMyfP5+WlpZ6l9Fu8eLFAEydOrXOlWw3Y8YM5s2bV+8yJEnaQZnJ5hYB2ZuGEbEE+Bnw0cxcthN1SdJut2HDhnqXIEnSHqFMiLgQOAB4KbAKuA1YAuwNHAWMA64B1gFHAH8PvDoijs3MpTWqV1I/0mifsLfVM3/+/DpXIklSYyszJuITwHOBTwL7Z+bLMvONmfkyYH/gU5Xt7wUOAj5KETo+WNuSJUmSJNVTmRDxSeD2zPxQZq6r3pCZ6zLzA8AdwCczszUzP0rRW/EXNatWkiRJUt2VCREvAW7ooc0NFJc7tbkRaJwRipIkSZJ2WZkQMRTYp4c2+1batVkLbC1blCRJkqTGVSZE3A68ISKe09nGiDgSeD3FJUxtpgMOqpYkSZL6kTJ3Z/oY8Cvg5oj4X+D3bL870wnAmcBg4OMAETEcOBm4pJYFS5IkSaqvXoeIzLwiIs4EvgqcBby9anMAq4G3Z+YVlXVDgDcA99WoVkmSJEkNoExPBJn5o4j4FfAa4HnAWGANcCvwy8x8uqrtauCKTg8kSZIkaY9VKkQAZOZa4HuVL0mSJEkDTJmB1ZIkSZJUviciImYBxwLjgeZOmmRmfnxXC5MkSZLUmHodIiJiDPBz4ESKgdRdSSp3aJIkSZLU/5Tpifg08DLgeuB/gEdxIjlJkiRpwCkTIl4D/Ak4MTNb+6geSZIkSQ2uzMDqscDVtQ4QEXFqRNwXES0R8YEu2syOiNsi4q6IuLaW55ckSZJUTpmeiIUUs1PXTEQ0A18GTgIWU8yGfXFm3l3VZhzwFeDUzHwkIibXsgZJkiRJ5ZTpifgy8BcRsV8Nz38s0JKZD2bmZuCHFJdNVXsT8PPMfAQgM5+q4fklSZIklVSmJ+JyioHVv4+IjwK3AKs6a9j2hr8X9qMYoN1mMXBchzYHA4Mj4hpgNPCFzLyw92VLkiRJqqUyIWIRxe1bA/hmN+2yxHE7u1VsdlgeBBwNvBwYDvwhIm7MzPt3OFDE2cDZANOmTevl6SVJkiSVVSZEXMgz3+DvqsXA/lXLU4HHO2mzLDPXAesi4jrgucAOISIzvw58HWDWrFm1rlOSJElSRa9DRGa+tQ/OfzMwMyIOBB4DzqAYA1Htl8CXImIQMITicqfP9UEtkiRJknqhTE9EzWXm1oh4N3AF0Ax8OzPvioh3VbZ/NTPviYj/A+4AWoFvZuad9atakiRJGtjqGiIAMvMy4LIO677aYfnTFDNmS5IkSaqzLkNERHybYgzEhzJzSWW5NzIzz6pJdZIkSZIaTnc9EW+lCBHnA0sqy72RgCFCkiRJ6qe6CxEHVr4/1mFZkiRJ0gDWZYjIzIe7W5YkSZI0MDXVuwBJkiRJe5bSd2eKiGbgEGA8xW1ZnyEzr9vFuiRJkiQ1qFIhIiL+DXgPMLaHpp2GC0mSJEl7vl6HiIj4F+CjwGrgu8CjwNY+qkuSJElSgyrTE/FOijs1PT8zl/ZRPZIkSZIaXJmB1fsDFxkgJEmSpIGtTIhYwk4MxJYkSZLUv5QJET8GToqIoX1VjCRJkqTGVyZEfAR4AvhpRDh7tSRJkjRAlbk86S5gMDAFeGVErAZWddIuM/NZNahNkiRJUgMqEyKaKG7p+kjVuuikXWfrJEmSJPUTvQ4RmTm9D+uQJEmStIfo9ZiIiPhIRLy5L4uRJEmS1PjKDKz+MHBEXxUiSZIkac9QJkQ8Bozpq0IkSZIk7RnKhIhfAK+IiOF9VYwkSZKkxlcmRPw7sBK4KCKe00f1SJIkSWpwZW7xejswBHg+cHtEbASeArJDO+eJkCRJkvqxsvNEbGHHeSLgmfNCOE+EJEmS1I85T4QkSZKkUsqMiZAkSZIkQ4QkSZKkcsqMiQAgIoYCxwD7AUM7a5OZF+5iXZIkSZIaVKkQERFvBz4FjO+qCcXdmgwRkiRJUj/V68uZIuJU4JvAE8D7KALDL4F/Ba6qLP8EeHvty5QkSZLUKMqMiXgvsBw4PjM/V1l3W2Z+MjNPBd4JnAY8UOMaJUmSJDWQMiHi+cAlmfl0Z/tn5reA31P0TEiSJEnqp8qEiJEUlzK12QiM6dBmAXDcrhYlSZIkqXGVCRFPApOqlp8ADunQZizQvKtFSZIkSWpcZe7OdBc7hobrgTMi4sWZeX1EPAd4faWd+sD8+fNpaWmpdxntFi9eDMDUqVPrXMl2M2bMYN68efUuQ5KkfsP3Hz0biO8/yvREXA68KCKmVJY/BWwDromIpcDtwGjgvNqWqEa1YcMGNmzYUO8yJEnSAOL7j8ZQpifiaxS3cF0JkJl3R8TLgQ8Dz6IYD/H5zLyi5lUKoOESbls98+fPr3MlkiSpr/j+Q53pdYjIzC3Akg7rbgReXeuiJEmSJDWuMpczSZIkSVKpy5kAiIgjgTcBhwEjM/MVlfXTgWOBqzJzZS2LlCRJktQ4SoWIiPgY8CG292Bk1eYm4AfAPwJfrEVxkiRJkhpPry9niogzKAZRXwUcBXyientmPkgxuPova1ifJEmSpAZTZkzEPKAFeE1m3gFs7qTNPcDMWhQmSZIkqTGVCRFHAFdkZmfhoc3jwN67VpIkSZKkRlYmRATQ2kObvYGNO1+OJEmSpEZXJkQsBI7vamNENAMnAHftalGSJEmSGleZEPFj4PkR8d4utn8QmAF8f5erkiRJktSwytzi9fPA64BPRcTrqdzeNSI+A7wYmAXcCHy9xjVKkiRJaiC9DhGZuSEiTgS+AJwJNFc2/RPFWIn/Bd6dmVtrXqUkSZKkhlFqsrnMXA28NSL+CTgGmACsBm7KzKV9UJ8kSZKkBlMqRLTJzBXAFR3XR8RHgVdn5tG7WpgkSZKkxlRmYHVvTKOYzVqSJElSP1XrECFJkiSpnzNESJIkSSplp8ZESJLUCObPn09LS0u9y2hoCxcuBGDevHl1rqRxzZgxw5+PVJIhQpK0x2ppaeHWu26FcfWupIG1Ft9ufezW+tbRqFbVuwBpz2SIkCTt2cZB6+zWelehPVTTNV7ZLe2MbkNERPy25PEO3YVaJEmSJO0BeuqJmL0Tx8yd2EeSJEnSHqKnEHHibqlCkiRJ0h6j2xCRmdfurkIkSZIk7RkcTSRJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsXJ5qQBZP78+bS0tNS7jIa1cOFCAObNm1fnShrbjBkz/BlJ0gBniJAGkJaWFu6/809MG7Wt3qU0pCFbis7ZjYturnMljeuRtc31LkGS1AAMEdIAM23UNj48a229y9Ae6rwFo+pdgiSpATgmQpIkSVIpXfZERMRHdvKYmZkf38l9JUmSJDW47i5nOreTdVn1ODpZH5XHhghJkiSpn+rucqYTO/m6BNgGXAi8DZhT+f7dyvpfAi8rU0BEnBoR90VES0R8oJt2x0TEtog4vczxJUmSJNVWlz0RmXlt9XJEvAU4CXhBZv6pQ/MLIuJLwHXAz3t78ohoBr5cOe5i4OaIuDgz7+6k3fnAFb09tiRJkqS+UWZg9XuAH3USIADIzAXAjyvteutYoCUzH8zMzcAPgdd00u4c4GfAUyWOLUmSJKkPlAkRhwBP9NDm8Uq73toPeLRqeXFlXbuI2A/4K+CrJY4rSZIkqY+UCRFrgBf10OYEoMwN6KOTddlh+fPA+zOz29mxIuLsiFgQEQuWLl1aogRJkiRJZZQJEb8CXhwRn4mI0dUbImJ0RHyWImRcUuKYi4H9q5anUvRmVJsF/DAiFgGnA1+JiNd2PFBmfj0zZ2XmrEmTJpUoQZIkSVIZZWas/iAwm2LMwzsi4jZgCbA3cBQwBngQ+FCJY94MzIyIA4HHgDOAN1U3yMwD2x5HxHeASzPzohLnkCRJklRDve6JyMyngGOAb1GEj5cAr6t8HwR8Aziu0q63x9wKvJvirkv3AD/OzLsi4l0R8a5ePwtJkiRJu02ZnggycwVwdkT8HXAoMBZYDdxbCQSlZeZlwGUd1nU6iDoz37oz55AkSZJUO6VCRJtKYLizxrVIkiRJ2gOUDhERMRh4OXAYMCozP15ZP4xiXMSyzGytaZWSJEmSGkaZuzMREacCiyju1PRZ4NyqzUdRzCPxhtqUJkmSJKkR9TpERMQs4CKKeRzeA3y/entm3gg8RDExnCRJkqR+qkxPxL8B64FZmTkfWNhJm5uB59aiMEmSJEmNqUyIeBFwUWY+2U2bR4F9d60kSZIkSY2sTIgYBSzroc2IkseUJEmStIcp84b/MeDZPbQ5imLWakmSJEn9VJkQcTlwSkSc0NnGiJgDHA9cWovCJEmSJDWmMiHiE8Aq4MqIOB84HCAiXlVZ/gnFLV7/q9ZFSpIkSWocvZ5sLjMfi4iTgR8D/1y16WIggAeA0zKzp3ETkiRJkvZgpWaszsw/RcQhwKuAFwITgNXAjcAvM3Nr7UuUJGlg27RiEw9f+jCr7llF6+ZWhk0axr4v3Ze9X7h3t/utW7yORy57hHWPrWPr2q1kazJk3BDGHz6e/U/dn8GjB7e3XfanZSy7ZRlrHlrD1rXFf+eTXzCZGW+a0afPTdKeqVSIAMjMbRS9DxfXvhxJ2jWPrdrMp36zhOta1rJucysHTRjKW4/bizOO3qvb/e56YgOfu/op7n5yI8vXb6W1FfYZM4jZM0fzj7MnM2Hkjn8ur7x3DV+5fin3LNnIkObguANG8i+v2JuDJw/ry6enAWjzms38+XN/ZvPqze3r1j++ngd+8ABb1mxh6ilTu9x3w5INrLxz5Q7rNi3fxJPXP8nqltUc9f6jiKYAYNkty1jx5xV98yQk9Tu9DhER8RZgfWb+tJs2RwJHZeaFtShOksp46uktnPatB3lyzfZO0XuWbOT9Fz/O0rVbOeelk7vc94Flm7jqvqd3WPfIyi1ceNMKbly0jiv+dgZNlTdbP799Je/5+WPt7TZuSa6672luXLSOn7/jIIOEaurRyx5tDxAz3zyTsYeM5d5v3Mvah9fy6P89yqRjJjF0r6Gd7jt0wlBmnDmDsQePZfCowax9ZC33fvNetq7byoYnNrDusXWM2n8UAGMPHsuYGWMYutdQ7vvWfbvt+UnaM5UZWP0d4EcR8eOI6PyvFfwV8D+7XJUk7YTPXf1Ue4D43GlTufl9h/C8qcMB+MK1S3ls1eYu991//BA+89r9+MM/Hcx9Hz6cn779QMaPaAbg/qc2cfeSjQBs2NzKuZc/WdlnML/7x4P53tzpDGkOnt7UynlXdDcfp1ROtibLbi2GGg7feziTjpnEkDFDmHLilGL7tmT5bcu73H/09NFMPm4yQ8cPpWlwE2OeNYYxM8a0b4/maH+870v3ZcqJUxg5dWQfPRtJ/UnZieFWAKcDV0fExD6oR5J2SmtrculdqwGYMWkopz13HJNHD+YdLyz+VG3Zllx295ou93/e1BG87nnjmTJ2CMMGN3HMASM57oDtb6YGV3ohrm55mtUbtgHw17P2Yv/xQzjhoFEcf1DR9voH1rJincPDVBsbl21kW+X1Nnzy8Pb11Y/XLV7Xq2O1bm1lzQNrWLOw+Hcw+qDRjNh3RA2rlTSQlB0T8UWgGfg34A8R8arMvL/2ZUlSOQ+v3Myaja0AHDRhSPv6A6se3/XEhl4da/PWVm57bAM3LirenB0zbQQHTy46YO98fGPVsbd3yh40YSjXLFxLa8LdSzZywkGjdv7JSBVbqwJp87DmTh9vWbulx+Pc9MGbdjjW6INGc9jZhxER3ewlSV3bmYHV/x4RDwDfAG6IiNMy87ralyZJvbdi/bb2x6OGbn+DNbrq8fJ12+jJUeffw8qqYx0zbQTfetMB7W+2Vq7f/kZs1NCmTh8vtydCNZKZXWzYteM+/eDT3PuNe3n2u5+9wyVNktRbZS9nAqAycPpUil6JKyPizJpWJUkldf1ea/uGnfnQ9eZH1vPOHzzM1m1ZOV7P5/ctmWpl8Kjtt2DdtnF7uN22aVunbbpy7CeO5QWffQFHvu9IRuxXXMK05oE13o1J0k4r3RPRJjOvjojjgV8BF0bEQbUrq/7mz59PS0tLvctoaAsXLgRg3rx5da6kcc2YMcOfz24yYeT2Hoe1VW+w1m5qbX+814hmenLb+w9j09ZW7n9qE//yy8e4+8mN/PHh9Vx53xpeefhY9hqx/c9m9bHXba46z8id/tMq7WDYxGE0D29m24ZtbHhq++V41Y97OxC6aXATo6aNYvJxk1n080XFcZb27hI/Sepol/6ny8x7IuI44FLgXKDrW0TsYVpaWrj1z3fTOqL7e8sPZLG5+Oj1lge8G01nmtb7Cd/udMD4IYwZ1sSaja08uHz7XZgeqnr87H2Hd7brMwwd1MQRU4bzuqPG8dH/K17fiyrHec6U7bdvfWj5pvbHD1YeNwUcvre3eFVtRFMw8XkTWXLDEjYs2cDSm5cy9pCxPH7148X25mDC8yYAcMO8GwCYdOwkZv71TAAevvRhRk8bzchpIxk8cjAbntrA0j8ubT/+sAnbX6vbNm2jdUsrW6su2cut2T7mojc9HpIGjl3+uCwzl0bES4HvUdzidRev1GwcrSP2YuPhr653GdpDDbv70nqXMKA0NQWvfvZYvn/LSlqWbuIXd6zihING8c0/FLfHHNwcvOrZxa0tD/j3OwE4/ahxfPaviom6PvXrJRw1dThHThnO+BHNPLBsEz+5bVX78aeNLwZonzhjNGOHN7N6wzb+d8EKXvXssTy8cjM3PFgMwn7xs0bZE6Ga2v+V+7PyrpVsXr2Zhd9duOO2U/dn6Piu7roOK+9ayWNXPtbptpHTRrLXkds/KHvwJw+y9KalO7RZumApSxcU646ff/zOPgVJ/VCZ/+kOBFZ1tiEzN0bE6RS3f/V+cZLq4j0nTua3C5/myTVb+cefLd5h2z+8dBJTxg7pYk+4euHTfPn6pZ1uO3LKcE4+dDQAw4c0ce6cfXjPzx/j0ZVbOOHz229QN3poEx8+ZZ8aPBNpuyFjhnDEe47g4UseZtW9q9i2aRvDJw9n35fuy94v3LvbfScfO5nlty9n49KNbF23lRgcDJ80nL2euxdTZk+hadBODY2UpN6HiMx8uIftCfxklyuSpJ00efRgfn7WQZz/6yVc98Ba1m9u5aAJQ3nrcXtxxtHdX5p4+lHjuOzuNSxavolVG7YxdFATB04YwimHjeGsF0xgSNWbrdOeO55RQ5v5yvVLuWfJRoY0B8ceMJJ/efnezlatPjF0r6EcPPfgbtt01lMw5cQp7RPT9WTmX89svwxKknpin7ukfmW/cUOYf/r+3bZ5+KPPeca6s144kbNe2Ps5NE8+dAwnHzqm54aSJPVDXYaIiHiQYnzDKzLzocpyb2RmPqsm1UmSJElqON31RDSx4yDpjstd8RbpkiRJUj/WZYjIzOndLUuSJKm2nKeqZ85T1bPdMU+VYyIkSZIaREtLC/fedhve561rbbe5WHXbbfUso2Htrtm7DBGSJEkNZB/gLK8O10761m6asq27gdVv2dmDZuaFO7uvJEmSpMbWXU/Edyg/+3RU9jFESJIkSf1UdyHibbutCkmSJEl7jO7uznTB7ixEkqSyFi9eDKuh6ZqmnhtLnVkFi3NxvauQ9jj+1ZUkSZJUindnkgaQxYsXs+7pZs5bMKrepWgP9fDTzYxc3Dif2k6dOpWlsZTW2a31LkV7qKZrmpi639R6lyHtcUqFiIgYCfwdcAqwHzC0k2aZmc+qQW2SJEmSGlCvQ0REjAN+BxwOrAHGAKuBIcDwSrPHgS21LVFSrUydOpWNW5/gw7PW1rsU7aHOWzCKYVP91FaSBroyYyI+TBEgzgLGV9Z9DhgFHA/8CXgAOKyWBUqSJElqLGVCxF8C12Xm/2Rm+/wRWbgReCVwKPCvNa5RkiRJUgMpEyL2p+htaNNK1ZiIzHwKuBw4ozalSZIkSWpEZQZWrwe2VS2vBvbp0GYJxYBr7SE2r1nOY9f/hKcX3cm2LRsZNn4fJj3/JCYe8dJu91v1wK2svOcPrH9yEVvWrSKaBzFsr32ZPOtUxh98zDPab92wlidvvIRVLbewZe1KmoYMY9heU9jnuFcz9qDn9tXTkyRJUh8oEyIepeiNaHM38JKIaM7MtnBxAvBkrYpT39qybhX3ff/jbFm7sn3dhqWP8sgV32bLutXs+4K/7HLfZbf+hjWL/lx1sE2se7yFhy7+Epte/Hr2Oe5VO5zn/h/8J5tWLWlft23DWtY9dj/rHl9oiJAkSdrDlAkR1wKvj4iojIn4ETAf+FVEXALMBl4A/HfNq1SfeOL3v2gPEAe88mzGHPBsHrhoPuufeIAnb7iICYe/iCFjJnS6bwwazORjXsnEI17C4FF7seLu3/Hory8E4Mk/XsLex5xKNDUD8Oivv8umVUuI5sHs//K/ZtzMWRDB+icfArLT40uSJKlxlQkRF1DcznUqRa/EV4GXAa8FTq60+T3FXZzU4DJbWXnfTQAM22sKEw5/EQB7H30KD136FbJ1Gyvvv5m9Z53a6f7TX/k3NA8Z1r486aiXs/S237Jx2WJaN29g6/qnGTxqHJufXsGqllsAmDzrFCYeObt9nzHTn9NHz06SJEl9qdchIjP/BPxt1fJW4LSIOBqYASwCbs5Mpw3dA2xa9RTbNq0HYOhe24e2VD/e8NTDXe5fHSDa5NZiipAYNJjm4cWMyGsX3weVm3lt3fA0d//PB9m0ailDRo9n4lEvZ/LRpxARu/6EJEmStNuUmrG6M5l5C3BLDWrRbrR1w/bJxqoDQfOQ4e2Pt6xf0+vjrbj7hvYxD3sddjxNzcVLa/PTK9rbLL/j2vbHm1Y9xWPX/IBtG9cy5YTTyz8BSZIk1U2ZW7y2i4iREfG8iHhxrQvSbpJdjUXYvj7oXQ/BmkV38vCV3wZg2IT9mDp7+11+s3X7Db2GjJnIs9/xGZ79js8wZMxEAJbc/H9s27yhZPGSJEmqp1IhIiKmRsTPgJXAAuDqqm0nRMTdETG7phWqTwwaMbr98bbNGzt9XN2mK2sW/ZkHLvo8uXULQ/fal5mv+xeah47Yfoxho9ofj5t5NEPHTWLouEmMm3k0ALltCxtXeEMvSZKkPUmvQ0RE7Av8EXgNcCnwB9jho+o/ApOBN9SyQPWNoeMmt7/Z31T1Jr768fDJB3R7jCJAfIHcuoVhE6dy8Bs+xOBR43ZoM2Lv6mN03rPRNGhwueIlSZJUV2V6Iv6dIiS8IjNPA66q3piZW4DrgRfVrjz1lYgmxh9yLAAbVzzOirtvYMu61Sy55Ypie1Nz+6Rxf/rMXP70mbksuvwb7fuveWh7gBi+9wEc/IYPMnjkmGecZ8Q+BzFk7CQAVi1cwKZVS9m0aimrFhbDaAaNGMOwvab06XOVJElSbZUZWP1K4OLMvKabNo8AjpPYQ+z7or9i9YO3s2XtShZd9rUdtu1z/Gu7nCMC4Mk/Xtp+N6YNSx7mji///Q7bZ77+A4yedhgRwbRXzKXlF59j85pl3PXN91W1CvabfQbRtFNDcyRJklQnZd697Q0s7KHNFmDkzpej3WnwyHEc8qZ/Y/xhL6R5+CiieTDDJ+3PtFPe3u1s1WWNOfAIZp7+z4yaeihNg4fSNGgII/ebybNOe0/7/BSSJEnac5TpiVgB7N9Dm4MBR8nuQYaMmcCBr3pXt22e/74LnrHu4DM+WOo8o6cdxuhph5XaR5IkSY2pTE/E74G/jIh9OtsYETOBU6m6Y5MkSZKk/qdMiPg0MAy4NiLmACOgfc6IOcAlQCvw2ZpXKUmSJKlh9Ppypsz8Y0ScDXyV4havbdqmNd4KvD0z76phfZIkSZIaTJkxEWTm/0TE74C/A14ATABWAzcCX8rM+2pfoiRJkqRGUipEAGTmQuA9XW2PiEmZuXSXqpIkSZLUsGp2g/6IGBsR/wk8UKtjSpIkSWo8veqJiIgDgKMp5oG4KTOXVG0bRtEz8T5gPLC+D+qUJEmS1CB67ImIiPkUvQs/AS4CFkXE31W2zQbuA84DhgNfAA7qm1IlSZIkNYJueyIiYi7wbopbt94DBHAIMD8i1gFfA5or38/LzMf7tlxJkiRJ9dbT5UxvBTYDJ2bmHwAi4iXAVcC3gMXAX2Tmn/uySEmSJEmNo6cQcSTwi7YAAZCZ10XERcDpFPNC9MsAsXjxYprWr2bY3Zf23FjqRNP65SxevLXeZUiSJNVcT2MixgItnaxfWPn+h062SZIkSerHeuqJaKK4I1NHWwAyc0PNK2oQU6dOZcmmQWw8/NX1LkV7qGF3X8rUqfvUuwxJkqSa680tXrPPq5AkSdKAsWTjRr6xaBE3r1zJhtZt7D98OKdNmcKr9tm32/1uWL6cXy99ivuefprlmzczpKmJaSNG8Lr99uOlEyd1ud95997Dr5cWcyFPGTaM7x9zbE2fz0DUmxBxbkSc29mGiNjWyerMzNIzYUuSJKn/W755M39/+20s27y5fd0D69bx6YULWbF5C2+eNq3LfX/xxOPcvHJl+/LG1lbuXLOGO9es4ezpG3nT/vs/Y587Vq9uDxCqnd7MWB0lv2o2C7YkSZL6l+88vKg9QHzokEP42XEv4LDRowG44JGHWbJxY5f7Dm1q4oypU7nw6FlcfvyLeM+zZrRv+96jj7A1d7yAZlsmX3ighSZgSJNvUWup259mZjbtzFeZAiLi1Ii4LyJaIuIDnWw/MyLuqHzdEBHPLfskJUmSVH+tmfy20itwwPARnDx5byYMGcLr95sKwNZMrl22rMv9P3TIobzrwIOYNmIEw5ubec2UKRw4YgQA67ZtY/WWzTu0v+jxx3lg3Tpetc8+jB88uI+e1cBU10gWEc3Al4E5wOHAGyPi8A7NHgJemplHAh8Hvr57q5QkSVItPL5xI+u2FVfDTx0xvH391OHbHy9ct7bL/Uc0Nz9j3ebWVqDoaRgzaHtQWLl5M99+eBFjBg3iHdMP3OXataN69+scC7Rk5oOZuRn4IfCa6gaZeUNmtl38diMwdTfXKEmSpBpYvWX7TT9HVgWC6serNnd2Y9DOXfXUEh6rXP500uTJDK66ZOmrDz3Eum3beOf0AxlrL0TN1XsA9H7Ao1XLi4Hjuml/FnB5n1YkSdqzrIKma+r9mVgDa/tQd1Rdq2hcqyjejWi3yC5u+rnD2ujdsRasXMmnFxZTl00fMYK/PfCg9m13rVnDlU8t4ZBRo3jVPt5uvS/UO0R09jLp9NUVESdShIgTuth+NnA2wLRuRvVLkvqPGTNm9NxogFtYeZM1c7+Zda6kQe3n62h3Gjd4SPvjtsuaANZXPR7Xi16Dm1eu4F/vvpvNra1MGz6czx5xBKMGbX9b+91HHyGBv9hnXx5Ytw6gfdD11kwWrl3L3kOHMsYeip1W7xCxGKi+F9dU4PGOjSLiSOCbwJzMXN7ZgTLz61TGS8yaNcu5LSRpAJg3b169S2h4bT+j+fPn17kSqZijYWRzM+u2bWPx+u1zFi/esP3xzJHdd5tVB4iDRozks0ccwfghQ3Zos6ESSj7TsvAZ+z+1aRPvvPVPvP/gg5mzt70UO6ve/b83AzMj4sCIGAKcAVxc3SAipgE/B96cmffXoUZJkiTVQFMEL5tUTAr38Ib1XPXUElZs3syPH1sMwKAIZk+aCMDs669j9vXX8Yn77mvf/6aqAHHwqFF87sgjnxEgtHvUtSciM7dGxLuBK4Bm4NuZeVdEvKuy/avAR4AJwFciAmBrZs6qV82SJEnaeW89YDp/WLGCZZs38x9VAQFg7rQDmDx0WJf7fu/RR9vvxnT/2rW85sY/7LD9c0ccyfPGjeMLRz5zRoA33PRHlmza5IzVNVLvy5nIzMuAyzqs+2rV43cA79jddUmSJKn2JgwZwpefexTfWPQQN61cyYZt25g2YgSnTZnCq/bZt97lqZfqHiIkSZI0sOw9bBgfPvSwbttc8+KXPGNdZz0MvfWjY7u7AajKqveYCEmSJEl7GEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUrzFqzTAPLK2mfMWjKp3GQ1pyfric5W9R7TWuZLG9cjaZg6udxGSpLozREgDyIwZM+pdQkPbvHAhAMOmz6xzJY3rYHwdSZIMEdKAMm/evHqX0NDafj7z58+vcyWSJDU2Q4QkSVKDWLx4MU8D3yLrXYr2UE8Aaxcv7vPzOLBakiRJUin2REiSJDWIqVOnsmrZMs4i6l2K9lDfIhk3dWqfn8eeCEmSJEmlGCIkSZIklWKIkCRJklSKIUKSJElSKYYISZIkSaV4d6ZuNK1fwbC7L613GQ0rNq4BIIeNqXMljalp/Qpgn3qXIUmSVHOGiC7MmDGj3iU0vIULnwZg5rN8o9y5fXwdSZKkfskQ0YV58+bVu4SG1/Yzmj9/fp0rkSRJ0u7kmAhJkiRJpRgiJEmSJJViiJAkSZJUiiFCkiRJUimGCEmSJEmlGCIkSZIklWKIkCRJklSKIUKSJElSKYYISZIkSaUYIiRJkiSVYoiQJEmSVIohQpIkSVIpg+pdgCRJkrZ7EvgWWe8yGtbyyvcJda2icT0JjNsN5zFESJIkNYgZM2bUu4SGt3ThQgDGzZxZ50oa0zh2z+vIECFJktQg5s2bV+8SGl7bz2j+/Pl1rmRgc0yEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJIkSSrFECFJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSjFESJIkSSrFECFJkiSpFEOEJEmSpFIMEZIkSZJKMURIkiRJKsUQIUmSJKkUQ4QkSZKkUgwRkiRJkkoxREiSJEkqxRAhSZIkqRRDhCRJkqRSDBGSJEmSSql7iIiIUyPivohoiYgPdLI9ImJ+ZfsdEfH8etQpSZIkqVDXEBERzcCXgTnA4cAbI+LwDs3mADMrX2cD/71bi5QkSZK0g3r3RBwLtGTmg5m5Gfgh8JoObV4DXJiFG4FxEbHv7i5UkiRJUmFQnc+/H/Bo1fJi4LhetNkPeKJvS2s88+fPp6Wlpd5ltFu4cCEA8+bNq3Ml282YMaOh6lH3fE33zNf0nqXRXtPg61q7rtFe176mG0O9Q0R0si53og0RcTbF5U5MmzZt1ytTj4YPH17vEqSa8jWt/sjXtfobX9ONITKf8X5895084oXAuZl5SmX5gwCZ+YmqNl8DrsnMH1SW7wNmZ2aXPRGzZs3KBQsW9GntkiRJUn8XEbdk5qyO6+s9JuJmYGZEHBgRQ4AzgIs7tLkYeEvlLk0vAFZ3FyAkSZIk9a26Xs6UmVsj4t3AFUAz8O3MvCsi3lXZ/lXgMuCVQAuwHnhbveqVJEmSVP8xEWTmZRRBoXrdV6seJ/D3u7suSZIkSZ2r9+VMkiRJkvYwhghJkiRJpRgiJEmSJJViiJAkSZJUiiFCkiRJUimGCEmSJEmlGCIkSZIklWKIkCRJklSKIUKSJElSKYYISZIkSaUYIiRJkiSVYoiQJEmSVIohQpIkSVIphghJkiRJpRgiJEmSJJUSmVnvGmouIpYCD9e7jgFiIrCs3kVINeRrWv2Rr2v1N76md58DMnNSx5X9MkRo94mIBZk5q951SLXia1r9ka9r9Te+puvPy5kkSZIklWKIkCRJklSKIUK76uv1LkCqMV/T6o98Xau/8TVdZ46JkCRJklSKPRGSJEmSSjFEaJdExPSIeNNO7PfWiPhSX9QkSQNFREyJiJ/W+JjnRsT7anlMSf2PIUK7ajrQaYiIiEG7txRJGlgy8/HMPL3edUi7KiKad3F/33PsZoaIAS4i3hIRd0TE7RHx3Yg4ICJ+U1n3m4iYVmn3nYiYHxE3RMSDEdH2n9YngRdHxG0R8Z5KD8NPIuIS4MqI2CsiLqoc78aIOLJuT1bqICL+NSLui4hfR8QPIuJ9EfGsiPi/iLglIq6PiEPrXacEEBHnR8TfVS2fGxHvjYg7K8vPjoibKn+P74iImZXe4jur9nlfRJxbefzOiLi58vf/ZxExYrc/KQ0IldfhvRFxQeW1+dOIGBERiyLiIxHxO+B1EfHGiPhzRNwZEedX7X9WRNwfEddExDfarmSovDf5r4i4Gji/q7/fEfG6yjFvj4jrKuue8e+lHj+bPZkhYgCLiGcD/wq8LDOfC/wD8CXgwsw8EvgeML9ql32BE4BXU4QHgA8A12fmUZn5ucq6FwJzM/NlwEeBWyvH+xBwYR8/LalXIuJo4AzgecBpwDGVTV8HzsnMo4H3AV+pT4XSM/wQeEPV8uuBm6uW3wV8ITOPAmYBi3s43s8z85jK3/97gLNqWKvU0SHA1yvvB9YAbYF4Y2aeAFwHnA+8DDgKOCYiXhsRU4B/A14AnAR0/GDnYOAVmfleuv77/RHglMpr/S8r68r+e1EHdv0MbC8DfpqZywAyc0VEvJDiDRXAd4FPVbW/KDNbgbsjYu9ujntVZq6oPD4B+H+V4/82IiZExNiaPgtp57wY+EVmrgeIiIuBYcDxwE8ioq3d0PqUJ+0oM2+NiMmVN1WTgJXAI1VN/gD8a0RMpQgIC6tex515TkScB4wDRgFX9E3lEgCPZubvK4//F5hXefyjyvdjgGsycylARHwPeEll27Vt7ysi4icUwaHNTzJzW0SMouu/378HvhMRPwZ+Xln3jH8vNXqeA4YhYmALoKd7/FZv39Rh366s66Gd9xVWo+j4WmwCVlU+mZIa0U+B04F9KHom2mXm9yPij8CrgCsi4h3A/ex41cGwqsffAV6bmbdHxFuB2X1XtvSMv7dty23vGbp6X9FtEq7av8u/35n5rog4juLfxm0RcVRn/14y87c9PQlt5+VMA9tvgNdHxASAiNgLuIHiEg+AM4Hf9XCMp4HR3Wy/rnIcImI2sCwz1+x8yVLNXAf8VUQMj4jRwF8A64GHIuJ1AFF4bj2LlDr4IcXf6NMpAkW7iDgIeDAz5wMXA0cCS4DJlV7goRSXo7YZDTwREYOp/J2W+tC0ytUOAG/kme8v/gi8NCImRjHI+o3AtcBNlfXjoxg8/f86O3jlvUWnf78j4lmZ+cfM/AiwDNi/i38vKsEQMYBl5l3AfwDXRsTtwH9RdC++LSLuAN5MMU6iO3cAWyuDld7TyfZzgVmV430SmFur+qVdkZl/ouhGvw34GXB9ZdOZwFmVfxN3Aa+pS4FSJyp/t0cDj2XmEx02vwG4MyJuo7hu/MLM3AJ8jOIN2qXAvVXt/62y/qoO66W+cA8wt/J+YC/gv6s3Vl7PHwSuBm4H/pSZv8zMx4D/pHit/hq4G1jdxTm6+vv96bYB2xQfIN1OJ/9eavVEBwpnrJYkijvdAGsz8zP1rkWS+pOImA5cmpnP2cn9R2Xm2kpPxC+Ab2fmL2pZo8qzJ0KSJEmN7NxKj8GdwEPARXWtRoA9EZIkSZJKsidCkiRJUimGCEmSJEmlGCIkSZIklWKIkCS1i4jBEfHRiFgYEZsiIiPitfWuq7ci4o0RcWtEPF2p/fNV206OiBsiYmVl20U7eY5zK/vPLrHPNRGRHdbNrhzn3J2pQ5LqyRmrJQ14Hd/c9cLbMvM7fVFLA3gv8BGKe6n/GNjCHjKHQGUiq+8BD1Lcg349cGNl23Tgl8Aq4H+ANewhz0uSGpEhQpLgo52s+0dgLPAFijee1W7r23Lq6tXAWuCkzNxc72JKehUQwFsy84YO214BDAPem5nf3+2Vde4m4DCKGXQlaY9iiJA04GXmuR3XRcRbKULE5zNz0W4uqZ6mAMv3wAABRe0Aj5fcVheZuR57QyTtoRwTIUm9EBHNEfFoRKyJiFFdtPlS5Rr3/1e1LivXw0+JiO9GxFMRsSEibomIN3VzvlMi4rKIWFYZm/BARHw6IsaVrHtsRHwiIu6LiI2V8QBXRMQrOrT7TuWyrgOBAyp1Z0Qs6sU5DoqIr0dES+W5rYiIP0fEVyNiQift3xARv6m02xgRiyLiBxExq0Pd/xwRv42IxRGxOSKWRsTFEfGCDsd7a6X2t1VWPVRVf9u2tt6mq6u2za46xsyIuDAiHquc6/HK8sxe/aC3H+eMyu92Q+V3/d2ImNJF207HRLSNn4iIQRHxoarxKY9GxPkRMaSL450ZEX/qeO7OxmNI0q6yJ0KSeiEzt0XENyjejL4R+Eb19ogYDpwJPAlc3GH38cANbL8efxzweuB7EbFfZn66w7E+UjnPCuBS4CngSOB9wCsj4oWZuaanmiuB4/fA4cDNwOeBiZVzXxkRf5uZX6s0vwhYRHEZF5W28MxLuTqeY9/KsccAlwE/o7hs6EDgzcCXgOWVtlF5/nMpLuH5ObAUmAqcCNwHLKgc+jDgPyjGZvwKWAlMA/4SmBMRf5GZ/1dpexvFz+u1wHPZ8RK0tm2zgZcCF1SeJ23fI+IY4NfAaIrf3d3AoRS/z9dExMszs62u7n4W7wH+q3LuCyvfT6H43a/uaf9OfB94MXA5xRiOVwL/Akxme2BqO/c/A5+i+DldUDnfSRS//505tyR1LzP98ssvv/zq8EXxBjOB6VXr9qUYaLygk/ZvrbT/jw7rs/L1Y6Cpav2BFCFhM3BQ1foTK+1vAMZ1cY7P9fI5fK3S/mtAVK2fSfHGclP186t63otK/JzOqZzjHzrZNhIYXrV8dqXtTcDYDm2bgX2rlscCEzs55lSKS5Lu6WTbdzr+zqq2nVvZNrvD+gDuqWw7s8O2N1TW39vhd/eMYwHTKz/PFR1eM00UwSqL/3J3OP7syvpzO6y/prL+FmCvDj/PFmAbsE/V+oMqr8ulwP4dntsPOju3X3755deufnk5kyT1UmY+QfGJ/dERcXSHzX8DtNKhh6JiG/D+zGytOtZDwHxgMMUn9m3mVb6/MzNXdTj/dyg+WT+zp1ojYjDw1xSDpD+Yme2Xs2Tmwsq5hwBv6elYvbSh44rMXJeZ1evPqXz/m8xc3aHttsrPt215dWY+Y8BxZi4GfgocGhHTalD38RS9Dn/IzO91ONePgN8BhwAn9HCcMyl+nl/MqjE0ld/5P1O8Nsp6f2auqDrWOoq7TzUBs6ravYniyoIvZuajVe0T+ADF60+SasoQIUnlfKXy/W/aVkTEEcALgCuy80HYj1RCQ0fXVL4/r2rdCyk+VX5dFPMR7PBF8UZ1UmdjDTo4FBgB3F79RrTKbzs59864mCKofDkifhYRZ0fEsyuXLrWLiJHAc4AlmXlrbw4cES+KiB9XxgK0zVmRbA8j++1i7QDPr3z/bRfbe/tzajvOtR03ZOaDwKMd1/dCZ5dQtR1nfNW6ttp+18m5H97Jc0tStxwTIUklZObVEXEP8MaIeG9mPs32QPG1LnZb0sX6Jyvfx1atm0Dxt/nfeyhlFJWxBl1oO+YTXWxvWz+uh/N0KzMfjohjKS7xORU4rbLp0Yj4TGbO73Cex3pz3Ij4K4oeh43AVcADwDqKT/RnU4xvGLortVfU6ufUdpzuftcH9L4s6NgTVbG18r25xLmXUFxuJUk1Y4iQpPK+SjF498yIuIDiUpbHKAZBd2bvLtbvU/lefWnPaorr7/faxRrbjrlPF9v37eTcOyUz7wHeEBGDKAY2v4Kit+ALEbEuM7/F9oHOve09+DjFeJFZleO3i4ivUYSIWqjVz6lt+97AXZ1s7+r4tdA2yL6rc3f1+pOkneblTJJU3gUUn4r/DcXg23HAtzKzq2vPp0UxY3JHsyvfqy/vuREYHxHP3sUa76OYsfmoiBjfyfYTK9//tIvnaZeZWzPzlsw8n+IOVlDcMantev47gb0jojeXUM0A7u4kQDTR8/iEMtp+9rO72N62vqefU9v2Z4SbiDgI2L9sYSW0PYdn/Fwi4oA+PrekAcoQIUklVQYF/wA4CjiPYuDqN7vZpRk4v/IGGICIOJBiEPVW4H+r2n6u8v0bnc0vEBEjO86T0EWNmykG4Y4CPtbhGM+qnHsL8N2ejtWdiDg2Ijr7pLtt3fqqdW2XNn0tIqov4SIimiq3i22zCJhZ/TOojLP4d4pb1tbK7ykC1wkRcXqHmk4HXgLcTyfjDTr4HsXP85zqwFj5nX+avv3/9vsUr6NzIqI9MFR+Xp9gx0ufJKkmvJxJknbOV4B3UFyec0n1XXE6cQdwHHBLRFxJcQ17Ww/Gv2TmA20NM/M3EfEBijd/CyPiMuAhijBwAMUn3b+jGH/Qkw9QzDPw7spcCFezfZ6I0cC7uxjwXcabgL+PiGspbj+6EngW8BcUtzz9fFXbb1J8Wv6WynP7JcVtSacALwO+TTG2Aoow9VXg1oj4GcUb9BdRBIhLKsffZZmZETGXYtzFjyo13UtxR6bXAk8Db6m+s1YXx1lU+b19tlLzjygucTqF4vd8B8VcHzWXmQ9U5hb5T+D2qnOfBOwF3N5X55Y0cNkTIUk7oXKHodsqi10NqG6zkuJWondRTBL2VopgcGZ2mGiucuzzKT4B/xXFG+d/BF5HEVi+Dny4lzWuoLjb06coBmz/U+U4NwGnZuZXutm9t35AEQ4mUYSTf6S4U9EPKcYz/KGqnszMuRS3nr2n0v6fKILR9VRN0pfFJHhvoxjYPJdi3MmjFGGsZpdgVc71R+AYik/0X0hxS9bjK8/tmMr23hznvyhC1UMUv+O3U1zCdTzFa6DPZOYnKMLZwxQ/t7MofsYvovjAsMfJCSWpjKi6dbgkqZciYjTFpGcrgAO7+qS6ckvSazNz9m4sTwIgIsZQ3J3ptsx8Yb3rkdR/2BMhSTvnbykuMfpKT5e6SH0tIiZVJhisXjeI4vKqYcAv6lKYpH7LMRGS1EuVwcB/S3FZ0TspLrWpxSVB0q76f8DHIuLXFJd97UVxSdzBFJfdfbF+pUnqjwwRktR74ykGPG8CbgHOqUw2J9XbHykG3L+EYvwLFGMz/gM4PzM31KswSf2TYyIkSZIkleKYCEmSJEmlGCIkSZIklWKIkCRJklSKIUKSJElSKYYISZIkSaUYIiRJkiSV8v8Bv8uz8aaBwicAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 936x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def make_boxplot(df: pd.DataFrame, y:str):\n",
    "    \"\"\"Creates a boxplot from dataframe df, attribute y\"\"\"\n",
    "    ylabel = y.replace(\"_\", \" \").title()\n",
    "    figure(num=None, figsize=(13,8))\n",
    "    order = [\"control\", \"qe\", \"visual\", \"progress\"]\n",
    "    ax = sns.boxplot(data=df, x=\"type\", y=y, order = order)\n",
    "\n",
    "    means = df.groupby(\"type\", sort=False)[y].mean()[order]\n",
    "    medians = df.groupby(\"type\", sort=False)[y].median()[order].values\n",
    "    \n",
    "    if \"time\" in y:\n",
    "        nobs = [f\"{means[_type]:.2f}s\" for x, _type in zip(df[\"type\"].value_counts()[order], order)]\n",
    "    else:\n",
    "        nobs = [f\"{means[_type]:.2f}\" for x, _type in zip(df[\"type\"].value_counts()[order], order)]\n",
    "\n",
    "    # Pair wise statistical tests\n",
    "    for i, j in [(\"control\", \"qe\"), (\"control\", \"visual\"), (\"control\", \"progress\"), (\"qe\", \"visual\"), (\"visual\", \"progress\")]:\n",
    "        p_value = ttest_ind(df[df.type==i][y], df[df.type==j][y]).pvalue\n",
    "        significant = \"Significant\" if p_value < 0.05 else \"Not singnificant\"\n",
    "        if i == \"qe\" or j == \"qe\":  #Just to make everything aligned.\n",
    "            print(f\"p-Value {i} vs {j}:\\t\\t{p_value:.5f}, {significant}\")\n",
    "        else:\n",
    "            print(f\"p-Value {i} vs {j}:\\t{p_value:.5f}, {significant}\")\n",
    "\n",
    "    pos = range(len(nobs))\n",
    "    for tick, label in zip(pos, ax.get_xticklabels()):\n",
    "        ax.text(pos[tick], medians[tick]+0.03, nobs[tick], horizontalalignment=\"center\", color=\"black\", weight=\"semibold\", fontsize=15)\n",
    "    ax.set_xlabel(\"Type of scaffolding\", fontsize=20)\n",
    "    ax.set_ylabel(ylabel, fontsize=20)\n",
    "    plt.show()\n",
    "    plt.clf()\n",
    "\n",
    "make_boxplot(df, \"realized_learning_gain\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "champion-acquisition",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}