{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Constructing Ego Networks from Retweets</h1><h2>(using pre-saved files instead of Twitter authentication)</h2>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yotam Shmargad<br>\n",
    "University of Arizona<br>\n",
    "Email: yotam@email.arizona.edu<br>\n",
    "Web: www.yotamshmargad.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Introduction</h2><br>\n",
    "Twitter has become a prominent online social network, playing a major role in how people all over the world share and consume information. Moreover, while some social networks have made it difficult for researchers to extract data from their servers, Twitter remains relatively open for now. This tutorial will go through the details of how to construct a Twitter user’s ego network from retweets they have received on their tweets. Instead of focusing on who follows who on Twitter, the method instead conceptualizes edges as existing between users if they have recently retweeted each other.<br><br>\n",
    "Conceptualizing edges as retweets has two primary benefits. First, it captures recent interactions between users rather than decisions that they may have made long ago (i.e. following each other) that may not translate into meaningful interaction today. Second, users often have many more followers than they do retweeters. The method proposed can thus be used to analyze even relatively popular users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>1. Importing libraries</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the libraries we need\n",
    "import json\n",
    "import time\n",
    "import networkx\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check working directory\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set working directory\n",
    "os.chdir('FOLDER FOR SAVING FILES')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check working directory\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>2. Pulling ego tweets</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Read saved ego tweets\n",
    "with open('egotweet.json', 'r') as file:\n",
    "    ego = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Looking at a json object\n",
    "ego[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Accessing an element of ego tweets\n",
    "ego[0][\"id_str\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Storing one of ego's tweet id\n",
    "egoid = ego[0][\"id_str\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Storing and printing ego tweet ids and retweet counts\n",
    "tweetids = []\n",
    "retweets = []\n",
    "\n",
    "if len(ego) != 0:\n",
    "    for egotweet in ego:\n",
    "        tweetids.append(egotweet[\"id_str\"])\n",
    "        retweets.append(egotweet[\"retweet_count\"])\n",
    "        print(egotweet[\"id_str\"],egotweet[\"retweet_count\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>3. Pulling retweeters</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sleep for 10 seconds\n",
    "time.sleep(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading saved ego retweeters\n",
    "with open('check.json', 'r') as file:\n",
    "    check = json.load(file)\n",
    "\n",
    "with open('self.json', 'r') as file:\n",
    "    self = json.load(file)\n",
    "    \n",
    "with open('allretweeters.json', 'r') as file:\n",
    "    allretweeters = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Printing tweet ids, retweet counts, \n",
    "# retweeters obtained, and whether a self tweet is included\n",
    "for a, b, c, d in zip(tweetids,retweets,check,self):\n",
    "    print(a, b, c, d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(allretweeters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allretweeters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>4. Visualizing the network of retweeters</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Assigning edge weight to be number of tweets retweeted\n",
    "weight = Counter()\n",
    "for (i, j) in allretweeters:\n",
    "    weight[(i, j)] +=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining weighted edges\n",
    "weighted_edges = list(weight.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weighted_edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the network object\n",
    "G = networkx.Graph()\n",
    "G.add_edges_from([x[0] for x in weighted_edges])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualizing the network\n",
    "networkx.draw(G, width=[x[1] for x in weighted_edges])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>5. Pulling retweeter tweets</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the set of unique retweeters\n",
    "unique = [x[0][1] for x in weighted_edges]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(unique)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading saved retweeter tweets\n",
    "with open('alters.json', 'r') as file:\n",
    "    alters = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(alters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Printing the number of tweets pulled for each retweeter\n",
    "for alt in alters:\n",
    "    print(len(alt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Storing and printing alter ids, tweet ids, and retweet counts\n",
    "altids = []\n",
    "alttweetids = []\n",
    "altretweets = []\n",
    "\n",
    "for alt in alters:\n",
    "    for alttweet in alt:\n",
    "        altids.append(alttweet[\"user\"][\"id_str\"])\n",
    "        alttweetids.append(alttweet[\"id_str\"])\n",
    "        altretweets.append(alttweet[\"retweet_count\"])\n",
    "        print(alttweet[\"user\"][\"id_str\"],alttweet[\"id_str\"],alttweet[\"retweet_count\"])        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>6. Pulling retweeters of retweeters</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading saved alter retweeters\n",
    "with open('altcheck.json', 'r') as file:\n",
    "    altcheck = json.load(file)\n",
    "\n",
    "with open('altself.json', 'r') as file:\n",
    "    altself = json.load(file)\n",
    "    \n",
    "with open('altretweeters.json', 'r') as file:\n",
    "    altretweeters = json.load(file)\n",
    "\n",
    "with open('allalt.json', 'r') as file:\n",
    "    allalt = json.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Printing alter user ids, tweet ids, retweet counts, \n",
    "# retweeters obtained, and whether a self tweet is included\n",
    "for a, b, c, d, e in zip(altids,alttweetids,altretweets,altcheck,altself):\n",
    "    print(a, b, c, d, e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(allalt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allalt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>7. Visualizing the full network of retweeters</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weight = Counter()\n",
    "for (i, j) in allalt:\n",
    "    weight[(i, j)] +=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_edges = weighted_edges + list(weight.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_edges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the full network object\n",
    "G = networkx.Graph()\n",
    "G.add_edges_from([x[0] for x in all_edges])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualizing the full network\n",
    "networkx.draw(G, width=[x[1] for x in all_edges])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
