An unfinished effort to update my earlier Netrunner project
About a year after my original Netrunner Social Network Analysis project (https://github.com/MTKnife/Netrunner-Social-Network-Analysis), I decided to do things right, with automated web-scraping. Using two different repositories would give a larger corpus of decks to analyze, and the release of additional expansions to the game in the previous year should have created a considerably broader range of options for deck design. The wider variety of decks would make any analysis more meaningful (back when I built the first version of first project, the number of cards available was fairly limited, making most decks look similar). Obviously, we don't have anything like a representative sample here, since there's no way to discover all of the decks players have actually used in the game (most of which were never posted), or how often they were used--or, for that matter, whether the decks that have been posted have actually been used. Nonetheless, I was hoping that analyzing a wide variety of decks would reveal meaningful patterns, especially for the serious players who care enough about the game to post their decks online.
I never finished this second version of the project, because by that point I'd lost interest in Netrunner. Now, over a year later, I'm not completely sure which file does what, but I'll try to provide as useful a guide as I can, for anyone who'd like to pick up the project up again.
I used CardGameDB (http://www.cardgamedb.com/index.php/netrunner/android-netrunner-submitted-decks), as I had in the first version of the project, but this time I wrote scripts to download the decks automatically, first by scraping the index pages to get a list of deck URL's ("CompileCGDBCorpURLList.R" and "CompileCGDBCorpURLList.R"), and then to scrape the decks themselves ("DownloadCGDBCorpDecks" and "DownloadCGDBRunnerDecks"); the deck lists turned out to be contained rather neatly in JSON objects, though I scraped the rest of the pages for some additional meta-data (which probably would never have been used--the JSON deck objects contain all the meta-data you'd really need).
I also used NetrunnerDB (http://netrunnerdb.com/en/decklists/popular), the only other Netrunner deck repository that made it practicable to browse all stored decks. It was pretty easy to scrape the deck URL's from the index pages, just I'd done for CardGameDB ("CompileNRDBCorpURLList.R" and "CompileNRDBRunnerURLList.R). Scraping the pages themselves proved more difficult, since the pertinent information was buried behind AJAX calls. I was ultimately able to scrape them, slowly, using RSelenium (https://cran.r-project.org/web/packages/RSelenium/index.html) to hover a mouse pointer over the right button, click it, and capture the text that was then displayed ("DownloadNRDBCorpDecks.R" and "DownloadNRDBRunnerDecks.R"). (This was an old version of Selenium--the current version, available at http://www.seleniumhq.org/, is easier to install and run, but I'm guessing most if not all of the RSelenium code will still work).
I was really proud of myself for constructing this elaborate solution, until I discovered, some months later, than NetrunnerDB had a nice, convenient API, http://netrunnerdb.com/en/apidoc (this is what happens when we don't read webpage footers), but by that time the site was temporarily down due to copyright issues, and I was never able to make use of the API.
"CleanCorpDB.R" is an unfinished script that was intended to put all the corp decks into the same format and save them as a single file.
"ExpansionsIndex.txt" contains a list of Netrunner expansions and their formal release dates, as well as the dates on which the first decks using them appeared online. Anyone wanting to do a meaningful analysis of decks would probably find it best to restrict the analysis to decks from only a certain period, such as the period over which the most recent series of expansions was released, to make sure you're not comparing apples and oranges, since older decks, built when fewer cards were available, would probably not be seen in a current game. (To make the data more homogeneous, you could look only at decks made between two expansion releases, meaning all decks were created with exactly the same cards available, but the dates between any two releases are too close together--two months at most--to provide a large enough sample of decks.)
The other files in the main directory should be self-explantory, though I'm not sure which of them are current, and which might need to be updated. The files in "examples" are example web pages, or pieces of pages, that I used to write and test the scraping scripts.
Anyone who'd like to fork this and/or my first project is welcome to, but please drop me a note and let me know what you've done!