Identify topics researched at the Alan Turing Institute
For more details see the blog post.
In summary, for each Turing fellow we downloaded up to 50 articles they published since 2012 (inclusive) using open access routes. The downloaded articles were converted to text files and cleaned. We analysed the text data using Latent Dirichlet Allocation (LDA) which identified 25 separate research topics in the dataset. The result is a visualisation of each Turing fellow and the extent to which their work falls under any of the identified research topics.
1: Identifying Turing fellows' publications
We used Microsoft's Academic Knowledge (AK) database to extract publications records for each Turing fellow. We first manually extracted a list of unique AK IDs for all fellows to get around entity matching issues to ensure we are targeting the right researchers in the database. We then used these IDs to retrieve a list of article titles, DOIs and source URLs.
Method: 1_generate_publications_list.ipynb
Results: publications_eng.csv
2: Publications download
(i) We built a web scraper to download each of the articles identified in step 1 (relying on open access routes)
Method: 2a_article_scraper.ipynb
(ii) The downloaded PDFs were converted to .txt files (using Contentmine's NORMA tool)
see 2b_pdf_to_txt.md for description
3: Data pre-processing
All data was cleaned and combined into final analysis-ready csv file
The final csv consists of 1 article per row and contains: fellow name, ID and affiliation, article title and ID, article full text and length, URLs to versions of the article available online and AK provided keywords associated with the article.
Method: 3_data_preprocessing.ipynb
Data in 2 parts: final_dataset_1.csv and final_dataset_2.csv
4: Analysis
The converted texts were analysed using LDA
Method: 4_LDA.ipynb
Results (topic proportions by researcher, used for visualisation purposes): data_original.csv
5: Visualisation
A visualisation of the final topics was built using an adaptation of the aster plot (see visualisation folder)
