We are data science master students at Frankfurt School of Finance and Management. For this project we teamed up with RegHub (https://www.reghub.io) to investigate how public news about competitors in the german banking space can be analysed on an ongoing basis. We worked with a dataset of 14609 news articles to train and test our approaches.
General exploration of the dataset and various visualizations.
Rule-based labelling of the news articles into one or more of the following categories:
- legal
- sanctions
- papers
- reports
- statements
- guidelines
- press
- personnel
- market
Testing various supervised deep learning algorithms, using the rule-based labels for training and testing. We settled on BERT as our primary model for categorization. Next to categorization we also include a short summary of the relevant events within the category. To generate this summary, Llama2 was used. Alongside as part of information retrieval from the news articles we also used name entity recognision models to extract the name entities. Also a similarity analysis was performed by training BERT-MLM, which can be used to filter out duplicate news articles.
Script that runs through the whole process of categorization and llama2 summary creation for a given dataset. Can be run at specified intervals to cover new news articles.
Collection of scripts that where used and or tested in the course of this project, but don't belong into the final main folders.
This folder is used to package the models, to be able to release them as a pip package later on.
The run file in the weekly analysis folder can be used to analyse the additional news articles of every week. It runs through the whole pipeline of BERT categorization and Llama2 summarization.
See the presentation of this project: https://1drv.ms/b/s!AsfpqRPTBA6DvH3pPeYWj7ub28lM?e=BIp20h
pip install reghub-pack
import reghub_pack
Install dependent packages: (terminal)
conda env create -n myenv -f reghub_packages.yml
Clone repository branch: (terminal)
git clone -b reghub_pack https://github.com/kirteshpatel98/RegHub_news_signal_analysis
Change directory to the clone repo
cd */RegHub_news_signal_analysis
Install poetry
pip install poetry
Build poetry
poetry build
Add package to your environment
pip install .
Import package in python
import reghub_pack
We would like to express our sincere gratitude to Gerrit Knippschild and RegHub for their invaluable assistance and support throughout the duration of our project. Their contribution in providing us with access to the dataset and offering expert guidance has been instrumental.