This repository contains two primary Python scripts: data-retrieval-automation-mno.py and data-retrieval-automation-iptsp.py. These scripts are designed to automate data retrieval tasks from a specific web application. Users can specify a time range, and the scripts will access the web portal, log in with provided credentials, and retrieve data within the specified timeframe. The data retrieved by these scripts serves as input for subsequent analysis.
Additionally, the repository provides access to the CSV-Error-Analyzer script, which is intended for automated error analysis of the CSV data generated by the Data-Retrieval-Automation scripts. This script reads the specific CSV files, extracts information from the 'message' field, and aggregates error occurrences based on specific criteria. The output is generated as a new CSV file, facilitating the identification of error patterns and trends within the data retrieved from the web application.
These automation tools are developed to enhance productivity and efficiency in data-related tasks, particularly within the context of Infozillion Teletech BD Ltd. The repository streamlines the data retrieval and analysis processes, reducing manual effort and the potential for human error.
MNOs provide wireless communication services and play a crucial role in delivering voice calling, text messaging, and mobile data.
IPTSPs offer telephony services over the internet using VoIP technology, enabling cost-effective voice calls and text messaging.
Infozillion Teletech BD Ltd, often referred to as the Central Platform (CP), plays a pivotal role in collecting the raw data generated by MNOs and IPTSPs. The raw files provided by these MNOs and IPTSPs primarily consist of Call Detail Records (CDRs), which are created each time a subscriber utilizes services like messaging. These raw data encompass a wide range of information related to MNOs, IPTSPs, and Aggregators.
To facilitate the efficient management of this data, a robust automation workflow has been established. The data-retrieval-automation-mno.py and data-retrieval-automation-iptsp.py scripts are key components of this process, serving as browser automation tools. These scripts are used to retrieve the raw data from the web application and download the corresponding CSV files containing this valuable information. Once these raw CSV files are secured, they undergo further processing with the assistance of the CSV-Error-Analyzer.
The CSV-Error-Analyzer tool is instrumental in detecting and categorizing errors and anomalies within the raw data. This in-depth analysis is a crucial step in ensuring data quality and accuracy. Once the data is transformed into a more useful format, it is categorized and reported to specific Aggregators. This reporting is tailored based on the type of errors identified and the number of occurrences.
In summary, the Central Platform at Infozillion plays a fundamental role in collecting, processing, and delivering valuable data to MNOs, IPTSPs, and Aggregators. This data forms the backbone of services and decision-making processes, ultimately contributing to improved efficiency and quality.
The Data-Retrieval-Automation is a set of Python scripts, data-retrieval-automation-mno.py and data-retrieval-automation-iptsp.py, designed to automate a series of tasks using the Selenium web automation framework. It is created to interact with a specific web application, automate login, and download reports.
Before using the scripts, ensure you have the following dependencies installed:
- Python
- Selenium (
pip install selenium) - PyAutoGUI (
pip install pyautogui) - Chrome WebDriver
-
Clone this repository to your local machine:
git clone https://github.com/iftekharmickey/Infozillion-Data-Toolkit.git
-
Navigate to the repository's directory:
cd Infozillion-Data-Toolkit
-
Install the required Python libraries:
pip install -r requirements.txt
-
Download the Chrome WebDriver for your specific Chrome version and platform. Make sure the WebDriver executable is in your system's PATH.
-
Run the script:
python data-retrieval-automation-mno.py
python data-retrieval-automation-iptsp.py
-
Follow the on-screen prompts to enter the start and end times for your task. Ensure you have an internet connection.
-
The script will automate the process and download the necessary reports.
-
The script starts by creating a Chrome WebDriver instance.
-
It prompts the user to input the start and end times for a time range.
-
It constructs a URL based on the user's time range input and opens that URL in the Chrome browser.
-
The script uses PyAutoGUI to enter a username and password and logs in to the web application.
-
It then clicks through several elements on the web application to reach the desired report download.
-
After downloading the report, the script waits for a few seconds and then closes the browser.
-
The username and password for the web application have been removed from the scripts for security reasons. Before running the scripts, you will need to manually add your username and password to the designated fields in the script. Be sure to keep your credentials secure and do not share your login information within the script files.
-
Ensure that the Chrome WebDriver is compatible with your Chrome browser version. You may need to update the WebDriver if you encounter any issues.
-
The scripts were created and tested for a specific web application. It may require modifications to work with other applications.
-
The scripts may not work if the structure of the web application changes. In such cases, you may need to update the XPath expressions used to locate elements.
The CSV-Error-Analyzer is another Python script developed to automate the process of analyzing error occurrences in CSV data. This tool is complementary to the Data-Retrieval-Automation scripts and is used to process the downloaded CSV files.
In the daily workflow of professionals at Infozillion Teletech BD Ltd., dealing with CSV data is common. These CSV files often contain error codes, and analyzing these error occurrences manually can be time-consuming and error-prone.
The CSV-Error-Analyzer automates this process, allowing you to:
- Quickly identify and categorize error occurrences.
- Generate comprehensive error reports for further analysis.
- Improve productivity by eliminating repetitive, manual tasks.
By using this tool in combination with the Data-Retrieval-Automation scripts, you can enhance your error analysis capabilities and make more informed decisions based on the data.
-
Clone the CSV-Error-Analyzer repository to your local machine.
-
Follow the instructions in the CSV-Error-Analyzer README to install and run the tool.
-
Use the CSV-Error-Analyzer to process and analyze the CSV files downloaded by the Data-Retrieval-Automation script.
This project is licensed under the MIT License - see the LICENSE file for details.
- Iftekhar Tahir
- Email: iftekhar.tahir@proton.me
- GitHub: https://github.com/iftekharmickey
- LinkedIn: https://www.linkedin.com/in/iftekharmickey/
These scripts have been created and tested for use with a specific web application with appropriate permissions. The author and contributors are not responsible for any misuse of the tools. It is essential to use these tools only with prior permission and in compliance with all relevant laws and regulations.
Please ensure that you have the necessary authorizations to access and automate interactions with web applications before using these scripts.