🌟 Hit star button to save this repo in your profile
The information on this Github is part of the materials for the subject High Performance Data Processing (SECP3133). This folder contains general big data information as well as big data case studies using Malaysian datasets. This case study was created by a Bachelor of Computer Science (Data Engineering), Universiti Teknologi Malaysia student.
- Essential Preparations for a Successful Start in High-Performance Data Processing Class
- Course Information
- Student information
- Exercise
| No | Module | Description | File | Submission |
|---|---|---|---|---|
| 1 | Task 1 | Portfolio Creation |  |
 |
| 2 | Task 2 | AWS Certification |  |
 |
| 3. | Assignment 1 | Data analysis using Google Sheets |  |
 |
| 4. | Assignment 2 | Exploratory data analysis |  |
 |
| 5. | Assignment 3 | EDA Big Data | |
 |
| 6. | Course | Feature Engineering | |
 |
| 7. | Assignment 4 | Feature Engineering | |
 |
| 8. | Assignment 5 | Automated Feature Engineering tools | |
|
| 9. | Assignment 6 | Mastering Big Data Handling |  |
|
| 10. | Assignment 7 | Comparison between libraries | |
|
| No | Module | Description | File |
|---|---|---|---|
| 1. | Lab 1 | Understanding Your Data | |
| 2. | Lab 2 | EDA Big Data | |
| 3. | Lab 3 | Feature Engineering | |
| No. | Content | File |
|---|---|---|
| 1. | Charting Your Path in Data and Machine Learning | |
| 2. | Navigating the Data Science Landscape | |
| 3. | Database Types | |
| 4. | Navigating the Database Landscape | |
| 5. | The Data Journey: From Raw to Refined | |
| 6. | The Data to MLOps Journey: An End-to-End Process | |
| 7. | Data Platforms Architecture: Governance and Operations | |
| 8. | Creating Data Products to Monetize Data | |
| 9. | Revolutionizing Data and Machine Learning with DataOps and MLOps | |
| 10. | Data Science for Beginners - A Curriculum | |
| No. | Content | File |
|---|---|---|
| 1. | Unveiling Instagram's Engagement Magic through Machine Learning | |
| 2. | Unlocking Spotify's Musical Enchantment with Machine Learning | |
| 3. | Netflix | |
| 4. | Big data in healthcare: management, analysis and future prospects | |
| 5. | Introduction to Big Data Computing for Geospatial Applications | |
| 6. | 40 Stats and Real-Life Examples of How Companies Use Big Data | |
| 7. | Leveraging High Performance Data Processing: Insights from Industry Leaders | |
| No. | Section | Content |
|---|---|---|
| 1 | Introduction to Big Data Management | A. What is Big Data? B. The Importance of Managing Big Data C. Why Big Data Management is Important? D. The History of Big Data |
| 2 | Understanding Big Data | A. Defining Big Data B. Characteristics of Big Data C. Sources of Big Data D. Challenges in Dealing with Big Data E. The workflow of Big Data Management |
| 3 | The Role of Big Data Management | A. Managing big data B. Benefits of Effective Big Data Management C. Risks of Ignoring Big Data Management D. Industries Benefiting from Big Data Management |
| 4 | Data Collection and Storage | A. Data Collection Methods 1. Traditional Data Sources 2. Emerging Data Sources B. Data Storage and Warehousing C. Data Security and Privacy Concerns |
| 5 | Data Processing and Analysis | A. Data Preprocessing B. Data Analysis Tools and Techniques C. Real-time Data Processing D. Machine Learning in Big Data Analysis |
| 6 | Data Integration and Governance | A. Data Integration Strategies B. Data Governance Best Practices C. Ensuring Data Quality |
| 7 | Data Visualization | A. Importance of Data Visualization B. Data Visualization Tools C. Creating Effective Data Visualizations |
| 8 | Case Studies | A. Successful Big Data Management Implementations B. Lessons Learned from Failed Big Data Projects C. Real-world Examples |
| 9 | Challenges in Big Data Management | A. Scalability Issues B. Security Concerns C. Compliance and Regulatory Challenges |
| 10 | Future Trends in Big Data Management | A. The Evolving Landscape of Big Data B. Predictions for the Future C. Preparing for the Next Generation of Big Data |
| 11 | Best Practices in Big Data Management | A. Key Takeaways B. Actionable Strategies C. Tips for Effective Big Data Management |
| 12 | Other References |
- Awesome Public Datasets
- Portal Data Terbuka Malaysia
- Department of Statistics Malaysia
- data.world
- Dataportal.asia
- knoema
- The World Bank
- Dataset Search - Google
- UCI Machine Learning Repository
- Kaggle datasets
- Awesome-public-datasets
- Datahub.io
- Earthdata
- CERN Open Data Portal
Please create an Issue for any improvements, suggestions or errors in the content.
You can also contact me using Linkedin for any other queries or feedback.