Explore a variety of tutorials and interactive demonstrations focused on Big Data technologies like Hadoop, Spark, and more, primarily presented in the format of Jupyter notebooks.
- Hadoop_Setting_up_a_Single_Node_Cluster.ipynb
Set up a single-node Hadoop cluster on Google Colab and run some basic HDFS and MapReduce examples
- Hadoop_single_node_cluster_setup_Python.ipynb
- Hadoop_Setting_up_Spark_Standalone_on_Google_Colab.ipynb
- Run_Spark_on_Google_Colab.ipynb
- PySpark_On_Google_Colab.ipynb
- MapReduce_Primer_HelloWorld.ipynb
- MapReduce_Primer_HelloWorld_bash.ipynb
- mapreduce_with_bash.ipynb An introduction to MapReduce using MapReduce Streaming and bash to create mapper and reducer
- simplest_mapreduce_bash_wordcount.ipynb A very basic MapReduce wordcount example
- mrjob_wordcount.ipynb A simple MapReduce job with mrjob
- Hadoop_spilling.ipynb Hadoop spilling explained
- demoSparkSQLPython.ipynb Pyspark basic demo
- ngrams_with_pyspark.ipynb
- Encoding+dataframe+columns.ipynb
- GutenbergBooks.ipynb
- generate_data_with_Faker.ipynb
- TestDFSio.ipynb Demo of TestDFSio for benchmarking Hadoop clusters
- Unicode.ipynb Exploring Unicode categories (
)
- polynomial_regression.ipynb Worked out example of polynomial regression with numpy
- Apache_Sedona_with_PySpark.ipynb
- docker_for_beginners.md Docker for beginners: an introduction to the world of containers
- Terraform for beginners.md Getting started with Terraform
- Terraform in 5 minutes A short introduction to Terraform, the powerful and popular tool for infrastructure provisioning and management
- online_resources.md Online resources for learning Big Data
Most executable Jupyter notebooks are tested on an Ubuntu virtual machine through a GitHub automated workflow. The log file for successful executions is named: action_log.txt.
Current status: 
The Github workflow is a starting point for what is known as Continuous Integration (CI) in DevOps/Platform Engineering circles.