These notes, talk slides and code samples are part of an on-going investigation at the nexus of personal data (i.e. from self-tracking, QS, service logs) and working with that data with Python (data collection, data processing, data analysis, data visualization and machine learning).
My overall aim is to help people to gain better self-understanding and empowered self-improvement through personal data.
This is an active work-in-progress, and there are currently two main presentations.
See below for Previous Versions and Related Writings.
In this presentation we will look at how to track a life and how to collect, analyze and visualize that data with python. In the first part, we will look briefly at different kinds of self-tracking and quantified self technologies. Second, using a collection of python notebook from QS Ledger, I'll walk you how to do analysis and create data visualizations of your own personal data!
Slides: https://rawgit.com/markwk/python4selftrackers/master/year-in-data-with-python/slides.html
From Data Collection and Data Processing to Data Analysis and Data Visualization to Machine Learning and Deep Learning with Your Personal Tracking Data
My purpose is help people to learn how to collect and use their personal data to better understand and improve yourself.
Curious how you are being tracking? Already like many people today tracking your life but wondering what more you can do with your personal data?
It’s easier than ever to track our lives, work and bodies with a smartphone, wearable, home sensor or computer. If you have an Android or iPhone, use a wearable like Fitbit, or even just watch videos on Youtube, then you already got plenty of personal data, and we haven't even started to formally track our lives.
With a world of data, the question becomes: What can we do with all this tracking data? Can it help you become a "better" version of yourself?
There are a lot of great ways to work with and use your personal data, even spreadsheet applications like Excel and Google Sheets and more powerful data visualization tools like Tableau and Google Data Studio can help.
But if you want to become a master of your personal data, then one of the best tools is Python. Python is a great tool for not only aggregating, collecting and cleaning up your data. But python enables you to do robust data analysis and machine learning too too.
In this talk / workshop, we are exploring personal data, self-tracking, and personal data analysis. In the first part, I’ll show the basics on you how to track your life and all about the so-called "quantified self." In the second part on QS Ledger, I'll explain how to use Python and its data science toolkit (Pandas, Matplot, Seaborn, etc.) for data collection and data processing. Part 3 looks at how to do your own Data Exploration, Analysis and Visualization on your own tracking data. Through some simple code, you can even create your own Personal Data Dashboard. Finally, in Part 4, we look at Machine Learning. We look at where ML is typically used for personal data and some examples of how you can use ML to better understand and improve yourself. In the conclusion, I share some of tips and recommendations for anyone wanting to pursue a data-driven life.
Bring together your personal data in interesting ways. Clean and process it. Find patterns with some simple data analysis. Make it beautiful with data visualization. Discover meaning, insights, and correlations using predictive analytics and machine learning. Finally, improve your life by transforming personal data into a data-driven life.
In short, can self-tracking and personal data help us become more productive, healthier, and happier? I think it can. Learn how to engage with your data to be smarter, make more informed decisions and become data-driven!
- Part 1: QS / Self-Tracking: How to measure a life?
- Part 2: Data Collection / Extraction / Processing of Personal Data with Python and QS Ledger
- Part 3: Data Exploration and Data Viz with Python and Tableau
- Part 4: Machine Learning for QS and Personal Tracking Data
- Conclusion: Tips on How to Become a Data-Driven You
- June 2018: PyCon in Kuala Lumpur
- June 17, 2018: PyCon in Bangkok, Thailand
- June 2, 2018: PyCon APAC in Singapore
Also see:
- How I Built A Photo Tracking App with React Native (January 24, 2018, Singapore JS Meetup)
- A Self Across Time: Time Seres Data Analysis with Python (PYCON MALAYSIA, August 24, 2019)
- How to Track a Life: Ultimate Guide of Tools, Apps and Techniques for Self-Tracking
- How I Tracked a Year in Time and What It Meant
- "Tracking Everything" Blog Series
Mark Koester (@markwkoester) is a self-tracker, writer, and web, mobile app and data developer. Creator: PhotoStatsApp, a photo tracking app without the cloud, PodcastTracker, a web app to log your podcast listening, and Biomarker Tracker, a health analytics service to better understand your blood test results. He currently runs a boutique dev shop (Int3c.com) and is an active open source contributor, including most recently QS Ledger, a Python data science project to collect and analyze personal tracking data. Former Regional Lead in Greater China at Techstars Accelerator. He regularly writes about self-tracking, quantified self and data-driven life at www.markwk.com.
- email: mark@int3c.com
- linkedin: https://www.linkedin.com/in/markwkoester
- website: http://www.markwk.com
- github: https://github.com/markwk