In this course, we will learn some advanced machine learning methods in the context of natural language processing (NLP) applications. Markov chains, hidden Markov models, topic modeling, recurrent neural networks.
- Carry out basic text preprocessing.
- Specify Markov chains and carry out generation and inference with them.
- Explain the general idea of stationary distribution in Markov chains.
- Explain what are hidden Markov models and carry out inference and decoding with them.
- Broadly explain Latent Dirichlet Allocation (LDA) approach to topic modeling and carry out topic modeling on text data.
- Broadly explain Recurrent Neural Networks (RNNs) and different types of RNNs.
- Broadly explain the idea of self-attention and transformers
Click to expand!
The following deliverables will determine your course grade:
| Assessment | Weight | Where to submit |
|---|---|---|
| Lab Assignment 1 | 15% | Gradescope |
| Lab Assignment 2 | 15% | Gradescope |
| Lab Assignment 3 | 15% | Gradescope |
| Lab Assignment 4 | 15% | Gradescope |
| Quiz 1 | 20% | Canvas |
| Quiz 2 | 20% | Canvas |
See Calendar for the due dates.
Click to expand!
| Role | Name |
|---|---|
| Lecture instructor | Varada Kolhatkar |
| Lab instructor | Varada Kolhatkar and Vincent Liu |
| Teaching assistant | Daniel Ramandi |
| Teaching assistant | Harsh Sharma |
| Teaching assistant | Mohit Pandey |
| Teaching assistant | Ngoc Bui |
This course will be run in person. We will meet three times every week: twice for lectures and once for the lab. You can refer to the Calendar for lecture and lab times and locations. Lectures of this course will be a combination traditional live lecturing, class activities, and pre-recorded videos. Drafts of the lecture notes for each week will be made available earlier in the week.
This course occurs during Block 6 in the 2023/24 school year.
| Lecture | Topic | Assigned videos/Readings | Resources and optional readings |
|---|---|---|---|
| 0 | Course Information | 📹 |
|
| 1 | Markov Models | ||
| 2 | Language models, PageRank, text preprocessing | 📹 |
|
| 3 | Hidden Markov models | ||
| 4 | HMMs decoding and inference) | (optional) HMM Baum-Welch (unlisted) | |
| 5 | Topic modeling | ||
| 6 | Introduction to Recurrent Neural Networks (RNNs) | ||
| 7 | Introduction to Transformers | ||
| 8 | Applications of Transformers |
The labs are going to be in person. We will also be holding a short 1-hour parallel Zoom session for each lab run by the TAs so that people who cannot join in person have an opportunity to ask questions and get help. You will be able to access appropriate Zoom links via Canvas. There will be a lot of opportunity for discussion and getting help during lab sessions. Please make good use of this time.
We are providing you with a conda environment file which is available here. You can download this file and create a conda environment for the course and activate it as follows.
conda env create -f env-dsci-575.yml
conda activate 575
In order to use this environment in Jupyter, you will have to install nb_conda_kernels in the environment where you have installed Jupyter (typically the base environment). You will then be able to select this new environment in Jupyter. If you're unable to see the environment in Jupyter, you might have to install the kernel manually. See the documentation here. For more details on this, refer to your 521 lecture 7.
I've only attempted to install this environment file on a few machines, and you may encounter issues with certain packages from the yaml file when executing the commands above. This is not uncommon and may suggest that the specified package version is not yet available for your operating system via conda. When this occurs, you have a couple of options:
- Modify the local version of the
yamlfile to remove the line containing that package. - Create the environment without that package.
- Activate the environment and install the package manually either with
conda installorpip installin the environment.
Note that this is not a complete list of the packages we'll be using in the course and there might be a few packages you will be installing using conda install later in the course. But this is a good enough list to get you started.
Click to expand!
We all are here to help you learn and succeed in the course and the program. Here is how we'll be communicating with each other during the course.
If there is any clarification on the lecture material or lab questions, I'll post a message on the course Slack channel and tag you. It is your responsibility to read the messages whenever you are tagged. (I know that there are too many things for you to keep track of. You do not have to read all the messages but please make sure to carefully read the messages whenever you are tagged.)
If you have questions about the lecture material or lab questions please post them on the course Slack channel rather than direct messaging me or the TAs. Here are the advantages of doing so:
- You'll get a quicker response.
- Your classmates will benefit from the discussion.
When you ask your question on the course channel, please avoid tagging the instructor unless it's specific for the instructor (e.g., if you notice some mistake in the lecture notes). If you tag a specific person, other teaching team members or your colleagues are discouraged to respond. This will decrease the response rate on the channel.
Please use some consistent convention when you ask questions on Slack to facilitate easy search for others or future you. For example, if you want to ask a question on Exercise 3.2 from Lab 1, start your post with the label lab1-ex2.3. Or if you have a question on lecture 2 material, start your post with the label lecture2. Once the question is answered/solved, you can add "(solved)" tag before the label (e.g., (solved) lab1-ex2.3). Do not delete your post even if you figure out the answer on your own. The question and the discussion can still be beneficial to others.
For each deliverable, after I return grades, I'll let you know who has graded what in our course Slack by opening an issue in the course GitHub repository. If you have questions related to grading
- First, make sure your concerns are reasonable (read the "Reasonable grading concerns" policy).
- If you believe that your request is reasonable, open a regrade request on Gradescope.
- If you are unable to resolve the issue with the TA, send a Slack message to the instructor, including the appropriate TA in the conversation.
I am open for a conversation with you. If you want to talk about anything sensitive, please direct message me on Slack (and tag me) rather than posting it on the course channel. It might take a while for me to get back to you, but I'll try my best to respond as soon as possible.
Click to expand!
- Google NLP API
- Stanford CS224d: Deep Learning for Natural Language Processing
- LDA2vec: Word Embeddings in Topic Models
- 7 Types of Artificial Neural Networks for Natural Language Processing
- https://distill.pub/
- Model-Based Machine Learning
- RNNs in TensorFlow, a practical guide and undocumented features
- A list of readings about RNNs
- For NLP in R, see Julia Silge's blog posts on sentiment analysis of Jane Austen novels: part 1, part 2, part 3, part 4.
- RNN resources
- Jurafsky, D., & Martin, J. H. Speech and language processing.
- Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1). Cambridge: MIT press.
- Jacob Eisenstein. Natural Language Processing
- Goldberg, Y. (2017). Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1), 1-309.
- Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python. O'Reilly Media, Inc.
Please see the general MDS policies.