create clusters based on most similar course descriptions #415
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[diff-counting] Significant lines: 140. |
michelleli01
requested changes
Nov 13, 2023
course-clusters/clusters.py
Outdated
Comment on lines
5
to
6
| subjects_url = "https://classes.cornell.edu/api/2.0/config/subjects.json?roster=FA23" | ||
| subjects = getSubjects(subjects_url) |
There was a problem hiding this comment.
Can we move this section into a new file and import the subjects variable.
course-clusters/clusters.py
Outdated
Comment on lines
8
to
18
| urls = [f"https://classes.cornell.edu/api/2.0/search/classes.json?roster=FA23&subject={sub}" for sub in subjects] | ||
|
|
||
| def fetchDescriptions(urls): | ||
| descriptions = [] | ||
| for x in urls: | ||
| descriptions += loadCourseDesc(x) | ||
| print(len(descriptions)) | ||
| return descriptions | ||
|
|
||
|
|
||
| course_descriptions = fetchDescriptions(urls) |
There was a problem hiding this comment.
Can we also move this to a separate file? We can add the getSubjects and fetchDescriptions functions into a separate scrapper.py file.
course-clusters/clusters.py
Outdated
Comment on lines
19
to
44
| def preprocess_text(text): | ||
| """ Tokenization and preprocessing with custom stopwords""" | ||
| custom_stopwords = ["of", "the", "in", "and", "on", "an", "a", "to"] | ||
| strong_words = ["technology","calculus","business", "Artificial Intelligence", "First-Year Writing", "computer","python","java","economics","US","writing","biology","chemistry", "physics", "engineering","ancient" | ||
| "programming", "algorithms", "data structures","art","software","anthropology" "databases","fiction","mathematics", "history","civilization"] | ||
|
|
||
| translator = str.maketrans("", "", string.punctuation) | ||
| text = text.lower() | ||
| text = text.translate(translator) | ||
| tokens = text.split() | ||
| tokens = [token for token in tokens if token not in custom_stopwords] | ||
| for word in strong_words: | ||
| if word in tokens: | ||
| tokens += [word] * 10 | ||
|
|
||
| return " ".join(tokens) | ||
|
|
||
| def removeStopwords(): | ||
| "Removes stopwords for all the descriptions " | ||
| preprocessed = [] | ||
| for desc in course_descriptions: | ||
| if desc: | ||
| preprocessed.append(preprocess_text(desc)) | ||
| return preprocessed | ||
|
|
||
| preprocessed_descriptions = removeStopwords() |
There was a problem hiding this comment.
Let's also move this preprocessing out. Separate the preprocessing/web scrapping with the actual model itself. Can we also create some persistent objects to store the web scrapping data and the preprocessing so we don't always have to do that?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Created clusters based on the similarity of course descriptions, using a library called genism to do those calculations for us.
Clusters are based on all courses offered in FA23.
Each cluster has 50 coursed inside, or lightly more if an outlier was merged inside. Also currently have 87 clusters.
NOTES
currently courses that don't have a course description aren't accounted for which is 97 courses