Reduce App Size with API to Send Requirement Data to Clients

[noschiff]

All of our requirement computation is done on the front-end. In order to build the requirement fulfillment graphs, our app needs to know which courses will count for which requirements. Currently, we use a script req-gen to take the information from our major, minor, and colleges typescript files (see cs.ts for an example), determine which courses fulfill the requirements in those files, and put all that information into a big JSON file: decorated-requirements.json. This JSON has a mapping from every requirement we handle to a list of courses that fulfill that requirement. As a result, it's quick to determine if a course fulfills a requirement on the front-end.

However, this JSON is incredibly large—currently 4.83 MB. The information in this JSON gets directly embedded into our compiled javascript script used to run the vue app. The entire JSON is basically verbatim in that file. When a user goes to courseplan for the first time, the browser has to download this file, which ends up being about 6 MB with the code used to run the vue app (see the screenshot at the bottom of this issue).

It's unnecessary to send the entire JSON to every client because the client only needs information about the requirements relevant to a user's majors, minors, and college. Most of the information in the JSON is unused by a given client since the client will only ever check the data for the user's necessary requirements. For example, a user who is majoring in computer science does not need to know which classes will fulfill the requirements for a history major. We should try to avoid sending unnecessary information because it slows the website load and requires the browser to store extra data.

Each client only needs to know about the requirements relevant to its user. So, we should only provide the necessary information. It's impossible to know what information each user needs before the user loads the website, so we must wait for the user to open courseplan before sending the data. This requires a backend and API for our front-end to use, and we can create this without too much additional setup with Firebase Cloud Functions since we already use Firebase for hosting our app. We can create a cloud function that responds to an API request from a client with the information about only the user's requirements. See this website about creating a function that's callable from your app. We can secure the function and prevent abuse with App Check. I believe that we could also only secure the function to only accept requests about a user's requirements if it's coming from the user, but it might just be easier to take in the desired requirements as a parameter for the function instead of searching through Firestore to determine which requirements should be sent. On the first load of the app and every time the user updates their college/majors/minors, we could resend this information. We could only send information for the new college/majors/minors if we wanted, but we can consider that optimization later.

This cloud function could "filter" the large JSON on the backend and send only the relevant information, or we could skip the creation of a JSON at all build on the requirement-json-generator.ts script to find the courses that fulfill the requested requirements on the fly. The latter would allow us to modify the .ts data files directly without having to run req-gen every PR, but it might be slower than the former. We can try both and see which is faster. Either way, sending only the relevant information from a backend would allow us to send significantly less data to the client. As a result, we become freer to improve our requirements data structure to handle more cases, such as majors whose requirements differ by year and more edge cases, without having to worry about the size of our JSON. Overall, we'll hopefully have a better user experience because of a smaller web app and more flexibility to handle every requirement without worrying about slowing the app further.

Files sent to the client

Size of index.js

[benjamin-shen]

See CoursePlan's initial backend (from Fall 2019) here: https://github.com/cornell-dti/course-plan-requirements