Technical design decisions

This page provides the elements that helped us shape Chamilo 2.*

Database structure

The Database structure of Chamilo 1.* has been modified over time but essentially matches the structure of the Claroline database back in 2001 with a major renaming in 2005-2006 (to make table names more meaningful) and a big refactoring in 2012 around the release of Chamilo 1.9 (to remove the one-database per course logic).

iid and a lacking implementation

One of the flaws Chamilo 1.10 and 1.11 (and all their minor versions) have is that we have defined, for all course-tool-related tables, a field called "iid" that uniquely identifies any resource in the whole table (meaning across all courses) whereas previous versions relied on a combination of the fields c_id + id to uniquely identify them (the id was specific to the course and the same "id" value could be found several times in the same table, only to be differentiated by the c_id).

This, in itself, was a good move: we can now identify each resource more efficiently through tables indexing of the iid field. However, that implementation was not completed, in that we didn't get rid of the use of the "id" field everywhere in the code: some code snippets still depend on id + c_id to find a resource. This has led to numerous issues being reported and some breaking changes when upgrading from one version to another. Why we didn't completely get rid of the "id + c_id" logic is entirely due to the lack of human resources to do all the changes needed before we released 1.10.0 and 1.11.0. Obviously, Chamilo 2.0 will get definitively rid of the id + c_id logic, to avoid pulling this issue along much longer.

iid rather than uid

If we wanted to use a new field as "unique id", why would we call it "iid" instead of "uid"? The answer boils down to the fact that we want Chamilo 2.0 (or later versions) to support many database systems, and that UID is a reserved keyword for Oracle.

Settings migration

Most settings were moved from app/config/configuration.php to the settings_current table through migration. To help understanding the migration process, we added this page with details of the process: Settings migration from 1 to 2.

Files management

It is important to note that, in Chamilo 2+, we differenciate files between assets (files with public, uncontrolled access) and resources (variable stuff with access control which imply management from the database).

It is also important to understand that the /assets directory at the root of Chamilo is not used to store uploaded files. Instead, it serves as a repository of mostly-public files that are shipped with Chamilo (most of the frontend part).

Instead, "assets" and "resources" that can be uploaded are all stored in /var/upload/{assets/resource}. The sections below mostly refer to "uploadable" files.

Uploadable Files structure

Files management has always been a bit of an issue in Chamilo. For efficiency and ease of use, we never wanted to put files as blobs inside the database. We believe the database should be fast even on small servers, and as such we prefer letting the filesystem deal with file reads and writes to adding the extra stress (to the database) of delivering high weight files to the remote user.

As such, files are stored on disk. However, our implementation has been... evolving over the years (Chamilo exists since 2010 and several in our team have worked on its ancestor Dokeos since at least 2004, making our team's total history inside the project more than 20 years old already).

For example, when we developed the social network inside Chamilo, we were faced with the issue that other files were always linked to a course, while files uploaded on the social network were not. As such, we decided (because course files were initially stored in a /courses directory at the root) to store them somewhere else. They ended up in /main/upload/users (/main is where the main code of the software lies), and then moved to /app/upload/users when the course files moved to /app/courses. In Chamilo 2, we have all uploadable files into some subfolder of /var.

The same can be said about the files used to illustrate the homepage, which initially ended up in the /home directory and later on in /app/home.

We did succeed in re-ordering it more cleanly over time, though (most user-uploaded files are in /app in Chamilo 1.11 and in /var in Chamilo 2.0), but we still had issues requiring other directories to be writeable (for example the CSS stylesheets uploaded afterwards are stored in /app/Resources/public/css/themes and copied to /web/css on cache refreshes in version 1.11).

In 1.11, for different reasons, several directories had to remain "writeable" by the web server: /app, /web/css, /main/default_course_document/images and optionally /main/lang, which complicated security rules and setting up a scaleable infrastructure.

In Chamilo 2+, this should be solved by having all web-uploadable files rest in /var (for "variable folder").

Files access method

Having a standardized method for files access (whereas currently we use several custom-developed methods to read or write files on disk) makes it easier to move to cloud-based infrastructures. Using a files management layer that supports REST-type services helps us deploy large scalable cloud systems that offer object files storage (like S3 in the case of AWS).

This is why we decided to use VichUploaderBundle starting from Chamilo 2.0. This uploader considers several naming strategies, called "Namers". Chamilo can sometimes have considerable amounts of files in a specific directory. For example, some of our customers have more than 80K courses, meaning the /app/courses/ directory gets filled with more than 80K entries, which is commonly considered a bad idea (because of inodes management limitations on Linux standard filesystems). Some others have more than 600K student directories, meaning that we could have 600K entries in the /app/upload/users/ directory. Luckily, we thought about that early and spread those directories by prefix of the user ID (only one level, so we can still have about 200K entries in one single directory in some cases).

As such, we need a strategy that will allow us to have an ideal spread of files so as not to clutter the filesystem. Using hashes is a practical way to do that, as files can be distributed randomly (and thus ideally) on the basis of the first characters of their hashes, although we know many administrators like to check stuff directly on disk (this would be an issue for them, which can be fixed by providing a command-line option to translate a hash into details of the file).

Having a naming convention that contains both a hash and a file name would be even better, at least when the filename represents something about the content itself (for example, a filename is not relevant in the case of an image file uploaded to illustrate some topic - it will usually rather be named by the context in which is was downloaded or generated, like a sequential number for a picture taken from a phone or camera).

Also, we want to avoid special characters in the original filename that would make it less manageable (Windows has some invisible spaces that are not the default ASCII invisible space, for example, and we've already had issues with several crazy stuff like that).

And this is exactly what VichUploaderBundle's SmartUniqueNamer namer offers.

So in Chamilo we're using the SmartUniqueNamer for files and the SubdirDirectoryNamer for some directories. The latter is only active when there is a potential for many files in a same directory. For example, for the user files (/app/upload/users/). The rest is natively spread enough to not be requiring any special subdirectorization. For example, if tasks ("student_publication" in Chamilo) can store many files in the context of sessions, they are already split by directories that contain the name of the task and the ID of the session, which highly reduces the probability of having a large number of files in the same directory.

Resources sorting and soft deletion

We are trying to make sorting of elements inside of lists a little more uniform in Chamilo 2. In Chamilo 1.*, some tables had a display_order or sort field which would give the order of the element on screen, when these elements can be sorted. For example, we allow learning paths and forums to be sorted manually, but not documents (yet). In this case, both the c_forum and c_lp tables had a display_order field.

But as we progress in time and want to generalize the idea of sorting elements manually, having a display_order in each table separately doesn't make a lot of sense. Even less so when you factor in the fact that we have a resource_node table that is a central reference to any resource on the system, and a resource_link table that indicates where each element is made available.

This last table in particular is just what we need, because when we define in which order elements should be shown, we define that in a context (in this course, in this session, in this group), which is already defined in the resource_link table.

The resource_link table thus now includes a display_order field that is essentially managed with Gedmo's "Sortable" and "SoftDeleteable" Doctrine extensions (Gedmo is a group of such extensions for Doctrine).

So now, when you delete a resource (this does not apply to things that are not resources yet, like survey questions), its resource_link.display_order is changed to the last item of the list, and the resource_link is marked with a deleted_at field that indicates that it was removed and when.

This is a "soft delete", as the resource itself is not deleted (rather, it goes to an internal "bin"), but disappears from all lists.

A special "bin" page will be made available soon to really delete items (or recover them) as needed.

Plugins as bundles

Coming soon(-ish)...