caching problem with gluster

[enovationIT]

we started hitting errors under very small load caused by core_component.php being stored on gluster share

[2018-05-23 08:30:03.116483] W [MSGID: 109065] [dht-rename.c:1387:dht_rename_lock_cbk] 0-data-dht: acquiring inodelk failed rename (/moodledata/cache/core_component.php.tmp:5198d4b6-82fd-48a5-90f2-0813370dc81e:data-replicate-0 /moodledata/cache/core_component.php::(null)) [Stale file handle]
[2018-05-23 08:30:03.116582] W [fuse-bridge.c:1788:fuse_rename_cbk] 0-glusterfs-fuse: 581920: /moodledata/cache/core_component.php.tmp -> /moodledata/cache/core_component.php => -1 (Stale file handle)
[2018-05-23 09:57:55.228929] E [MSGID: 109040] [dht-helper.c:1198:dht_migration_complete_check_task] 0-data-dht: /moodledata/cache/core_component.php: failed to lookup the file on data-dht [Stale file handle]
[2018-05-23 09:57:55.229002] W [fuse-bridge.c:1355:fuse_unlink_cbk] 0-glusterfs-fuse: 586329: UNLINK() /moodledata/cache/core_component.php => -1 (No such file or directory)

changing it to local file system improved performance and got rid of the error

$CFG->alternative_component_cache = '/tmp/core_component.php';

[hosungsmsft]

Wow, we are currently struggling with perf deficiencies with gluster, and it's very surprising to see this issue posted at the same time. However, we didn't see this error under very small load, and it's mostly the high latency, not errors like this. Maybe I didn't dig enough to confirm this myself, but if errors like these occur, I suppose it'll manifest as HTTP failures, not as high latency. Still, very worth considering and improvement.

I think eventually we'll need to provide ways for users to tweak the various configs like this after the deployment. This config.php change can be done on the controller VM and I'll also try to see if this improves our current high latency gluster-based Moodle deployment. Thanks again for your contribution!

[hosungsmsft]

Update: We ourselves haven't experienced the No such file or directory issue (or maybe oblivious) with a high load test plan. We also tried the alternative_component_cache tip on a high load perf testing run, but it didn't help much (latency still very unacceptable). We've heard from the engaged partner that they observed some weird behavior with the Gluster file system, and I suspect this might be related to that symptom? We'll need some deeper understanding of Gluster and their current bugs/issues to figure this out.

[enovationIT]

Is Moodle configured with REDIS for application cache for your testing?

[SorraTheOrc]

In the environment @hosung refers to above we discovered Moodle 3.2 and earlier has a bug preventing effective use of Redis. We therefore disabled Redis. This and a number of other improvements (including using NFS rather than Gluster FS) resulted in good results.

For Moodle 3.3+ Redis works.

[enovationIT]

in Moodle config Redis is only set up to be used as session cache store and not application, I imagine it would make a significant difference, regarding the use of NFS, I understand you have a non-HA version?

[hosungsmsft]

@enovationIT -- Our install_moodle.sh configures redis only for session, and we didn't know redis can be used as application cache. We can certainly givea try, but I'm not that hopeful, unless that option can completely offload almost contentious gluster access.

Yes, currently NFS is non-HA, but we are investigating HA NFS using DRBD. Unfortunately, the DRBD kernel module is not built in the distributed Linux-azure kernels, so we are working with our kernel team to include that.

[hosungsmsft]

@enovationIT -- We'd still be interested in configuring redis for application cache. Could you provide us with how to do that? Thanks in advance!

[noveck]

https://docs.moodle.org/34/en/Redis_cache_store The Moodle docs mention that redis support was introduced in 3.2. I think it may be of significant value as an app cache rather than session only. I've yet to test it, though. It's on my agenda for my on-prem deployment.

…

On Wed, May 30, 2018, 11:20 AM Hosung Song ***@***.***> wrote: @enovationIT <https://github.com/enovationIT> -- We'd still be interested in configuring redis for application cache. Could you provide us with how to do that? Thanks in advance! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#126 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVdY_Ed1CJXxBQANM7KyW5Oszp7S6Yks5t3rjIgaJpZM4UMf7h> .