[#5228] Quota use django signals instead of irods microservice and AVUs #5461

devincowan · 2024-04-18T18:45:21Z

This PR alters are quota system to remove reliance on the iRods microservices.
More info is available in
https://github.com/hydroshare/hydroshare/blob/5228-remove-irods-quota-avus/quota-info.md

This PR attempts to "capture" all relevant file operations using Django signals.
Each signal triggers an asynchronous call to update_quota_usage

Resolves #5228 in conjunction with #5229
Relies on #5443 in order to get accurate file sizes from irods storage
Resolves #5418 by entirely removing the need for irods microservice in the dataZone
Resolves #5235 because removing the irods microservice will vastly improve performance of filesystem operations
Resolves #5441 because quota will be refreshed before sending emails and we can be confident in correct values
Resolves #5384 because we don't need a microservice anymore
Resolves #5195 because we don't need a microservice anymore
Resolves #5286 because we don't need a microservice anymore

Pull Request Checklist:

Positive Test Case Written by Dev

Automated Testing

Sufficient User and Developer Documentation

Passing Jenkins Build

Peer Code review and approval

Positive Test Case

Suggest that you open a private window and login as admin, then open the userQuotas model in the django admin panel
Login as non-admin user in the non-private window
With that user, conduct the following operations

uploading a file to a resource
creating a resource
copying a resource
versioning a resource
transferring ownership of a resource to "user"
ingesting a bag from the userZone
zipping a file without removing the original
unzipping a file
creating aggregations
moving files into subdirectories

In all cases, verify that the user's quota (viewed in the private browser, admin panel) adjusts in the expected way, either increasing, decreasing, or remaining the same.

devincowan · 2024-04-18T18:48:21Z

hs_core/hydroshare/resource.py

        if not quota_user == user:
            continue
        aggregate_size += res.size
    return aggregate_size


-def update_quota_usage(username):


moved this to hs_core.tasks

devincowan · 2024-04-18T18:49:02Z

hs_core/migrations/0080_baseresource_quota_holder.py

+    operations = [
+        migrations.AddField(
+            model_name='baseresource',
+            name='quota_holder',


store the quota_holder on the resource model (was previously stored in AVU)

devincowan · 2024-04-18T18:49:33Z

hs_core/receivers.py

+@receiver(post_copy_resource)
+@receiver(post_version_resource)
+@receiver(post_publish_resource)
+def hs_update_quota(sender, **kwargs):


use signals to asyc update the stored quota sizes

devincowan · 2024-04-18T18:50:27Z

theme/management/commands/migrate_quota_holder_from_irods.py

+
+
+class Command(BaseCommand):
+    help = "Migrate quota_holder info from irods AVU into Django UserQuota model."


command to migrate the quota_holder metadata from irods AVU into django resource model

devincowan · 2024-04-19T17:32:24Z

Looks like I have a test failing
http://ci.hydroshare.org:8080/job/hydroshare-pull-requests/7502/testReport/junit/hs_core.tests.api.native.test_irods_user_zone_federation/TestUserZoneIRODSFederation/test_resource_operations_in_user_zone/

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/lib/python3.9/unittest/case.py", line 592, in run
    self._callTestMethod(testMethod)
  File "/usr/local/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/hydroshare/hs_core/tests/api/native/test_irods_user_zone_federation.py", line 118, in test_resource_operations_in_user_zone
    self.assertEqual(BaseResource.objects.all().count(), 0,
  File "/usr/local/lib/python3.9/unittest/case.py", line 837, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/lib/python3.9/unittest/case.py", line 830, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 0 : Number of resources not equal to 0

I'm guessing it has to do with a race condition because the test passes when run locally.
Perhaps something about the post_delete_resource signal...?
https://github.com/hydroshare/hydroshare/blob/5228-remove-irods-quota-avus/hs_core/tasks.py#L1123-L1125

Testing that theory here:
#5463
and here
#5464

devincowan · 2024-04-29T14:28:31Z

I believe this django signals work will be replaced by #5469

devincowan · 2024-05-06T14:32:07Z

closing in favor of #5469

devincowan added 15 commits April 18, 2024 14:44

[#5228] store quota-holder in resource model

767a502

[#5228] Asynchronously update_quota_usage without using irods datazone

7891519

fix broken import

5eff59e

Fix updating quota holder in AbstractResource model

9d095c7

[#5228] Update tests and add management commands

044a7a9

[#5228] Fix resource storage usage calculation

096dd78

[#5228] Always update quota usage before sending email notifications

04c1605

WIP tests

2d381fc

[#5228] add signals for updating quota usage

6904014

Refactor import statements in hs_core/hydroshare/utils.py

3385d5a

[#5228] WIP update quota tests

63b1045

[#5228] WIP more tests

6ebb881

[#5228] adding signals for copy, versioning, publishing resource

940e58d

[#5228] fix unzip signal args

5cd6970

[#5228] send signal on system_metadata instead of post add file

9f3bb13

devincowan mentioned this pull request Apr 18, 2024

5228-remove-irods-dz-quota #5458

Closed

5 tasks

devincowan changed the title ~~[#5228] send signal on system_metadata instead of post add file~~ [#5228] Quota use django signals instead of irods microservice and AVUs Apr 18, 2024

devincowan commented Apr 18, 2024

View reviewed changes

[#5228] add documentation

ab295e8

This was linked to issues Apr 18, 2024

iRods microservice does not catch ibun (zip/unzip) actions #5418

Closed

Duration to delete directory increases exponentially based on # of files #5235

Closed

devincowan mentioned this pull request Apr 18, 2024

[#5228] hard quota enforcement #5229

Merged

5 tasks

devincowan requested a review from pkdash April 18, 2024 19:31

This was referenced Apr 18, 2024

[#5441] Only send quota emails if quota enforcement is on #5442

Closed

Only send quota emails if confident that quota sizes are correct #5441

Closed

devincowan linked an issue Apr 18, 2024 that may be closed by this pull request

Only send quota emails if confident that quota sizes are correct #5441

Closed

martinseul added this to In Progress in HydroShare Issue Management Board via automation Apr 18, 2024

martinseul moved this from In Progress to In Review in HydroShare Issue Management Board Apr 18, 2024

martinseul removed this from In Review in HydroShare Issue Management Board Apr 18, 2024

martinseul added this to In Progress in HydroShare Issue Management Board via automation Apr 18, 2024

martinseul removed this from In Progress in HydroShare Issue Management Board Apr 18, 2024

This was linked to issues Apr 18, 2024

iRods microservice updates #5384

Closed

Migrate iRODS quota microservice to Python API #5195

Closed

Explicitly specify rule engine on irule invocations #5286

Closed

devincowan force-pushed the 5228-remove-irods-quota-avus branch from 659eb72 to ab295e8 Compare April 19, 2024 17:20

devincowan linked an issue Apr 22, 2024 that may be closed by this pull request

Enforce file quota #5228

Closed

38 tasks

[#5228] add the ability to bypass AVU query on every quota update

92c5fce

devincowan closed this May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[#5228] Quota use django signals instead of irods microservice and AVUs #5461

[#5228] Quota use django signals instead of irods microservice and AVUs #5461

devincowan commented Apr 18, 2024 •

edited

devincowan Apr 18, 2024

devincowan Apr 18, 2024

devincowan Apr 18, 2024

devincowan Apr 18, 2024

devincowan commented Apr 19, 2024 •

edited

devincowan commented Apr 29, 2024

devincowan commented May 6, 2024



		class Command(BaseCommand):
		help = "Migrate quota_holder info from irods AVU into Django UserQuota model."

[#5228] Quota use django signals instead of irods microservice and AVUs #5461

[#5228] Quota use django signals instead of irods microservice and AVUs #5461

Conversation

devincowan commented Apr 18, 2024 • edited

Pull Request Checklist:

Positive Test Case

devincowan Apr 18, 2024

Choose a reason for hiding this comment

devincowan Apr 18, 2024

Choose a reason for hiding this comment

devincowan Apr 18, 2024

Choose a reason for hiding this comment

devincowan Apr 18, 2024

Choose a reason for hiding this comment

devincowan commented Apr 19, 2024 • edited

devincowan commented Apr 29, 2024

devincowan commented May 6, 2024

devincowan commented Apr 18, 2024 •

edited

devincowan commented Apr 19, 2024 •

edited