Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement periodic sync command and functions. #297

Merged
merged 3 commits into from
May 31, 2024

Conversation

rochacbruno
Copy link
Member

@rochacbruno rochacbruno commented Apr 9, 2024

Introduces resource_registry/tasks/sync.py and resource_sync management command.

Depends on: #311 and resource_server#351

Config on django.conf.settings looks like

RESOURCE_SERVER = {
    "URL": "https://localhost",
    "SECRET_KEY": "<VERY-SECRET-KEY-HERE>",
    "VALIDATE_HTTPS": False,
}
RESOURCE_JWT_USER_ID = "97447387-8596-404f-b0d0-6429b04c8d22"
RESOURCE_SERVICE_PATH = "/api/server/v1/service-index/"
$ django-admin resource_sync --asyncio

----- RESOURCE SYNC STARTED -----

>>> shared.organization
Deleting 1 orphaned resources
Processing 2 resources with asyncio executor.
CREATED 3e3cc6a4-72fa-43ec-9e17-76ae5a3846ca Acme
NOOP 3e3cc6a4-72fa-43ec-9e17-76ae5a389999
Processed 3 | Created 1 | Updated 0 | Conflict 0 | Unavailable 0 | Skipped 1 | Deleted 1

>>> shared.team
Processing 1 resources with asyncio executor.

NOOP f43938cf-a618-4a73-bc90-922a6b217e4d
CREATED f43938cf-a618-4a73-bc90-922a6b28888
Processed 2 | Created 1 | Updated 0 | Conflict 0 | Unavailable 0 | Skipped 1 | Deleted 0

>>> shared.user
Processing 4 resources with asyncio executor.

UPDATED 31daab14-cb67-4c62-8dcd-39f411c82242 joe
NOOP 97447387-8596-404f-b0d0-6429b04c8d22
NOOP 4d01427e-a11e-4d0e-9408-4ef7c2b478d6
CONFLICT 53886798-29e7-426f-b9cb-f7f74b346072 mary
Processed 4 | Created 0 | Updated 1 | Conflict 1 | Unavailable 0 | Skipped 2 | Deleted 0

----- RESOURCE SYNC FINISHED -----

OR run in a task queue such as Celery, RQ or Pulp Task

from celery import Celery
from celery.schedules import crontab
from ansible_base.resource_registry.tasks.sync import SyncExecutor

# Initialize Celery app
celery = Celery('your_app_name', broker='redis://localhost:6379/0')

# Define Celery task
@celery.task
def execute_sync_executor():
    SyncExecutor(
        asyncio=False,
        api_client=ResourceAPIClient(....),  # Omit to have a default client created
        retries=3,
        retrysleep=60,
        retain_seconds=300,  # Avoid delete resources created in the latest 5 minutes
        stdout=sys.stdout  # omit to silence it
    ).run()

# Schedule the task to run every 15 minutes
celery.conf.beat_schedule = {
    'execute-every-15-minutes': {
        'task': 'your_module.execute_sync_executor',
        'schedule': crontab(minute='*/15'),
    },
}

@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 2 times, most recently from 8abb9c6 to ac2e6cf Compare April 9, 2024 19:27
@john-westcott-iv john-westcott-iv changed the title Implement periodic sync command and functions. [WIP] Implement periodic sync command and functions. Apr 11, 2024
@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 2 times, most recently from ffc4363 to 52f2c94 Compare April 16, 2024 21:41
@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 4 times, most recently from 24c20d3 to d3923e7 Compare May 16, 2024 16:16
@rochacbruno rochacbruno changed the title [WIP] Implement periodic sync command and functions. Implement periodic sync command and functions. May 16, 2024
@rochacbruno rochacbruno marked this pull request as ready for review May 16, 2024 16:17
@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 3 times, most recently from f60b579 to 5fd9b3e Compare May 17, 2024 15:42
@rochacbruno rochacbruno added the Ready for review This PR is ready for review either initially or comments have been address label May 17, 2024
@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 3 times, most recently from 4ddf889 to 3d67e79 Compare May 20, 2024 13:36
@john-westcott-iv john-westcott-iv added the question Further information is requested label May 23, 2024
@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 2 times, most recently from 48e7c88 to 30cf650 Compare May 27, 2024 16:24
@rochacbruno rochacbruno force-pushed the aap-20922-resource-sync branch 2 times, most recently from 8522664 to e1da4aa Compare May 29, 2024 19:57
Copy link
Member

@AlanCoding AlanCoding left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for responding to comments.

Introduces `resource_registry/tasks/sync.py` and `resource_sync` management command.
Changes as requested by reviewer
Copy link

sonarcloud bot commented May 31, 2024

@rochacbruno rochacbruno merged commit 8946815 into ansible:devel May 31, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested Ready for review This PR is ready for review either initially or comments have been address
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants