fix:start Celery with a Redis cluster not in the same slot (#4472)

[vicwang3]

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #4472

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update, included: Dify Document
• Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• Dependency upgrade

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• TODO

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods
• optional I have made corresponding changes to the documentation
• optional I have added tests that prove my fix is effective or that my feature works
• optional New and existing unit tests pass locally with my changes

[bowenliang123]

Some questions for further investigation:

1. how the global_keyprefix config of celery helps to fix the slot problem in Redis cluster concepts.
2. What's {celery_global_prefix} in the config example? Any reference explanation for the usage ? Is that template holder in some way?

[binarx2]

The configuration option should be result_backend_transport_options ?

https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html#global-keyprefix

[vicwang3]

Some questions for further investigation:

1. how the global_keyprefix config of celery helps to fix the slot problem in Redis cluster concepts.
2. What's {celery_global_prefix} in the config example? Any reference explanation for the usage ? Is that template holder in some way?

The root of the issue is that in Redis cluster mode, keys are distributed across different nodes, and the cluster requires that all keys involved in a specific operation must be located within the same slot since such operations are designed to be executed atomically on a single node.
Solution
Utilizing Redis’s hash tag feature when defining keys for the message queue can ensure that related keys are assigned to the same slot. Users can customize the hash tag CELERY_GLOBAL_PREFIX={celery_global_prefix} in the .env configuration. When operating on REDIS, CELERY_GLOBAL_PREFIX will automatically be used as a prefix. Redis will then assign keys with the same hash tag to the same SLOT.

[crazywoola]

This one can be reviewed as well.
#4520