Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arc-ce: Export arc-ce feature from aquilon at RAL #232

Draft
wants to merge 47 commits into
base: umd-4
Choose a base branch
from

Conversation

jrha
Copy link
Member

@jrha jrha commented Mar 5, 2020

As the title suggests, this is a lightly sanitised copy of the (ARC 5) CE feature used at RAL, which is itself a hurriedly transcribed copy of the configuration from our old SCDB instance.

Nevertheless it is provided here in the hope it may be of use to someone.

May resolve #221.

jrha added 30 commits March 5, 2020 11:00
This prevents the named user accounts performing actions at the HTCondor level.
Specify lists of central service nodes, submitters, users and worker nodes.
These groups are then used for priorities, fairshares and limits.
"Without the condor_shared_port daemon, approximately 2.1 ephemeral ports per running job are required, and possibly more, depending on the rate of job completion."
For compatibility with monitoring scripts and tooling that parses output directly
It is very improtant to clean these up automatically, otherwise disk space will disappear very quickly.
Very large output files are known to cause problems with grid frameworks, so we knobble them.
…g use

Add base of arc-ce feature - number of running & idle jobs by VO

Add arc-ce jobs config

Clean up jobs.pan

Add condorjobs.sh
* scaling_factors_plugin.py
* job_logs_to_s3.py
* default_rte_plugin.py
This contains an empty mapping for all DNs to nobody as we only want ARGUS providing these mappings.
jrha and others added 17 commits March 5, 2020 11:00
* Map queue name to the operating system version, we provide two images the jobs will run in - EL6 and EL7.
* Specify the tags (versions) of the containers to allow us to control testing and rollout of updated images.
* Make use of external data scraped for DockerHub by and hourly cron to validate that specified tags do actually exist.
Clean up service.pan

Add queues.conf

Add arc.conf
Clean up atlas-site-lcg.pan
Clean up biomed.pan
Clean up enmr.pan
Clean up glite.pan
Clean up proxy.pan
Clean up t2k.pan
Cleanup runtime-environments config
Move condor related file to subdirectory and update references - condor/limits
Move condor related file to subdirectory and update references - resourcelimits
@sartiran
Copy link
Collaborator

sartiran commented Mar 5, 2020

HI,
I comment this PR just because I'm preparing CondorCE templates at GRIF. I see plenty of stuff in features/arc-ce/condor. There are already features/htcondor template for configuring HTCondor with CREAM and (soon I hope) with CondorCE. Does it really make sense to have a separated set of HTCondor configuration templates?
Regards,
Andrea

@jrha
Copy link
Member Author

jrha commented Mar 5, 2020

Probably not, there is definitely a large number of things to be improved with these templates.

@jrha jrha added this to the 20.12 milestone Mar 19, 2020
@jrha jrha removed this from the 24.6 milestone Jun 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Support for new CE types (ARC CE, HTCONDOR CE)
2 participants