Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Experimental: update flux-pgrep to use RFC 35 constraint syntax #4915

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

grondo
Copy link
Contributor

@grondo grondo commented Feb 3, 2023

This experimental PR resurrects some of the flux-pgrep features from #4849 to allow a compete query interface.

There's still quite a bit to do here including settling on the exact syntax of some of the operators and tests, but I thought this might be useful in conjunction with the discussion in #4914, since this gives flux-pgrep the ability to generate the kinds of RFC 31 constraints we're discussing there.

In this PR, a new PgrepConstraintParser class is added as a subclass of ConstraintParser which can handle some of the specific syntax I'm proposing for flux pgrep. For instance @expr is syntactic sugar for "job was running at this time" and @dt1..dt2 translates to running between datetimes dt1 and dt2.

The rest of the operations are handled by the PgrepConstraint class, which compiles the RFC 31 specification into an object that can match JobInfo objects. The set of supported operations is at the top of the class definition. Any operation derived from Comparison allows values to be prefixed with < or -, > or + for greater than/less than and <= or -=, >= or += for less than or equal to/greather than or equal to. (the -=/+= syntax is provided to avoid the need for shell quoting). The range operator min..max can be used in a comparison to expand to min <= value <= max.

The current version has --debug and --dry-run options which can help experiment with the syntax, e.g.:

$ flux pgrep --dry-run --debug ^flux is:failed @'8am yesterday'
LexToken(TOKEN,'^flux',1,0)
LexToken(TOKEN,'is:failed',1,6)
LexToken(TOKEN,'@8am yesterday',1,17)
expression: '^flux' 'is:failed' (start:'<=8am yesterday' and end:'>=8am yesterday')
{"and": [{"name": ["^flux"]}, {"is": ["failed"]}, {"and": [{"t_run": ["<=2023-02-02 08:00:00-08:00"]}, {"t_cleanup": [">=2023-02-02 08:00:00-08:00"]}]}]}


$ flux pgrep --dry-run --debug not exception:cancel @"yesterday..today"
LexToken(NOT,'not',1,0)
LexToken(TOKEN,'exception:cancel',1,4)
LexToken(TOKEN,'@yesterday..today',1,21)
expression: not 'exception:cancel' (not ('end:<yesterday' or 'start:>today'))
{"and": [{"not": [{"exception": ["cancel"]}]}, {"not": [{"or": [{"t_cleanup": ["<2023-02-02 00:00:00-08:00"]}, {"t_run": [">2023-02-03 00:00:00-08:00"]}]}]}]}

etc. The program works by pulling down a list of jobs locally and using the constraint to list only the matching jobs. The default is only to pull down 1000 jobs, so that might give some surprising results by default.

Problem: The parse_datetime() utility function is statically configured
to assume future dates when a term like "Friday" is given. That is,
instead of giving the date for the previous Friday, the function will
return then next Friday instead. This behavior should be configurable.

Add an assumeFuture parameter to the function which defaults to True.
If set to False, then parse_datetime() will assume dates in the past
instead.


class Duration(Comparison):
def convert(self, duration):

Check failure

Code scanning / CodeQL

Superclass attribute shadows subclass method

Method convert is shadowed by an [attribute](1) in super class 'Comparison'.


class Datetime(Comparison):
def convert(self, dt):

Check failure

Code scanning / CodeQL

Superclass attribute shadows subclass method

Method convert is shadowed by an [attribute](1) in super class 'Comparison'.
@codecov
Copy link

codecov bot commented Feb 3, 2023

Codecov Report

Merging #4915 (e0583b7) into master (604eb88) will decrease coverage by 24.40%.
The diff coverage is 54.47%.

@@             Coverage Diff             @@
##           master    #4915       +/-   ##
===========================================
- Coverage   83.40%   59.01%   -24.40%     
===========================================
  Files         423      416        -7     
  Lines       73806    71705     -2101     
===========================================
- Hits        61559    42315    -19244     
- Misses      12247    29390    +17143     
Impacted Files Coverage Δ
src/bindings/python/flux/util.py 94.28% <50.00%> (-0.37%) ⬇️
src/bindings/python/flux/job/pgrep.py 53.04% <53.04%> (ø)
src/cmd/flux-pgrep.py 92.72% <83.33%> (-2.52%) ⬇️
src/common/libutil/setenvf.c 0.00% <0.00%> (-100.00%) ⬇️
src/modules/job-manager/plugins/submit-hold.c 0.00% <0.00%> (-100.00%) ⬇️
src/modules/job-list/job_util.c 0.00% <0.00%> (-91.61%) ⬇️
src/modules/cron/datetime.c 0.00% <0.00%> (-91.55%) ⬇️
src/cmd/builtin/dmesg.c 2.87% <0.00%> (-91.37%) ⬇️
src/modules/job-manager/urgency.c 0.00% <0.00%> (-90.70%) ⬇️
src/bindings/lua/lutil.c 0.00% <0.00%> (-90.63%) ⬇️
... and 305 more

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant