New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add nx.config
dict for configuring dispatching and backends
#7225
Merged
Merged
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
52f96fe
Add `nx.backend_config` dict for configuring dispatching and backends
eriknw 023c47f
Merge branch 'main' into backend_config
eriknw 0510588
Rename `nx.backend_config` to `nx.config`
eriknw ee437ad
Merge branch 'main' into backend_config
eriknw 93ba39e
"fallback_to_nx" is for testing, not for user config
eriknw 276a371
Merge branch 'main' into backend_config
eriknw dc8366a
Move config of backends to e.g. `nx.config["backends"]["cugraph"]`
eriknw 438c83a
How do you like this mypy?!
eriknw 9980f6a
Merge branch 'main' into backend_config
eriknw ba886c7
Rename `automatic_backends` to `backend_priority` (and env variables)
eriknw 40a27a7
Merge branch 'main' into backend_config
eriknw ae480b2
Merge branch 'main' into backend_config
eriknw c81595e
Create a class to handle configuration
eriknw c4b7892
Oops thanks mypy
eriknw 65c8583
Fix to work with more strict config
eriknw dd0096e
Support (and test) default values
eriknw eb7ea72
Merge branch 'main' into backend_config
eriknw 2323189
Remove `__class_getitem__` and add docstring
eriknw f45b1d2
Allow `strict=False` when defining subclasses.
eriknw 71d37e7
Move `__init_subclass__`
eriknw File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,228 @@ | ||
import collections | ||
import typing | ||
from dataclasses import dataclass | ||
|
||
__all__ = ["Config", "config"] | ||
|
||
|
||
@dataclass(init=False, eq=False, slots=True, kw_only=True, match_args=False) | ||
class Config: | ||
"""The base class for NetworkX configuration. | ||
|
||
There are two ways to use this to create configurations. The first is to | ||
simply pass the initial configuration as keyword arguments to ``Config``: | ||
|
||
>>> cfg = Config(eggs=1, spam=5) | ||
>>> cfg | ||
Config(eggs=1, spam=5) | ||
|
||
The second--and preferred--way is to subclass ``Config`` with docs and annotations. | ||
|
||
>>> class MyConfig(Config): | ||
... '''Breakfast!''' | ||
... | ||
... eggs: int | ||
... spam: int | ||
... | ||
... def _check_config(self, key, value): | ||
... assert isinstance(value, int) and value >= 0 | ||
>>> cfg = MyConfig(eggs=1, spam=5) | ||
|
||
Once defined, config items may be modified, but can't be added or deleted by default. | ||
``Config`` is a ``Mapping``, and can get and set configs via attributes or brackets: | ||
|
||
>>> cfg.eggs = 2 | ||
>>> cfg.eggs | ||
2 | ||
>>> cfg["spam"] = 42 | ||
>>> cfg["spam"] | ||
42 | ||
|
||
Subclasses may also define ``_check_config`` (as done in the example above) | ||
to ensure the value being assigned is valid: | ||
|
||
>>> cfg.spam = -1 | ||
Traceback (most recent call last): | ||
... | ||
AssertionError | ||
|
||
If a more flexible configuration object is needed that allows adding and deleting | ||
configurations, then pass ``strict=False`` when defining the subclass: | ||
|
||
>>> class FlexibleConfig(Config, strict=False): | ||
... default_greeting: str = "Hello" | ||
>>> flexcfg = FlexibleConfig() | ||
>>> flexcfg.name = "Mr. Anderson" | ||
>>> flexcfg | ||
FlexibleConfig(default_greeting='Hello', name='Mr. Anderson') | ||
""" | ||
|
||
def __init_subclass__(cls, strict=True): | ||
cls._strict = strict | ||
|
||
def __new__(cls, **kwargs): | ||
orig_class = cls | ||
if cls is Config: | ||
# Enable the "simple" case of accepting config definition as keywords | ||
cls = type( | ||
cls.__name__, | ||
(cls,), | ||
{"__annotations__": {key: typing.Any for key in kwargs}}, | ||
) | ||
cls = dataclass( | ||
eq=False, | ||
repr=cls._strict, | ||
slots=cls._strict, | ||
kw_only=True, | ||
match_args=False, | ||
)(cls) | ||
if not cls._strict: | ||
cls.__repr__ = _flexible_repr | ||
cls._orig_class = orig_class # Save original class so we can pickle | ||
instance = object.__new__(cls) | ||
instance.__init__(**kwargs) | ||
return instance | ||
|
||
def _check_config(self, key, value): | ||
"""Check whether config value is valid. This is useful for subclasses.""" | ||
|
||
# Control behavior of attributes | ||
def __dir__(self): | ||
return self.__dataclass_fields__.keys() | ||
|
||
def __setattr__(self, key, value): | ||
if self._strict and key not in self.__dataclass_fields__: | ||
raise AttributeError(f"Invalid config name: {key!r}") | ||
self._check_config(key, value) | ||
object.__setattr__(self, key, value) | ||
|
||
def __delattr__(self, key): | ||
if self._strict: | ||
raise TypeError( | ||
f"Configuration items can't be deleted (can't delete {key!r})." | ||
) | ||
object.__delattr__(self, key) | ||
|
||
# Be a `collection.abc.Collection` | ||
def __contains__(self, key): | ||
return ( | ||
key in self.__dataclass_fields__ if self._strict else key in self.__dict__ | ||
) | ||
|
||
def __iter__(self): | ||
return iter(self.__dataclass_fields__ if self._strict else self.__dict__) | ||
|
||
def __len__(self): | ||
return len(self.__dataclass_fields__ if self._strict else self.__dict__) | ||
|
||
def __reversed__(self): | ||
return reversed(self.__dataclass_fields__ if self._strict else self.__dict__) | ||
|
||
# Add dunder methods for `collections.abc.Mapping` | ||
def __getitem__(self, key): | ||
try: | ||
return getattr(self, key) | ||
except AttributeError as err: | ||
raise KeyError(*err.args) from None | ||
|
||
def __setitem__(self, key, value): | ||
try: | ||
setattr(self, key, value) | ||
except AttributeError as err: | ||
raise KeyError(*err.args) from None | ||
|
||
__delitem__ = __delattr__ | ||
_ipython_key_completions_ = __dir__ # config["<TAB> | ||
|
||
# Go ahead and make it a `collections.abc.Mapping` | ||
def get(self, key, default=None): | ||
return getattr(self, key, default) | ||
|
||
def items(self): | ||
return collections.abc.ItemsView(self) | ||
|
||
def keys(self): | ||
return collections.abc.KeysView(self) | ||
|
||
def values(self): | ||
return collections.abc.ValuesView(self) | ||
|
||
# dataclass can define __eq__ for us, but do it here so it works after pickling | ||
def __eq__(self, other): | ||
if not isinstance(other, Config): | ||
return NotImplemented | ||
return self._orig_class == other._orig_class and self.items() == other.items() | ||
|
||
# Make pickle work | ||
def __reduce__(self): | ||
return self._deserialize, (self._orig_class, dict(self)) | ||
|
||
@staticmethod | ||
def _deserialize(cls, kwargs): | ||
return cls(**kwargs) | ||
|
||
|
||
def _flexible_repr(self): | ||
return ( | ||
f"{self.__class__.__qualname__}(" | ||
+ ", ".join(f"{key}={val!r}" for key, val in self.__dict__.items()) | ||
+ ")" | ||
) | ||
|
||
|
||
# Register, b/c `Mapping.__subclasshook__` returns `NotImplemented` | ||
collections.abc.Mapping.register(Config) | ||
|
||
|
||
class NetworkXConfig(Config): | ||
"""Configuration for NetworkX that controls behaviors such as how to use backends. | ||
|
||
Attribute and bracket notation are supported for getting and setting configurations: | ||
|
||
>>> nx.config.backend_priority == nx.config["backend_priority"] | ||
True | ||
|
||
Config Parameters | ||
----------------- | ||
backend_priority : list of backend names | ||
Enable automatic conversion of graphs to backend graphs for algorithms | ||
implemented by the backend. Priority is given to backends listed earlier. | ||
|
||
backends : Config mapping of backend names to backend Config | ||
The keys of the Config mapping are names of all installed NetworkX backends, | ||
and the values are their configurations as Config mappings. | ||
""" | ||
|
||
backend_priority: list[str] | ||
backends: Config | ||
|
||
def _check_config(self, key, value): | ||
from .backends import backends | ||
|
||
if key == "backend_priority": | ||
if not (isinstance(value, list) and all(isinstance(x, str) for x in value)): | ||
raise TypeError( | ||
f"{key!r} config must be a list of backend names; got {value!r}" | ||
) | ||
if missing := {x for x in value if x not in backends}: | ||
missing = ", ".join(map(repr, sorted(missing))) | ||
raise ValueError(f"Unknown backend when setting {key!r}: {missing}") | ||
elif key == "backends": | ||
if not ( | ||
isinstance(value, Config) | ||
and all(isinstance(key, str) for key in value) | ||
and all(isinstance(val, Config) for val in value.values()) | ||
): | ||
raise TypeError( | ||
f"{key!r} config must be a Config of backend configs; got {value!r}" | ||
) | ||
if missing := {x for x in value if x not in backends}: | ||
missing = ", ".join(map(repr, sorted(missing))) | ||
raise ValueError(f"Unknown backend when setting {key!r}: {missing}") | ||
|
||
|
||
# Backend configuration will be updated in backends.py | ||
config = NetworkXConfig( | ||
backend_priority=[], | ||
backends=Config(), | ||
) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here, it might be more optimal to have a separate function(like
get_info
) that returns only the backend's configurations, likeget_config(config_dict=None)
(ifconfig_dict
is notNone
then the configuration dictionary is updated and returned otherwise the default configurations are returned) because config dictionary would probably be updated quite frequently and we probably shouldn't load all thebackend_info
every time. Please let me know if I'm missing something here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I completely understand your comment.
get_info
is called exactly once at import time for each backend (before this PR and in this PR). Among other things, this helps the dispatch machinery know which backends implement which functions.By optionally including
"default_config"
in the dict returned byget_info
, we are able to initialize the default configuration for each backend at import time. This happens only once. This may be nice for users so they can do things like:nx.backend_config
is how we expect users to update configuration.We could make a separate entry point for the default config, but I don't know if it would be worth doing since it seems to me that the "backend_info" entry point works well enough.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, I thought this kind of thing was possible :
so I thought we would be extracting all the configurations and adding them in the
joblib.Parallel()
in all the functions, so having a separateget_config
function made sense to me.But, I think, based on the clarification above, that all the configuration code lines will be executed while importing networkx so the final configuration for both the function calls would be
{"n_jobs" : -1, "backend" : "threading"}
, right?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Coming back to this question:
get_info
function.I think your
get_config
function would live in nx-parallel and look atnx.config.backends.parallel.<name>
for each option. Or, we could use the whole config object like:joblib.Parallel(**nx.config.backends.parallel)
. But the user would be able to change them between each function call.