[BUG] - Config generated by guided init should contain default settings #2122
Labels
Comments
rsignell
added
needs: triage 🚦
|
Yes, indeed. This would be super helpful. |
|
We discussed in the community meeting and we agree that displaying the entire config in nebari-config.yaml is better and we would accept PRs doing that (reverting to prior Nebari behavior). |
|
It looks like we're manually putting each thing we want to be in the config in If we want to prevent some attributes from showing up in the config, I think we could do something where we add a private _hide_from_config var on each model where we don't want some fields to show up in the config by default (e.g. terraform overrides, maybe). I put together a small demo showing how this might work. I assume we'd modify Base class in from pydantic import BaseModel
class Base(BaseModel):
_hide_from_config: set = {}
@classmethod
def _get_config_exclude_attrs(cls):
exclude_dict = {}
for attr_name, field in cls.__fields__.items():
if attr_name in cls._hide_from_config:
exclude_dict[attr_name] = True
else:
if hasattr(field.type_, '_get_config_exclude_attrs'):
exclude_dict[attr_name] = field.type_._get_config_exclude_attrs()
return exclude_dict
def write_config(self):
return self.json(exclude=self._get_config_exclude_attrs())
class SubSubModel(Base):
_hide_from_config = {'hidden_sub_sub_model_value'}
sub_sub_model_value: int = 2
hidden_sub_sub_model_value: int = -9999
class SubModel(Base):
_hide_from_config = {'hidden_sub_model_value'}
sub_model_value: SubSubModel = SubSubModel()
hidden_sub_model_value: int = -9999
class MyModel(Base):
_hide_from_config: set = {'should_not_be_written'}
should_be_written: SubModel
should_not_be_written: str = None
other_should_be_written: str = '1'
mm = MyModel(should_be_written=SubModel())
config = mm.write_config()
print('='*80)
print(mm.dict(exclude={
'should_not_be_written': True,
'should_be_written': {
'hidden_sub_model_value': True,
'sub_model_value': {
'hidden_sub_sub_model_value': True
},
}
}))
print('='*80)
print(mm.write_config())
print('='*80)
# Output looks like
# ================================================================================
# {'should_be_written': {'sub_model_value': {'sub_sub_model_value': 2}}, 'other_should_be_written': '1'}
# ================================================================================
# {"should_be_written": {"sub_model_value": {"sub_sub_model_value": 2}}, "other_should_be_written": "1"}
# ================================================================================Update: Looked into this a bit more. Potential issues are:
|
|
With a very small code change we could get the config file to look something like what I've pasted below. Issues:
I think we can solve this by using more specific Pydantic Models e.g. SelfSignedCertificate and SecretCertificate instead of capturing both cases in one Pydantic model. Update: I think I fixed the certs using more specific Pydantic models. I'd want to fix the providers also though. |
10 tasks
|
Well I think even with its issues this would be a big improvement! |
|
A few more changes, the config now looks like what's below (got rid of extra aws, gcp, etc. sections) I want to take a look at the other config files (aws, gcp, etc.) and see if we should make any other changes. |
10 tasks
|
@Adam-D-Lewis , awesome, that alone will be super useful! |
marcelovilla
added
area: user experience 👩🏻💻
area: nebari-cli
and removed
needs: triage 🚦
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
We generated a config using guided-init , and deployed Nebari on AWS with no apparent issues.
We then tried to use the deployment for training and students were not able to get the 30 workers they requested. We double checked our service quota in the region we were running the training and it was set to 500.
So we were mystified (and frustrated) why things weren't working.
Then after the training we realized that the generated config didn't contain the node groups section with the max limits for the different worker pools. In other words, the amazon section just looked like:
So I imagine we were just getting the default max, which must have been something small.
When I added the section to:
and rerendered and redeployed, multiple students could spin up clusters of 30 again.
Expected behavior
The expected behaviour would be to have the guided-init create a config that contains the default settings so they could (1) show the user what the defaults were, and (2) provide a template that facilitates easy modification by the user.
OS and architecture in which you are running Nebari
aws
How to Reproduce the problem?
use the guided init on aws
Command output
No response
Versions and dependencies used.
nebari 2023.10.1
Compute environment
AWS
Integrations
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: