Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[confmap] confmap honors Unmarshal methods on config embedded structs. #9635

Merged
merged 5 commits into from Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
25 changes: 25 additions & 0 deletions .chloggen/support_embedded_structs_confmap.yaml
@@ -0,0 +1,25 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: bug_fix

# The name of the component, or a single word describing the area of concern, (e.g. otlpreceiver)
component: confmap

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: confmap honors `Unmarshal` methods on config embedded structs.

# One or more tracking issues or pull requests related to the change
issues: [6671]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
4 changes: 2 additions & 2 deletions cmd/mdatagen/metricdata.go
Expand Up @@ -125,7 +125,7 @@ func (d *gauge) Unmarshal(parser *confmap.Conf) error {
if err := d.MetricValueType.Unmarshal(parser); err != nil {
return err
}
return parser.Unmarshal(d)
return parser.Unmarshal(d, confmap.WithIgnoreUnused())
}

func (d gauge) Type() string {
Expand Down Expand Up @@ -155,7 +155,7 @@ func (d *sum) Unmarshal(parser *confmap.Conf) error {
if err := d.MetricValueType.Unmarshal(parser); err != nil {
return err
}
return parser.Unmarshal(d)
return parser.Unmarshal(d, confmap.WithIgnoreUnused())
}

// TODO: Currently, this func will not be called because of https://github.com/open-telemetry/opentelemetry-collector/issues/6671. Uncomment function and
Expand Down
31 changes: 31 additions & 0 deletions confmap/confmap.go
Expand Up @@ -157,6 +157,7 @@ func decodeConfig(m *Conf, result any, errorUnused bool) error {
mapstructure.StringToTimeDurationHookFunc(),
mapstructure.TextUnmarshallerHookFunc(),
unmarshalerHookFunc(result),
unmarshalerEmbeddedStructsHookFunc(result),
zeroSliceHookFunc(),
),
}
Expand Down Expand Up @@ -261,6 +262,36 @@ func mapKeyStringToMapKeyTextUnmarshalerHookFunc() mapstructure.DecodeHookFuncTy
}
}

func unmarshalerEmbeddedStructsHookFunc(_ any) mapstructure.DecodeHookFuncValue {
atoulme marked this conversation as resolved.
Show resolved Hide resolved
atoulme marked this conversation as resolved.
Show resolved Hide resolved
mx-psi marked this conversation as resolved.
Show resolved Hide resolved
return func(from reflect.Value, to reflect.Value) (any, error) {
if to.Type().Kind() != reflect.Struct {
mx-psi marked this conversation as resolved.
Show resolved Hide resolved
return from.Interface(), nil
}
fromAsMap, ok := from.Interface().(map[string]any)
if !ok {
return from.Interface(), nil
}
mx-psi marked this conversation as resolved.
Show resolved Hide resolved
for i := 0; i < to.Type().NumField(); i++ {
if to.Type().Field(i).IsExported() && to.Type().Field(i).Anonymous {
if unmarshaler, ok := to.Field(i).Addr().Interface().(Unmarshaler); ok {
if err := unmarshaler.Unmarshal(NewFromStringMap(fromAsMap)); err != nil {
return nil, err
}
conf := New()
if err := conf.Marshal(unmarshaler); err != nil {
return nil, err
}
resultMap := conf.ToStringMap()
for k, v := range resultMap {
fromAsMap[k] = v
}
Comment on lines +288 to +295
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this? This is not done in unmarshalerHookFunc

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We run after the main unmarshaler from unmarshalerHookFunc and must merge with the result, rather than set it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oki, can you add a comment about this? Both in the function itself as well as in mapstructure's configuration to make sure we don't change the order.

Also, I think it would be good to have a test that only triggers unmarshalerHookFunc and test that only triggers this hook (so, one struct with only embedded structs inside and one struct with no embedded structs inside in the tests).

Copy link
Contributor Author

@atoulme atoulme Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Comment: yes,sure can do
  2. oh no
  3. sure can do, I believe https://github.com/open-telemetry/opentelemetry-collector/blob/main/confmap/confmap_test.go#L337 already covers this well.

2. is a big "oh no" because of #7102. We have a top level if/else that explicitly forks the flow if the struct implements Unmarshaler or not.
If it does, it calls directly the Unmarshal function on the struct.
If it doesn't, it goes to mapstructure and then runs through the hooks.
This is this problem: #9635 (comment)

The workaround right now is that all structs with embedded structs that themselves implement Unmarshaler must themselves implement Unmarshaler and call mapstructure as part of their behavior.

We can straighten this up, but maybe in the next PR, if that's ok.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explaining and being patient with me :) Here is my proposal on what to do:

  1. On this PR, we add a new (failing) test for (2) and that uses t.Skip with a TODO linking to Remove component.UnmarshalConfig #7102. We also add a note on Remove component.UnmarshalConfig #7102 about said skip to not forget about it.
  2. On a future PR tackling Remove component.UnmarshalConfig #7102 we remove this t.Skip

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, added.

}
}
}
return fromAsMap, nil
}
}

// Provides a mechanism for individual structs to define their own unmarshal logic,
// by implementing the Unmarshaler interface.
func unmarshalerHookFunc(result any) mapstructure.DecodeHookFuncValue {
Expand Down
121 changes: 119 additions & 2 deletions confmap/confmap_test.go
Expand Up @@ -309,8 +309,78 @@ func newConfFromFile(t testing.TB, fileName string) map[string]any {
}

type testConfig struct {
atoulme marked this conversation as resolved.
Show resolved Hide resolved
Next *nextConfig `mapstructure:"next"`
Another string `mapstructure:"another"`
Next *nextConfig `mapstructure:"next"`
Another string `mapstructure:"another"`
EmbeddedConfig `mapstructure:",squash"`
EmbeddedConfig2 `mapstructure:",squash"`
}

type testConfigWithoutUnmarshaler struct {
Next *nextConfig `mapstructure:"next"`
Another string `mapstructure:"another"`
EmbeddedConfig `mapstructure:",squash"`
EmbeddedConfig2 `mapstructure:",squash"`
}

type testConfigWithEmbeddedError struct {
Next *nextConfig `mapstructure:"next"`
Another string `mapstructure:"another"`
EmbeddedConfigWithError `mapstructure:",squash"`
}

type testConfigWithMarshalError struct {
Next *nextConfig `mapstructure:"next"`
Another string `mapstructure:"another"`
EmbeddedConfigWithMarshalError `mapstructure:",squash"`
}

func (tc *testConfigWithEmbeddedError) Unmarshal(component *Conf) error {
if err := component.Unmarshal(tc, WithIgnoreUnused()); err != nil {
return err
}
return nil
}

type EmbeddedConfig struct {
Some string `mapstructure:"some"`
}

func (ec *EmbeddedConfig) Unmarshal(component *Conf) error {
if err := component.Unmarshal(ec, WithIgnoreUnused()); err != nil {
return err
}
ec.Some += " is also called"
return nil
}

type EmbeddedConfig2 struct {
Some2 string `mapstructure:"some_2"`
}

func (ec *EmbeddedConfig2) Unmarshal(component *Conf) error {
if err := component.Unmarshal(ec, WithIgnoreUnused()); err != nil {
return err
}
ec.Some2 += " also called2"
return nil
}

type EmbeddedConfigWithError struct {
}

func (ecwe *EmbeddedConfigWithError) Unmarshal(_ *Conf) error {
return errors.New("embedded error")
}

type EmbeddedConfigWithMarshalError struct {
}

func (ecwe EmbeddedConfigWithMarshalError) Marshal(_ *Conf) error {
return errors.New("marshaling error")
}

func (ecwe EmbeddedConfigWithMarshalError) Unmarshal(_ *Conf) error {
return nil
}

func (tc *testConfig) Unmarshal(component *Conf) error {
Expand Down Expand Up @@ -340,12 +410,59 @@ func TestUnmarshaler(t *testing.T) {
"string": "make sure this",
},
"another": "make sure this",
"some": "make sure this",
"some_2": "this better be",
})

tc := &testConfig{}
assert.NoError(t, cfgMap.Unmarshal(tc))
assert.Equal(t, "make sure this", tc.Another)
assert.Equal(t, "make sure this is called", tc.Next.String)
assert.Equal(t, "make sure this is also called", tc.EmbeddedConfig.Some)
assert.Equal(t, "this better be also called2", tc.EmbeddedConfig2.Some2)
}

func TestEmbeddedUnmarshaler(t *testing.T) {
cfgMap := NewFromStringMap(map[string]any{
"next": map[string]any{
"string": "make sure this",
},
"another": "make sure this",
"some": "make sure this",
"some_2": "this better be",
})

tc := &testConfigWithoutUnmarshaler{}
assert.NoError(t, cfgMap.Unmarshal(tc))
assert.Equal(t, "make sure this", tc.Another)
assert.Equal(t, "make sure this is called", tc.Next.String)
assert.Equal(t, "make sure this is also called", tc.EmbeddedConfig.Some)
assert.Equal(t, "this better be also called2", tc.EmbeddedConfig2.Some2)
}

func TestEmbeddedUnmarshalerError(t *testing.T) {
cfgMap := NewFromStringMap(map[string]any{
"next": map[string]any{
"string": "make sure this",
},
"another": "make sure this",
"some": "make sure this",
})

tc := &testConfigWithEmbeddedError{}
assert.EqualError(t, cfgMap.Unmarshal(tc), "error decoding '': embedded error")
}

func TestEmbeddedMarshalerError(t *testing.T) {
cfgMap := NewFromStringMap(map[string]any{
"next": map[string]any{
"string": "make sure this",
},
"another": "make sure this",
})

tc := &testConfigWithMarshalError{}
assert.EqualError(t, cfgMap.Unmarshal(tc), "error decoding '': error running encode hook: marshaling error")
}

func TestUnmarshalerKeepAlreadyInitialized(t *testing.T) {
Expand Down