Is there any way to preserve ordering when outputing json?

[aeb-dev]

I have following template:

{{{ `{"hello":"world", "a": "b"}` | data.JSON | data.ToJSONPretty "  " }}}

then I call gomplate -f input.tmpl -o output.tmpl
the output is:

{
  "a": "b",
  "hello": "world"
}

what I want is:

{
  "hello": "world",
  "a": "b"
}

Is there any way to preserve ordering when outputing json?

[jeremych1000]

Likewise YAML is ordered alphabetically when using ToYAML.

Python's PyYAML has sort_keys=False in its newest version, is something similar possible here?

[hairyhenderson]

This behaviour is currently expected due to the way that maps work in Go - the order is internally randomized when it's parsed with data.JSON (or data.YAML). Then, when the object is marshalled back to JSON or YAML, it's ordered alphabetically for consistency.

To dive into the details why this happens - JSON and YAML are internally unmarshalled into a map[string]interface{} value, which makes it possible to convert back and forth to various key/value representations, and also to access elements directly with the . operator. I could solve the ordering problem by unmarshalling directly into a yaml.Node struct, and this would have the advantage of also preserving comments. However, the ability to access elements directly with the . operator would then be lost. IMO this would be a bad tradeoff, given the main purpose of gomplate's JSON and YAML parsing (which is to read values from datasources).

I suppose this all begs the question: why would you need to parse JSON/YAML only to re-render it directly?

And, why does preserving order matter?

[jeremych1000]

I suppose this all begs the question: why would you need to parse JSON/YAML only to re-render it directly?

I agree it's a niche use case. Here's mine.

• Template A reads input.json and templates out a YAML file called foo.yaml - let's call the template A
• This YAML file represents a single user's configuration
• I change template A (add whitespace, add section, whatever), or edit the YAML of the user directly (change value from 2-3 etc.)
• I have to "regenerate" foo.yaml with the new template
• Storing the original input.json doesn't work as it doesn't have the updated values
• I therefore have to make a "reverse" template which extracts out values from the live YAML
• This "override" file is then used in place of input.json in template A
• Whenever we update the template, we "regenerate" all users' files

This works well, and involves parsing JSON/YAML only to immediately re-render it.

And, why does preserving order matter?

Yes, technically it doesn't, but it makes things more readable. For example,

compute:
  name: default-nodegroup
  instanceType: a1
  minSize: 0
  maxSize: 16
  tags: 
    foo: bar

is more readable than the alphabetically ordered version

compute:
  instanceType: a1
  minSize: 0
  maxSize: 16
  name: default-nodegroup
  tags: 
    foo: bar

Currently I've worked around that by doing

{{- range $c := $overrides.tenant_compute }}
  - name: {{ $c.name }}
    {{ $c | coll.Omit "name" | data.ToYAML | indent "    " | trimSpace }}
{{- end }}

[hairyhenderson]

Ok, I think I understand - what's going on is that there are the input comes from potentially two different and divergent sources. This is obviously tricky to deal with!

I wonder - have you looked at using the merge: datasource? That would allow you to put overrides in a separate file which is not modified by any re-generation step.

[jeremych1000]

I wonder - have you looked at using the merge: datasource? That would allow you to put overrides in a separate file which is not modified by any re-generation step.

Ah - wasn't aware of merge! Am I correct in saying that that will allow me to not have to define things like

if override
$override value
else
normal value
end

If so, I presume the variable names have to match so I have to update my reverse template. What happens if a variable isn't defined and I try to merge?

[hairyhenderson]

Am I correct in saying that that will allow me to not have to define things like [...]

yes, that's right - the data is presented to the template as if it came from a single source

If so, I presume the variable names have to match so I have to update my reverse template. What happens if a variable isn't defined and I try to merge?

I'm not certain what you mean by "variable" here... Maybe it's best to explain merge: with an example... Let's say you have a.yaml:

compute:
  instanceType: a1
  minSize: 0
  maxSize: 16

and b.yaml:

compute:
  name: default-nodegroup
  minSize: 5
  tags: 
    foo: bar

and you merge them together like so:

$ gomplate -d data=merge:a.yaml|b.yaml -f mytemplate.tmpl
...

then, mytemplate.tmpl will "see" the data datasource as if it was:

compute:
  name: default-nodegroup
  instanceType: a1
  minSize: 5
  maxSize: 16
  tags: 
    foo: bar

Hope that makes sense! Given that the original question was about preserving order, if you have any followups maybe you could start a separate discussion topic?

[jmls]

another reason to keep order for us is if we rebuild a json file using gomplate then you could have false positives for git changes and have lines added and deleted due to the order change, whereas the file content as a whole has not changed.

[hairyhenderson]

@jmls that's fair, but the order is stable once it's changed, is it not? so in theory you could commit the re-ordered file to git and then you'd be set from that point on...