Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate/hcl: support module calls through source keyword #130

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

tormath1
Copy link
Contributor

In this PR, we add the support for module calls. It supports local, remote and Terraform registry ones.

The function moduleInstall is called recursively in order to walk through the ModuleCalls of each module to feed a ManagedResource slice. It stores a trace of each installed element to avoid infinite loops and modules are cached into $XDG_CACHE directory.

Then it loops over the ManagedResource slice with the current behavior.

A few notes:

  1. I'd like to use in-memory FS rather than actual FS but go-getter needs to be updated consequently (Abstract the os access via interface hashicorp/go-getter#83: I started to work on it btw)
  2. I still need to find a way to properly testing this :)

Output example using this stack: https://github.com/cycloid-community-catalog/stack-gke

$ ./inframap generate --hcl generate/testdata/stack-gke
strict digraph G {
        "google_container_node_pool.pools"->"google_container_cluster.primary";
        "null_resource.delete_default_kube_dns_configmap"->"google_container_cluster.primary";
        "null_resource.delete_default_kube_dns_configmap"->"google_container_node_pool.pools";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns-upstream-namservers"->"google_container_node_pool.pools";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains"->"google_container_node_pool.pools";
        "kubernetes_config_map.ip-masq-agent"->"google_container_cluster.primary";
        "kubernetes_config_map.ip-masq-agent"->"google_container_node_pool.pools";
        "null_resource.wait_for_cluster"->"google_container_cluster.primary";
        "null_resource.wait_for_cluster"->"google_container_node_pool.pools";
        "google_service_account.cluster_service_account"->"random_string.cluster_service_account_suffix";
        "kubernetes_config_map.kube-dns"->"null_resource.delete_default_kube_dns_configmap";
        "kubernetes_config_map.kube-dns"->"google_container_cluster.primary";
        "kubernetes_config_map.kube-dns"->"google_container_node_pool.pools";
        "google_container_cluster.primary" [ shape=ellipse ];
        "google_container_node_pool.pools" [ shape=ellipse ];
        "google_service_account.cluster_service_account" [ shape=ellipse ];
        "kubernetes_config_map.ip-masq-agent" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns-upstream-nameservers-and-stub-domains" [ shape=ellipse ];
        "kubernetes_config_map.kube-dns-upstream-namservers" [ shape=ellipse ];
        "null_resource.delete_default_kube_dns_configmap" [ shape=ellipse ];
        "null_resource.wait_for_cluster" [ shape=ellipse ];
        "random_string.cluster_service_account_suffix" [ shape=ellipse ];

}

Closes: #56, #54

Copy link
Member

@xescugc xescugc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you also still working on adding the afero.FS implemtation into the TF lib?

Comment on lines +63 to +66
managedResources := make(map[string]*configs.Resource)
for rk, rv := range mod.ManagedResources {
managedResources[rk] = rv
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing we are doing right now on the State is check if we have to prefix a resource with the module because the same resource could exist on another module aws_instance.front could be in multiple modules.

We only do that when we have more than 1 module we should try to do the same as we'll find the same issue.

Comment on lines +353 to +368
for _, vers := range meta.Versions {
v, err := version.NewVersion(vers.Version)
if err != nil {
return fmt.Errorf("unable to create version from string: %w", err)
}

if latest == nil || v.GreaterThan(latest) {
latest = v
}

if call.Version.Required.Check(v) {
if match == nil || v.GreaterThan(match) {
match = v
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment a bit this logic.

generate/hcl.go Show resolved Hide resolved
generate/hcl.go Outdated
Comment on lines 399 to 408
dst := path.Join(cachePath, name)
// TODO: we should add a logic to invalidate
// the cache
if _, err := os.Stat(dst); os.IsNotExist(err) {
client := &getter.Client{
Src: src,
Dst: dst,
Pwd: pwd,
Mode: getter.ClientModeDir,
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we have versions, we can assume versions are immutable, so you could use a dst := path.Join(cachePath, fmt.Sprintf("%s-%s", name, version))

Comment on lines +421 to +426
// fill the final map of managed resources
// using the config freshly loaded
for rk, rv := range m.ManagedResources {
(*mRes)[rk] = rv
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially prefix it with them module if more than one or we'll have collision.

cmd/root.go Outdated
Comment on lines 26 to 31
PersistentPreRun: func(cmd *cobra.Command, args []string) {
if debug {
logsOut = os.Stdout
}
log.Init(logsOut, debug)
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer not to include TC for this, copy the part you need (activation of TF logs) and open an issue to add Debugging to IM (if we do not have one already)

@xescugc
Copy link
Member

xescugc commented Sep 2, 2021

Ping @Thomas-lhuillier hehe.

@Thomas-lhuillier
Copy link

What do I have to do? 😅

@tormath1
Copy link
Contributor Author

tormath1 commented Sep 2, 2021

@Thomas-lhuillier I guess @xescugc wanted to ping me instead - but feel free to work on this PR if you want to. 🙈

Anyways, thanks for the ping ! Besides the comments to address, I guess there is not so much effort to provide to get this PR land into a new release.

Are you also still working on adding the afero.FS implemtation into the TF lib?

Nope. It was quite a mess to bring afero.FS support to the lib IIRC.

@Thomas-lhuillier
Copy link

@Thomas-lhuillier I guess @xescugc wanted to ping me instead - but feel free to work on this PR if you want to. 🙈

Makes sense 👍 Well, this all seems out of reach to me, maybe after a couple months/years training :trollface:

@xescugc xescugc force-pushed the master branch 2 times, most recently from c0bd35f to d9632dd Compare September 3, 2021 07:54
@xescugc
Copy link
Member

xescugc commented Sep 6, 2021

Yes I think it was an autocomplete error from GH and I did not check haha the ping is for @tormath1 (ok if i do @th the first one is Thomas haha)

@tormath1 tormath1 mentioned this pull request Oct 11, 2022
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

hcl: support module sources
3 participants