Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metadata provider for Oracle, Alicloud, Ikoula and Azure #3543

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

stevefan1999-personal
Copy link

@stevefan1999-personal stevefan1999-personal commented Aug 20, 2020

- How to verify it
Make a new instance from Oracle Cloud, there's a free tier available so it is quite handy to test on, I've tested it personally across all my available servers and it works very well. Recall that my main goal is to hope it would work on k3os.

- Description for the changelog
Just added support for more platforms. This will cover most well known major cloud platforms because they had free trials.

Although I've also tested it personally and its great however, on Azure I have issues obtaining user data that Azure team disabled it for some reason (this is stupid), so the user data field is going to be empty. This issue had been in triage of backlog for a year now. There's currently no way to get it unless we reverse engineer WALinuxAgent and replay the entire process but their code is written in Python...

Also, Ikoula is really doing its best to disgust developers, as they do not serve the metadata in a static location, but rather served it on the DHCP server (technically also the virtual router server of Apache CloudStack). I have to pull in a huge library that can send DHCP request, intercept which server responded the request at the cost of increasing the binary size by 30%. Should this not be necessary I can remove this, because there might be people complaining about the size of this particular executable now because it is not following the "less is more" philosophy.

I also wanted to renovate this project a little as I observed this project still uses an outdated "vendor tool" that doesn't even work at all (let alone the pain of waiting 15 minutes to download all that vendored code that should not happen in the first place), so I tried to convert the project to use go modules but failed at my best, the remaining works is left opened as #3545. Because school season is kicking in I might had limited time to do this. Anyone had better expertise can try and finish it.

- Description of a shorter changelog

  • convert the main executable to use go mod, this should improve build time because we can refresh go mod easily
    • this also means we can leverage go mod vendor to provide and cache vendor packages in CI once and for all
    • this is also part of the plan to refresh this project with modern Go practices
  • see tile
  • added a simpler way to dump the metadata in a neato way. this is not the best practice however.

- A picture video of a cute animal (not mandatory but encouraged)
https://www.youtube.com/watch?v=1bhn-z8xA5M

@stevefan1999-personal
Copy link
Author

uh, where's the gomod file?

@stevefan1999-personal
Copy link
Author

I've tested it manually on my own k3os image in Oracle Cloud. It works flawlessly.

@stevefan1999-personal stevefan1999-personal changed the title Add oracle metadata provider Add metadata provider for Oracle, Alicloud and Azure Aug 21, 2020
@stevefan1999-personal
Copy link
Author

@rn CC

@rn
Copy link
Member

rn commented Aug 22, 2020

Could you please squash/re-arrange the commits so that they are logical rather than historical (i see quite a few later commits fixing up earlier commits, for example)

@rn
Copy link
Member

rn commented Aug 22, 2020

The CI is also failung with"

provider_oci.go:8:2: cannot find package "github.com/go-resty/resty" in any of:
	/usr/lib/go/src/github.com/go-resty/resty (from $GOROOT)

The other CI failure is unrelated

@stevefan1999-personal
Copy link
Author

stevefan1999-personal commented Aug 22, 2020

@rn may I know how do you add vendor packages correctly?

Also regarding the git commit issues...that's the way git is...the underlying data structure git uses to keep commits, i.e. merkle tree is specifically used to track layered hashes and history...so viewing it chronologically like a replay machine makes more sense. I will squash as much as I can.

edit: oh thank goodness the rebase didn't f*ck up like it always used to be

@rn
Copy link
Member

rn commented Aug 22, 2020

it's been quite a while for me and the go eco system has moved quite a bit, but docs/vendoring.md might help

@stevefan1999-personal
Copy link
Author

@rn thanks, I searched vendor.conf and it lead me to rancher/trash and the format your tool uses are similar so I'm quite confused, now I should be able to add the deps and fix the build...

@stevefan1999-personal stevefan1999-personal force-pushed the patch-oracle-meta branch 4 times, most recently from e13790e to e2b2e08 Compare August 22, 2020 17:56
@stevefan1999-personal
Copy link
Author

error: unable to create file fixtures/compatibility/notary0.1/private/tuf_keys/docker.com/notary0.1/samplerepo/7fc757801b9bab4ec9e35bfe7a6b61668ff6f4c81b5632af19e6c728ab799599_targets.key: Filename too long
error: unable to create file fixtures/compatibility/notary0.1/private/tuf_keys/docker.com/notary0.1/samplerepo/a55ccf652b0be4b6c4d356cbb02d9ea432bb84a2571665be3df7c7396af8e8b8_snapshot.key: Filename too long
error: unable to create file fixtures/compatibility/notary0.1/trusted_certificates/docker.com/notary0.1/samplerepo/4ff57dc987163053a12c066f2dd36b1ae6037a92f5416d381fe311a3db1868d8.crt: Filename too long

well...

@stevefan1999-personal
Copy link
Author

okay so its apparently the limitation of that vndr tool...it keeps deleting the package if the module is deemed "unused", so it can't track transitive requirements at all

@stevefan1999-personal stevefan1999-personal force-pushed the patch-oracle-meta branch 9 times, most recently from adc07a9 to e5b3953 Compare August 24, 2020 11:14
@stevefan1999-personal
Copy link
Author

stevefan1999-personal commented Aug 24, 2020

I think I figured out the reason the packages can't build: my git metadata is corrupted. It recognized some vendor folder as dangling submodules, and I have to replace the dangling data back with a correct pointer to a tracked folder forcefully, it should build just fine now after I re-add some transitive dependencies

@rn I've finally managed to fix it

@stevefan1999-personal stevefan1999-personal force-pushed the patch-oracle-meta branch 4 times, most recently from 8dc9c5e to 9c69cd2 Compare August 24, 2020 11:46
…nfrastructure

Signed-off-by: Steve Fan <29133953+stevefan1999-personal@users.noreply.github.com>
Signed-off-by: Steve Fan <29133953+stevefan1999-personal@users.noreply.github.com>
@stevefan1999-personal stevefan1999-personal force-pushed the patch-oracle-meta branch 3 times, most recently from e765469 to f2f7b01 Compare August 26, 2020 13:52
…cloud infrastructure

Signed-off-by: Steve Fan <29133953+stevefan1999-personal@users.noreply.github.com>
@stevefan1999-personal stevefan1999-personal changed the title Add metadata provider for Oracle, Alicloud and Azure Add metadata provider for Oracle, Alicloud, Ikoula and Azure Aug 26, 2020
@stevefan1999-personal
Copy link
Author

@rn any chance for lgtm?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants