bug: pip_url not consistent between .lock file and meltano.yml

[ReubenFrankel]

Meltano Version

2.20.0

Python Version

3.8

Bug scope

CLI (options, error messages, logging, etc.)

Operating System

Linux - Ubuntu 22.04

Description

Given a plugin in a project and its upstream definition updated with a different pip_url, when updating the plugin in the project with

meltano lock --update <plugin>

the pip_url for the plugin entry in the meltano.yml is not updated (i.e a discrepancy between the plugin .lock file and meltano.yml entry).

This raises the following questions for me (not sure if answers to these are documented anywhere):

1. Is Meltano using the pip_url from the plugin .lock file or the meltano.yml entry during install?
2. Why is pip_url included in the plugin meltano.yml entry by default?

---

Might relate to #8176

Code

No response

[edgarrmondragon]

1. Is Meltano using the pip_url from the plugin .lock file or the meltano.yml entry during install?

It should be using pip_url from meltano.yml. Otherwise there's a bug.

2. Why is pip_url included in the plugin meltano.yml entry by default?

The idea of having it in meltano.yml is that users can easily identify it and override it with a fork, for example. Though I don't know how common of a need that is. Perhaps we could omit it by default but still let users override it, but that'd require significant docs updates and regression testing.

[ReubenFrankel]

Yeah, so can confirm Meltano is using the pip_url from the meltano.yml during install like you said, so no issue there. I was mainly asking because when we pinned tap-shopify to v0.2.0 on Meltano Hub, a user was asking about how to update and I wasn't sure which pip_url would be used, since both point to the same commit currently.

I think #8176 could address this, but I'm not sure in what situation you would use meltano lock --update in then?

[edgarrmondragon]

I think #8176 could address this, but I'm not sure in what situation you would use meltano lock --update in then?

At the moment it only serves to update plugin settings and capabilities, but ideally the pip_url should get updated too (like #6416)

[ReubenFrankel]

Just caught up with Office Hours. I'm not really sure what the consensus was on pinning plugins to specific versions on Meltano Hub in the end - we do seem to be the only ones actively doing this... Although, the dbt utilities are pinned (e.g. dbt-postgres) and I assume whenever these are updated on the Hub to reference new versions, someone else will run into this same issue?

Our motivation for this is purely that it's more stable for users to be pinned to a specific version, and they can update whenever - manually or through whatever means Meltano provides. I agree that it's good to have the pip_url in the meltano.yml for clarity, but the fact that it can be different from the .lock file without a user having chosen to override it themselves is where I think this issue stems from.

[ReubenFrankel]

My main problem with an unpinned pip_url for a plugin definition on the Hub is that there is no guarantee that the plugin won't break (i.e. as a user, I just want a version that works). Currently, there is no coupling between the version of the plugin that actually gets installed and the defined settings and/or capabilities - it could be invalided as soon as a breaking change is made and pushed to master/main, so the Hub definition must be kept up-to-date (or the user has to start overriding/redeclaring settings/capabilities in the meltano.yml). I realise that there is also a deeper issue here around the plugin not breaking, concerning dependencies of the plugin itself (discussed more in #6416).

Also, if a plugin's pip_url is overridden (pinned) in the meltano.yml, meltano lock --update becomes a useless (and potentially dangerous) command , since the Hub definition might be or could become incompatible with that pinned version.

[pnadolny13]

@ReubenFrankel I was asking the question of whether its the hub/meltano's responsibility to pin the plugin vs the user's responsibility. You make a good point that its mixed right now since your variants and dbt imply that its the hub's responsibility but the majority of the other hub plugins offload that responsibility to the user.

I could see it making sense to update the pip_url in the meltano.yml anytime the lockfile is updated. Even if the hub pip_url isnt pinned, if someone asks to get an updated lock file then they'd probably also want the updated plugin code to go with it since the hub definition usually tracks the latest version. If we dont update the pip_url during a lock --update and the user has it pinned then its likely that new settings will be included in their lock file that arent supported by their pinned version. The only downside to this is that in most cases (excluding matatika and dbt) plugins get unpinned after a lock --update, so the user will need to remember to update and pin them again, which actually might be a good process to encourage.

[ReubenFrankel]

I could see it making sense to update the pip_url in the meltano.yml anytime the lockfile is updated. Even if the hub pip_url isnt pinned, if someone asks to get an updated lock file then they'd probably also want the updated plugin code to go with it since the hub definition usually tracks the latest version.

This is actually how #8176 is implemented at the moment with meltano add --update, which is why I was asking about the overlap with meltano lock --update.

If we dont update the pip_url during a lock --update and the user has it pinned then its likely that new settings will be included in their lock file that arent supported by their pinned version. The only downside to this is that in most cases (excluding matatika and dbt) plugins get unpinned after a lock --update, so the user will need to remember to update and pin them again, which actually might be a good process to encourage.

In my mind, this feels a bit off because it would mean meltano lock operating outside the scope of just .lock files (as the command name implies). I'm curious on how you think it might be a good process for a user to re-pin after an update though, since it might be relevant to #8176.