Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

General: custom staging dir functionality migration #5207

Draft
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

jakubjezek001
Copy link
Member

@jakubjezek001 jakubjezek001 commented Jun 28, 2023

Changelog Description

The functionality of the staging directory has exceeded its original purpose of publishing. Considering its increased usage in the creation process, we need to migrate it to a dedicated module within the pipeline parent.

Additional info

  • adding functionality to Creator base class
  • consistency of naming across all features
  • making sure backward compatibility to older names and locations remain in settings and api
  • anatomy templates with Staging Dir category

Testing notes:

  1. all should be working as it was before

@ynbot
Copy link
Contributor

ynbot commented Jun 28, 2023

@ynbot ynbot added size/S Denotes a PR changes 100-499 lines, ignoring general files type: feature Larger, user affecting changes and completely new things host: Photoshop host: Substance Painter labels Jun 28, 2023
Copy link
Collaborator

@BigRoy BigRoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about the changes and whether it improves any readability, especially the circular imports (and it being worked around) feels problematic.

What stood out to me is that get_instance_staging_dir does not rely on the custom staging dir info at all nor the other way around. Which makes me wonder how do I actually get the correct staging directory for an instance. I'd have expected that get_instance_staging_dir would've considered also the custom transient dir setting, etc. but it seems that's still up to the implementation to check first whether an instance should have a custom staging dir, and if not.. then use get_instance_staging_dir which seems confusing from the name of that function.

openpype/pipeline/stagingdir.py Outdated Show resolved Hide resolved
openpype/pipeline/publish/lib.py Outdated Show resolved Hide resolved
openpype/pipeline/create/creator_plugins.py Outdated Show resolved Hide resolved
openpype/pipeline/create/creator_plugins.py Outdated Show resolved Hide resolved
@jakubjezek001 jakubjezek001 marked this pull request as ready for review June 30, 2023 09:46
@jakubjezek001 jakubjezek001 marked this pull request as draft June 30, 2023 10:58
@jakubjezek001 jakubjezek001 marked this pull request as ready for review July 3, 2023 10:45
Comment on lines 696 to 738
# deprecated: backward compatibility only
# TODO: remove in the future
def get_instance_staging_dir(instance):
return _get_instance_staging_dir(instance)


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# deprecated: backward compatibility only
# TODO: remove in the future
def get_instance_staging_dir(instance):
return _get_instance_staging_dir(instance)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you suggesting we don't need to maintain compatibility for this function?

@jakubjezek001 jakubjezek001 marked this pull request as draft July 4, 2023 12:04
@jakubjezek001 jakubjezek001 marked this pull request as ready for review July 6, 2023 09:58
**kwargs: Arbitrary keyword arguments. See below.

Keyword Arguments:
force_temp (bool): If True, staging dir will be created as tempdir.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come we are not adding them as optional arguments?

)
else:
ctx_data = get_template_data_with_names(
project_name, asset_name, task_name, system_settings
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be passing the host_name here as well?

# get customized tempdir path from `OPENPYPE_TMPDIR` env var
custom_temp_dir = _create_custom_tempdir(
anatomy.project_name, anatomy)

Copy link
Member

@Minkiu Minkiu Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be only for both? And then _create_local_staging_diris no longer needed :

 return os.path.normpath(
            tempfile.mkdtemp(
                prefix=prefix,
                suffix=suffix,
                dir=custom_temp_dir
            )
        )

If _create_custom_tempdir returns None, so we don't need to do the conditional check.

"""

output = {
instance.id: -1 if use_value_for_missing else None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we are assuming all of them are missing, right? But later on I can't see where do we check for missing entities? So it could be that we return an instance with -1 that has entities? i.e. Line 50

@jakubjezek001 jakubjezek001 marked this pull request as draft July 31, 2023 13:49
@@ -194,5 +208,5 @@

# Backwards compatible function names
"install",
"uninstall",
"uninstall"
Copy link
Member

@iLLiCiTiT iLLiCiTiT Aug 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We add the comma, so a future PRs which add after or move it does not have to add commit to the line.

create_ctx = self.create_context
asset_name = instance.get("asset")
subset = instance.get("subset")
if not any([asset_name, subset]):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually faster and easier to read.

Suggested change
if not any([asset_name, subset]):
if not asset_name or not subset:

if version == -2:
output[instance_id] = None
elif version == -1:
output[instance_id] = 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has changed recently and requires get_versioning_start from openpype.pipeline.version_start.

@jakubjezek001 jakubjezek001 force-pushed the feature/OP-5264_migrating_staging-dir-functions branch from 6967ca4 to 14f89e7 Compare October 13, 2023 13:35
@@ -723,6 +725,58 @@ def get_pre_create_attr_defs(self):
return self.pre_create_attr_defs


def apply_staging_dir(self, instance):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too many blank lines (2)

template_name: str = Field(
"",
title="Template Name",
placeholder="Shared templates: ayon+anatomy://templates/staging_directories"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line too long (84 > 79 characters)

@mkolar mkolar added sponsored Client endorsed or requested port to AYON labels Feb 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
host: Photoshop host: Substance Painter port to AYON size/S Denotes a PR changes 100-499 lines, ignoring general files sponsored Client endorsed or requested type: feature Larger, user affecting changes and completely new things
Projects
Status: Review In Progress
Development

Successfully merging this pull request may close these issues.

None yet

6 participants