[CLI]: autologger not working for Stable Video Diffusion

[bdytx5]

Describe the bug

The autologger doesnt seem to work for stable video diffusion

from diffusers.utils import load_image, export_to_video
from diffusers import StableVideoDiffusionPipeline
import torch
import os
import numpy as np
import wandb
from PIL import Image


from wandb.integration.diffusers import autolog


# Initialize the video diffusion pipeline
pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid", torch_dtype=torch.float16, variant="fp16"
)
pipe.enable_model_cpu_offload()

autolog(init=dict(project="sdv-autologger"))

# List of image URLs


# List of image URLs
image_urls = [
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png",
]


generator = torch.manual_seed(42)


motion_buckets = [120, 180, 240, 300, 360]  # Example motion bucket IDs
noise_aug_strength = 0.1  # Keeping noise augmentation strength constant


# Iterate over each motion bucket first
for bucket in motion_buckets:
    # Initialize Weights & Biases run for each bucket
    # wandb.init(entity='byyoung3', project='mlnews2', name=f'bucket_{bucket}', reinit=True, group='exp_1')


    # Then, process each image for the current motion bucket
    for i, url in enumerate(image_urls):
        # Load the conditioning image
        image = load_image(url).convert("RGB")  # Ensure image is in RGB format
        image = image.resize((1024, 576))


        # Generate video frames with the current motion bucket
        result = pipe(image, decode_chunk_size=8, generator=generator, motion_bucket_id=bucket, noise_aug_strength=noise_aug_strength)


        # Convert frames to numpy array and transpose the axes to (time, channel, height, width)
        frames_np = np.stack([np.array(frame) for frame in result.frames[0]])
        frames_np = frames_np.transpose((0, 3, 1, 2))  # Transpose to (time, channels, height, width)


        # Log video to wandb with specific motion bucket ID
        # wandb.log({f"image_{i}": wandb.Video(frames_np, fps=7, format="mp4")})


    # Finish the wandb run for the current motion bucket
    wandb.finish()

Additional Files

Media doesnt show up in wandb

Environment

WandB version: Newest

OS: Linux

Python version: 3.8

Versions of relevant libraries: Newest

Additional Context

No response

[ArtsiomWB]

Hi @bdytx5, thank you for writing in. Can you send me your whole error stack that you are seeing when running the code above. Just ran the code myself and would love to see if we are seeing a similar behavior.

[bdytx5]

when I get a chance, I will run it again. I had created a few different versions of the script (1 of which did have a wandb"init" error of some sort). But I was able to somehow get rid of that error, with the issue still being that no media was being uploaded to wandb. Can you confirm it was an error related to wandb.init ?

[ArtsiomWB]

Not sure if this error is fully related to wandb. Are you still not able to see any media in wandb, or have you been able to resolve this already as well?

[ArtsiomWB]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[bdytx5]

As far as I know the logger simply does not log the videos. Can you confirm that you have tested this? I tried several variations of the script, and videos are not logged.