Recreate Towhee image embedding with timm

[orion512]

In order to fully understand how Towhee works I wanted to recreate image embeddings with timm.
However I can't seem to get the same result and Towhee is built in a way unfamiliar to me thus I can't find the source code I am missing.

EXAMPLE_IMG = "path to some image"

Towhee code

towhee.glob(*[EXAMPLE_IMG]).exception_safe() \
                    .image_decode() \
                    .image_embedding.timm(model_name="resnet50") \
                    .drop_empty() \
                    .tensor_normalize() \
                    .to_list()

This gives me a result like below

[array([0.        , 0.01364289, 0.00837596, ..., 0.        , 0.        ,
        0.        ], dtype=float32)]

While running towhee I looked into ~/.cache/torch/hub/checkpoint for the model weights that were downloaded so that I could use the same.

resnet50_a1_0-14fe96d1.pth

Timm code

Now I tried to recreating the same result purely in Timm.

import timm
import torch

# read image
read_image = PILImage.open(EXAMPLE_IMG)
arr_image = cv2.cvtColor(np.array(read_image), cv2.COLOR_RGB2BGR).astype('float32')
image = torch.as_tensor(arr_image).transpose(2,0)[None]

PATH_RES50_WEIGHTS = "/my_path/resnet50_a1_0-14fe96d1.pth"

model_timm = timm.create_model('resnet50', pretrained=False, checkpoint_path=PATH_RES50_WEIGHTS)
model_timm.reset_classifier(0)
output_cust = model_timm(image)

# same as towhee tensor_normalize()
x = np.array(output_cust.tolist()[0])
gen_vector = x / np.linalg.norm(x, axis=0)

gen_vector

The result I get however is different

array([0.03503343, 0.01141444, 0.01016041, ..., 0.00766325, 0.02729872,
       0.00292635])

I would like to understand how I can recreate the same result as produce by Towhee (the same vector).

[junjiejiangjjj]

https://towhee.io/image-embedding/timm/src/branch/main/timm_image.py
If use timm:

import timm
import torch
from PIL import Image as PILImage

from timm.data.transforms_factory import create_transform
from timm.data import resolve_data_config
from timm.models.factory import create_model

model_name = 'resnet50'
model = create_model(model_name, pretrained=True, num_classes=1000)
model.eval()
config = resolve_data_config({}, model=model)
tfms = create_transform(**config)


read_image = PILImage.open(EXAMPLE_IMG)
inputs = torch.stack([tfms(read_image)])
features = model.forward_features(inputs)
global_pool = torch.nn.AdaptiveAvgPool2d(1)
features = global_pool(features)
features = features.flatten(1)
vecs = features.squeeze(0).detach().numpy()
print(vecs)

[orion512]

Thanks a lot for this.
I was trying to figure it out for a while.

model.eval() seems to be what I was missing.

[orion512]

@junjiejiangjjj
I adjusted your code a little bit to run for multiple images.

model = create_model(MODEL_NAME, pretrained=False, checkpoint_path=PATH_RES50_WEIGHTS, num_classes=1000)
model.eval()
config = resolve_data_config({}, model=model)
tfms = create_transform(**config)

# read image & preprocess image
read_images = [PILImage.open(img) for img in MULTIPLE_IMAGE_PATHS]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)

# image through the model
inputs = inputs.to(device)
features = model.forward_features(inputs)

... the rest of the code is the same (but the problem can be observed after the forward_features call

If I read in 15 images and compare the results to towhee with 15 images, the results are identical.
But if I read process more than 15 images (20, 30, 200, etc.)
The timm code starts rounding the numbers differently. The difference won't be large but a few decimals will be different than the Towhee code.

I've been investigating whole afternoon to understand why but for the life of my I can't understand why with multiple images at some point the rounding starts to change.

[junjiejiangjjj]

Towhee will try to use GPU firstly
you can try to use timm with GPU

...
model = create_model(model_name, pretrained=True, num_classes=1000)
# 1
model.to('cuda')
model.eval()
...
inputs = torch.stack([tfms(read_image)])
# 2
inputs = inputs.to('cuda')
....
features = global_pool(features)
# 3
features = features.to('cpu').flatten(1)

[orion512]

@fzliu @junjiejiangjjj I did suspect CUDA first however I am running this on a machine with AMD GPU and CUDA is not available.
so this is running CPU only.

[junjiejiangjjj]

torch.cuda.is_available() try to run this,
it seems AMD reuse the CUDA interface
https://pytorch.org/docs/stable/notes/hip.html?highlight=amd#

[orion512]

Yeah I tried that.

'cuda' if torch.cuda.is_available() else 'cpu'

this return "cpu".

I also checked the source code that towhee uses for image_embeddings and this seems to be the same.

[orion512]

@junjiejiangjjj @fzliu

I prepared another piece of code which should explain the above problem exactly.

This completely confesses me.

Below I simply run forward features 3x times.

• once with first 10 images
• once with first 15 images
• once with first 30 images

keep in mind the first 10 images are always the same.
Then when I compare the results to each other the batch with 30 images produces different tensors.
I believe the difference is in grad_fn= but am not sure

# model init
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = create_model("resnet50", pretrained=True, num_classes=1000)
model.to(device)
model.eval()
config = resolve_data_config({}, model=model)
tfms = create_transform(**config)

# Preprocess 10 images
read_images = [PILImage.open(img) for img in all_demo_paths][:10]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)
inputs = inputs.to(device)
features_10 = model.forward_features(inputs)

# Preprocess 15 images
read_images = [PILImage.open(img) for img in all_demo_paths][:15]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)
inputs = inputs.to(device)
features_15 = model.forward_features(inputs)

# Preprocess 30 images
read_images = [PILImage.open(img) for img in all_demo_paths][:30]
imges_list = [tfms(img) for img in read_images]
inputs = torch.stack(imges_list)
inputs = inputs.to(device)
features_30 = model.forward_features(inputs)

for i in range(10):
    print(f"[10-15] Comparing tensor {i}: {torch.eq(features_10[i], features_15[i]).all()}")
    print(f"[10-30] Comparing tensor {i}: {torch.eq(features_10[i], features_30[i]).all()}")
    print("---")

the output is

[10-15] Comparing tensor 0: True
[10-30] Comparing tensor 0: False
---
[10-15] Comparing tensor 1: True
[10-30] Comparing tensor 1: False
---
[10-15] Comparing tensor 2: True
[10-30] Comparing tensor 2: False
---
[10-15] Comparing tensor 3: True
[10-30] Comparing tensor 3: False
---
[10-15] Comparing tensor 4: True
[10-30] Comparing tensor 4: False
---
[10-15] Comparing tensor 5: True
[10-30] Comparing tensor 5: False
---
[10-15] Comparing tensor 6: True
[10-30] Comparing tensor 6: False
---
[10-15] Comparing tensor 7: True
[10-30] Comparing tensor 7: False
---
[10-15] Comparing tensor 8: True
...
---
[10-15] Comparing tensor 9: True
[10-30] Comparing tensor 9: False
---