"Detect missing frame + Warning: the value is empty for raft" ERROR when extraction_fps=1

[TikaToka]

Hi, first of all, thank you for sharing your amazing work!

I am trying to extract optical features for Charades-sta dataset with command

python main.py \
    feature_type=raft \
    batch_size=8 \
    extraction_fps=1
    on_extraction=save_numpy \
    device="cuda:4" \
    keep_tmp_files=true \
    video_paths="[/home/jckim/workspace/video_features/tmp/raft/0A8CF_new_fps.mp4]"

(never mind for video_paths the reason for new_fps video's are there is that i've tested if this was problem of re-encoding. neither raw video or re-encoded videos occur same problem)
With the command, I could get 1fps re-encoded video. But when extracting features,

happens like this

and the features extraceted are all blank.

Is there any way to solve this problem?

Thank you in advance!

[v-iashin]

Hi, could you attach the video file?

[TikaToka]

WPR8G.mp4

sample videos are here

name: WPR8G.mp4
from Charades dataset

[v-iashin]

Sorry for the late response but have you tried to use a smaller batch_size?

I tried to reproduce this problem on google colab but I couldn't. Could you try as well?

https://colab.research.google.com/drive/18I95Rn1B3a2ISfD9b-o4o93m3XuHbcIY?usp=sharing

[TikaToka]

i've also tried with batch_size =1 but same result

Also, as i need a npy file and there are a lots of videos in charades, I can't use colab as workspace :(

[TikaToka]

also tried your jupyter file in my server, does not work

[TikaToka]

For extra information)
In colab works well as you mentioned.. idk why it does not work only at my server :(

• pip freeze result

antlr4-python3-runtime @ file:///home/conda/feedstock_root/build_artifacts/antlr-python-runtime-meta_1638309185939/work
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1670263926556/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1659291887007/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
certifi==2022.12.7
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1671179356964/work
comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1670575068857/work
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1674522341001/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work
ffmpeg-python==0.2.0
ftfy @ file:///home/conda/feedstock_root/build_artifacts/ftfy_1647200718722/work
future @ file:///home/conda/feedstock_root/build_artifacts/future_1673596611778/work
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1672612343532/work
iniconfig @ file:///home/conda/feedstock_root/build_artifacts/iniconfig_1603384189793/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1676322140253/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1676047456691/work
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1675178684608/work
jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1675109858760/work
llvmlite==0.36.0
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mkl-fft==1.2.0
mkl-random==1.1.1
mkl-service==2.3.0
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
numba @ file:///tmp/build/80754af9/numba_1616774046117/work
numpy @ file:///tmp/build/80754af9/numpy_and_numpy_base_1596233721170/work
olefile==0.46
omegaconf @ file:///home/conda/feedstock_root/build_artifacts/omegaconf_1636817774483/work
opencv-python==4.4.0.44
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow @ file:///tmp/build/80754af9/pillow_1594307295532/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1670613602756/work
pluggy @ file:///home/conda/feedstock_root/build_artifacts/pluggy_1648772598846/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1670414775770/work
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1667885878918/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
py @ file:///home/conda/feedstock_root/build_artifacts/py_1636301881863/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1672682006896/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work
pytest==7.1.2
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
PyYAML==5.3.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1673612649529/work
regex @ file:///tmp/build/80754af9/regex_1648447708112/work
resampy==0.2.2
scipy @ file:///tmp/build/80754af9/scipy_1597686649129/work
six==1.15.0
SoundFile @ file:///home/conda/feedstock_root/build_artifacts/pysoundfile_1607011569782/work
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
torch==1.7.1
torchaudio==0.7.0a0+a853dff
torchvision==0.8.2
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1666788592778/work
tqdm @ file:///tmp/build/80754af9/tqdm_1600709023549/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1638334978229/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1675982654259/work

[v-iashin]

does it work with the provided conda environment?

If not, one option could be to use a docker environment (or a singularity container if docker is not allowed)

What might be the problem? I think you need to check which ffmpeg you are using and try the command:

ffmpeg -i original.mp4 -filter:v fps=fps=1 original_1fps.mp4

and check the video.

[TikaToka]

Also Tried at docker env before.. but same error occurs...)

jckim@vtt-server:~/workspace/video_features$ docker run --gpus '"device=0"' --name=jckim --shm-size 8G -i -t -v /data:/data iashin/video_features:latest /bin/bash

(torch_zoo) ubuntu@8807696ec356:~/video_features$ ffmpeg -i /data/mjjung/Charades_v1/WPR8G.mp4 -filter:v fps=fps=1 WPR8G_1fps.mp4
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 7.5.0 (crosstool-NG 1.24.0.131_87df0e6_dirty)
configuration: --prefix=/home/ubuntu/miniconda3/envs/torch_zoo --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1596712246804/_build_env/bin/x86_64-conda-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-gpl --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-libx264 --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame
libavutil 56. 51.100 / 56. 51.100
libavcodec 58. 91.100 / 58. 91.100
libavformat 58. 45.100 / 58. 45.100
libavdevice 58. 10.100 / 58. 10.100
libavfilter 7. 85.100 / 7. 85.100
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 7.100 / 5. 7.100
libswresample 3. 7.100 / 3. 7.100
libpostproc 55. 7.100 / 55. 7.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/data/mjjung/Charades_v1/WPR8G.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf56.40.101
Duration: 00:00:31.30, start: 0.000000, bitrate: 1561 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1426 kb/s, 16.63 fps, 16.63 tbr, 16631 tbn, 33.26 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 131 kb/s (default)
Metadata:
handler_name : SoundHandler
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
[libx264 @ 0x55c6905c4980] using SAR=1/1
[libx264 @ 0x55c6905c4980] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0x55c6905c4980] profile High, level 3.1
[libx264 @ 0x55c6905c4980] 264 - core 152 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=22 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=1 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'WPR8G_1fps.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.45.100
Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=-1--1, 1 fps, 16384 tbn, 1 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc58.91.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
Metadata:
handler_name : SoundHandler
encoder : Lavc58.91.100 aac
frame= 22 fps=0.0 q=0.0 size= 0kB time=00:00:23.35 bitrate= 0.0kframe= 31 fps= 20 q=-1.0 Lsize= 2135kB time=00:00:31.27 bitrate= 559.1kbits/s speed=20.4x
video:1635kB audio:493kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.330588%
[libx264 @ 0x55c6905c4980] frame I:2 Avg QP:11.66 size: 67910
[libx264 @ 0x55c6905c4980] frame P:18 Avg QP:13.28 size: 56428
[libx264 @ 0x55c6905c4980] frame B:11 Avg QP:13.64 size: 47428
[libx264 @ 0x55c6905c4980] consecutive B-frames: 45.2% 19.4% 9.7% 25.8%
[libx264 @ 0x55c6905c4980] mb I I16..4: 17.6% 57.2% 25.2%
[libx264 @ 0x55c6905c4980] mb P I16..4: 14.1% 56.7% 16.5% P16..4: 5.5% 5.4% 1.5% 0.0% 0.0% skip: 0.4%
[libx264 @ 0x55c6905c4980] mb B I16..4: 7.2% 26.9% 8.7% B16..8: 20.0% 21.7% 4.9% direct: 9.6% skip: 1.0% L0:50.2% L1:38.5% BI:11.3%
[libx264 @ 0x55c6905c4980] 8x8 transform intra:63.8% inter:58.9%
[libx264 @ 0x55c6905c4980] coded y,uvDC,uvAC intra: 71.2% 89.1% 68.8% inter: 64.7% 93.9% 26.3%
[libx264 @ 0x55c6905c4980] i16 v,h,dc,p: 24% 26% 14% 36%
[libx264 @ 0x55c6905c4980] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 26% 13% 4% 4% 5% 8% 5% 6%
[libx264 @ 0x55c6905c4980] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 36% 27% 9% 3% 6% 5% 8% 4% 3%
[libx264 @ 0x55c6905c4980] i8c dc,h,v,p: 40% 22% 27% 12%
[libx264 @ 0x55c6905c4980] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x55c6905c4980] ref P L0: 55.8% 16.9% 16.5% 10.7%
[libx264 @ 0x55c6905c4980] ref B L0: 83.6% 15.7% 0.8%
[libx264 @ 0x55c6905c4980] ref B L1: 91.5% 8.5%
[libx264 @ 0x55c6905c4980] kb/s:431.80

Result Video

WPR8G_1fps.mp4

(torch_zoo) ubuntu@8807696ec356:~/video_features$ bash run.sh
feature_type: raft
extraction_fps: 1
extraction_total: null
side_size: null
resize_to_smaller_edge: true
finetuned_on: sintel
device: cuda:0
batch_size: 1
on_extraction: save_numpy
output_path: ./output/raft
tmp_path: ./tmp/raft
keep_tmp_files: true
show_pred: false
config: null
video_paths:

• /data/mjjung/Charades_v1/WPR8G.mp4
  file_with_video_paths: null

Saving features to ./output/raft
Device: cuda:0
The number of specified videos: 1
0%| | 0/1 [00:00<?, ?it/s]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/WPR8G_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/WPR8G_timestamps_ms.npy
100%|███████████████████████████████████████| 1/1 [00:01<00:00, 1.71s/it]

[TikaToka]

• new information found)

I tried to extract features for all videos, most of the videos had a same problem, but few returned features well.. (about 1~2%? videos succeeded. tried at docker env provided)

(torch_zoo) ubuntu@c4dd7bf2d508:~/video_features$ python main.py \
>     feature_type=raft \
>     on_extraction=save_numpy \
>     keep_tmp_files=true \
>     file_with_video_paths=charades.txt \
>     extraction_fps=1

feature_type: raft
extraction_fps: 1
extraction_total: null
side_size: null
resize_to_smaller_edge: true
finetuned_on: sintel
device: cuda:0
batch_size: 1
on_extraction: save_numpy
output_path: ./output/raft
tmp_path: ./tmp/raft
keep_tmp_files: true
show_pred: false
config: null
video_paths: null
file_with_video_paths: charades.txt

Saving features to ./output/raft
Device: cuda:0
The number of specified videos: 9848
0%| | 0/9848 [00:00<?, ?it/s]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/JRSPH_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/JRSPH_timestamps_ms.npy
0%| | 1/9848 [00:02<5:49:03, 2.13s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/KVWRK_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/KVWRK_timestamps_ms.npy
0%| | 2/9848 [00:03<5:02:10, 1.84s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/TVBLW_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/TVBLW_timestamps_ms.npy
0%| | 4/9848 [00:16<13:03:53, 4.78s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/X42YA_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/X42YA_timestamps_ms.npy
0%| | 5/9848 [00:18<10:14:36, 3.75s/it]Detect missing frame
0%| | 6/9848 [00:24<12:24:33, 4.54s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/2L6E2_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/2L6E2_timestamps_ms.npy
0%| | 7/9848 [00:27<11:15:52, 4.12s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/D260T_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/D260T_timestamps_ms.npy
0%| | 8/9848 [00:29<9:08:19, 3.34s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/55PFS_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/55PFS_timestamps_ms.npy
0%| | 9/9848 [00:30<7:20:06, 2.68s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/VVMYX_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/VVMYX_timestamps_ms.npy
0%| | 10/9848 [00:31<6:24:43, 2.35s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/5C3NY_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/5C3NY_timestamps_ms.npy
0%| | 12/9848 [00:40<10:02:10, 3.67s/it]Detect missing frame
0%| | 13/9848 [00:46<11:42:47, 4.29s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/9SP9P_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/9SP9P_timestamps_ms.npy
0%| | 14/9848 [00:48<9:32:13, 3.49s/it]Detect missing frame
Warning: the value is empty for raft @ ./output/raft/7YV59_raft.npy
Warning: the value is empty for timestamps_ms @ ./output/raft/7YV59_timestamps_ms.npy