Skip to content

Commit

Permalink
vpm: fix regression, so `v install sdl && v run ~/.vmodules/sdl/setup…
Browse files Browse the repository at this point in the history 
….vsh` works again
  • Loading branch information
@spytheman
spytheman committed Apr 17, 2024
1 parent 20c470a commit 27cc277
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion cmd/tools/vpm/vcs.v
View file
Expand Up @@ -29,7 +29,7 @@ fn init_vcs_info() !map[VCS]VCSInfo {
git_installed_raw_ver := parse_git_version(os.execute_opt('git --version')!.output) or { '' }
git_installed_ver := semver.from(git_installed_raw_ver)!
git_submod_filter_ver := semver.from('2.36.0')!
mut git_install_cmd := 'clone --depth=1 --recursive --shallow-submodules --filter=blob:none'
mut git_install_cmd := 'clone --recursive --shallow-submodules --filter=blob:none'
if git_installed_ver >= git_submod_filter_ver {
git_install_cmd += ' --also-filter-submodules'
}
Expand Down

6 comments on commit 27cc277

@ttytm
Copy link
Member

@ttytm ttytm commented on 27cc277 Apr 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this fix not be made on sdl or the commands side?

When sdls setup script requires a repo with deeper depth, then why not unshallowing it prior to running the script or - if it's possible -inside the script before trying to get branch information?

The cost seems too much in terms of performance and bandwidth.

@spytheman
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That requires, the authors of modules to know about not just their own repo's setup, but also more details about what vpm does (which is subject to change and evolution), and whether it was used or not (compared to just git clone some_repo_url).

I would prefer, if that is not required of them.

v install should automate module installation for users, and decrease the amount of efforts required by authors, in the first place, not make it harder, and less predictable.

The optimization, done by --depth=1 is nice to have, but should not be the default.

@spytheman
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cost seems too much in terms of performance and bandwidth.

The cost for most repos will be minimal.
It will be bigger for repos with thousands of commits, but those are rare, and even in those cases, it is seconds.

I do not have very fast internet connection to github (~2-3MB/s currently), and yet:

#0 03:38:35 ᛋ master /v/oo❱xtime git clone https://github.com/vlang/v base_v
Cloning into 'base_v'...
remote: Enumerating objects: 169613, done.
remote: Counting objects: 100% (14835/14835), done.
remote: Compressing objects: 100% (1014/1014), done.
remote: Total 169613 (delta 14197), reused 13953 (delta 13819), pack-reused 154778
Receiving objects: 100% (169613/169613), 68.55 MiB | 2.87 MiB/s, done.
Resolving deltas: 100% (120183/120183), done.
CPU: 15.95s     Real: 32.17s    Elapsed: 0:32.17        RAM: 61660KB    git clone https://github.com/vlang/v base_v
#0 03:39:13 ᛋ master /v/oo❱
#0 03:39:15 ᛋ master /v/oo❱
#0 03:39:15 ᛋ master /v/oo❱xtime git clone --depth=1 https://github.com/vlang/v v_depth_1
Cloning into 'v_depth_1'...
remote: Enumerating objects: 8545, done.
remote: Counting objects: 100% (8545/8545), done.
remote: Compressing objects: 100% (7399/7399), done.
remote: Total 8545 (delta 439), reused 6295 (delta 359), pack-reused 0
Receiving objects: 100% (8545/8545), 9.70 MiB | 3.13 MiB/s, done.
Resolving deltas: 100% (439/439), done.
CPU: 0.89s      Real: 5.19s     Elapsed: 0:05.19        RAM: 21136KB    git clone --depth=1 https://github.com/vlang/v v_depth_1
#0 03:39:33 ᛋ master /v/oo❱
#0 03:39:39 ᛋ master /v/oo❱du -s base_v
139748  base_v
#0 03:39:43 ᛋ master /v/oo❱du -s v_depth_1
74432   v_depth_1
#0 03:39:48 ᛋ master /v/oo❱

Note, that the V repo, currently has 17479 commits at the moment.

Note also that github do not recommend --depth=1 (shallow clones) for anything, but build environments where the repository will be deleted after a single build. https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/ .
A module/package repo, that can be updated (fast forwarded), or that can switch branches (like vlang/sdl's setup.vsh script does), will benefit more from a clone that is not shallow, and has the metadata for all commits, even if some of the blobs are not there initially).

@spytheman
Copy link
Member Author

@spytheman spytheman commented on 27cc277 Apr 18, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note also that --filter=blob:none (which is still kept), offers a compromise between developer convenience, and used network and disk resources:

#0 03:53:10 ᛋ master /v/oo❱xtime git clone --recursive --shallow-submodules --filter=blob:none https://github.com/vlang/v v_filter_blob_none
Cloning into 'v_filter_blob_none'...
remote: Enumerating objects: 107285, done.
remote: Counting objects: 100% (5187/5187), done.
remote: Compressing objects: 100% (376/376), done.
remote: Total 107285 (delta 4969), reused 4890 (delta 4809), pack-reused 102098
Receiving objects: 100% (107285/107285), 20.31 MiB | 3.20 MiB/s, done.
Resolving deltas: 100% (71372/71372), done.
remote: Enumerating objects: 7781, done.
remote: Counting objects: 100% (2551/2551), done.
remote: Compressing objects: 100% (2446/2446), done.
remote: Total 7781 (delta 177), reused 155 (delta 105), pack-reused 5230
Receiving objects: 100% (7781/7781), 9.42 MiB | 2.83 MiB/s, done.
Resolving deltas: 100% (418/418), done.
Updating files: 100% (7945/7945), done.
CPU: 6.65s      Real: 16.74s    Elapsed: 0:16.74        RAM: 158852KB   git clone --recursive --shallow-submodules --filter=blob:none https://github.com/vlang/v v_filter_blob_none
#0 03:54:07 ᛋ master /v/oo❱
#0 03:54:09 ᛋ master /v/oo❱du -s v_filter_blob_none
98936   v_filter_blob_none
#0 03:55:26 ᛋ master /v/oo❱

@spytheman
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are the results for vlang/sdl, which does not have as many commits, and is still bigger than most (because of its history, and its many branches):

#0 03:58:13 ᛋ master /v/oo❱xtime git clone https://github.com/vlang/sdl  sdl_full
Cloning into 'sdl_full'...
remote: Enumerating objects: 4170, done.
remote: Counting objects: 100% (670/670), done.
remote: Compressing objects: 100% (163/163), done.
remote: Total 4170 (delta 547), reused 589 (delta 492), pack-reused 3500
Receiving objects: 100% (4170/4170), 2.83 MiB | 3.16 MiB/s, done.
Resolving deltas: 100% (2853/2853), done.
CPU: 0.46s      Real: 1.90s     Elapsed: 0:01.90        RAM: 19048KB    git clone https://github.com/vlang/sdl sdl_full
#0 03:58:17 ᛋ master /v/oo❱
#0 03:58:20 ᛋ master /v/oo❱du -s sdl_full/
5672    sdl_full/
#0 03:58:30 ᛋ master /v/oo❱
#0 03:58:32 ᛋ master /v/oo❱xtime git clone --recursive --shallow-submodules --filter=blob:none https://github.com/vlang/sdl  sdl_filter_blob_none
Cloning into 'sdl_filter_blob_none'...
remote: Enumerating objects: 2158, done.
remote: Counting objects: 100% (497/497), done.
remote: Compressing objects: 100% (115/115), done.
remote: Total 2158 (delta 400), reused 442 (delta 367), pack-reused 1661
Receiving objects: 100% (2158/2158), 669.49 KiB | 2.42 MiB/s, done.
Resolving deltas: 100% (996/996), done.
remote: Enumerating objects: 98, done.
remote: Counting objects: 100% (61/61), done.
remote: Compressing objects: 100% (59/59), done.
remote: Total 98 (delta 3), reused 3 (delta 2), pack-reused 37
Receiving objects: 100% (98/98), 1.43 MiB | 2.71 MiB/s, done.
Resolving deltas: 100% (3/3), done.
Updating files: 100% (99/99), done.
CPU: 0.24s      Real: 2.21s     Elapsed: 0:02.21        RAM: 19124KB    git clone --recursive --shallow-submodules --filter=blob:none https://github.com/vlang/sdl sdl_filter_blob_none
#0 03:58:45 ᛋ master /v/oo❱du -s sdl_filter_blob_none/
4860    sdl_filter_blob_none/
#0 03:58:50 ᛋ master /v/oo❱
#0 03:58:51 ᛋ master /v/oo❱
#0 03:58:55 ᛋ master /v/oo❱xtime git clone --depth=1 https://github.com/vlang/sdl  sdl_depth_1
Cloning into 'sdl_depth_1'...
remote: Enumerating objects: 120, done.
remote: Counting objects: 100% (120/120), done.
remote: Compressing objects: 100% (108/108), done.
remote: Total 120 (delta 3), reused 58 (delta 2), pack-reused 0
Receiving objects: 100% (120/120), 1.43 MiB | 2.99 MiB/s, done.
Resolving deltas: 100% (3/3), done.
CPU: 0.10s      Real: 1.26s     Elapsed: 0:01.26        RAM: 19036KB    git clone --depth=1 https://github.com/vlang/sdl sdl_depth_1
#0 03:59:30 ᛋ master /v/oo❱
#0 03:59:31 ᛋ master /v/oo❱du -s sdl_depth_1/
4124    sdl_depth_1/
#0 03:59:36 ᛋ master /v/oo❱

@ttytm
Copy link
Member

@ttytm ttytm commented on 27cc277 Apr 18, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, the --filter=blog:none allows to preserve most benefits.

E.g. cloning v-analyzer(no submodules) [shallow clone: 14MB, filtered clone: 14,6MB, regular clone: 53.7MB].

v-analyzer stores some generated code, and for me, it was just the case that I had some negative experiences with projects that kept lots of generated code in the repository, along with regularly updated thirdparty and assets and binaries. The full history was huge there.

Thanks for the response 👍

Please sign in to comment.