Skip to content

⚡Fast filysystem fingerprinting using xxHash

License

Notifications You must be signed in to change notification settings

lunatic-cat/xxhashdir

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

xxhashdir

Build Status

Usage

this package does fast filesystem fingerprinting using xxHash

# instead of "find . -type f -exec xxhsum {} \+"
$ ./xxhashdir .
...
880788507839261490    README.md
11541949788444589007  .travis.yml
6467850080536788703   bin/xxhashdir.go
...

typical CLI use:

./xxhashdir dir > before
# modify fs
./xxhashdir dir > after
diff <(sort before) <(sort after) | sort -nk3

Speed

Times faster than find + exec. Digesting xcode-10.2 with >250K files:

Time Cmd
656 sec time find /Applications/Xcode.app -type f -exec xxhsum {} ; > xxhsum.txt
88 sec time find /Applications/Xcode.app -type f -exec xxhsum {} + > xxhsum.txt
45 sec time ./xxhashdir /Applications/Xcode.app > xxhsumdir.txt

Golang api

func Hashdir(root string, out chan Entry)

where

type Entry struct {
    Path   string
    Xxhash uint64
}