Preserve original path for archives; add to output #180

egibs · 2024-05-03T00:34:53Z

Closes: #178

When scanning archives, we only showed the temporary directory used to hold and extract the archive. This PR preserves the originally-provided path and displays it when scanning archives.

This change applies to both normal scans as well as diffs (output provided in the comments below).

Examples --

Terminal:

Scanned Path: /var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_amd64.tar.gz4091940928/apko_0.13.2_darwin_amd64/apko [🚨 CRITICAL]
Original Path: /Users/egibs/Downloads/apko_tar_gzs/apko_0.13.2_darwin_amd64.tar.gz

---------------------------------------------------------------------------------------------------------------------------------------------------
RISK  KEY                              DESCRIPTION                                           EVIDENCE
---------------------------------------------------------------------------------------------------------------------------------------------------
LOW   compression/bzip2                Works with bzip2 files                                bzip2
LOW   compression/gzip                 works with gzip files                                 gzip
LOW   compression/zstd                 Zstandard: fast real-time compression algorithm       (�/�
                                                                                             zstd
LOW   crypto/aes                       Supports AES (Advanced Encryption Standard)           AES
                                                                                             crypto/aes
LOW   crypto/ecdsa                     Uses the Go crypto/ecdsa library                      crypto/ecdsa
LOW   crypto/ed25519                   Elliptic curve algorithm used by TLS and SSH          ed25519
LOW   crypto/tls                       tls                                                   TLS13
                                                                                             TLSVersion
                                                                                             crypto/tls
LOW   data/embedded/pem/certificate    Contains embedded PEM certificate                     -----BEGIN CERTIFICATE-----
LOW   data/embedded/pem/test_key       Contains TESTING KEY directive                        TESTING KEY-----
LOW   data/embedded/ssh/signature      Contains embedded SSH signature                       --BEGIN SSH SIGNATURE--
LOW   encoding/base64                  Supports base64 encoded strings                       base64
LOW   encoding/json                    Supports JSON encoded objects                         encoding/json
LOW   encoding/json/decode             Decodes JSON messages                                 json.Unmarshal
LOW   encoding/json/encode             encodes JSON                                          MarshalJSON
LOW   env/HOME                         Looks up the HOME directory for the current user      HOME
                                                                                             getenv
LOW   env/USER                         Looks up the USER name of the current user            USER
                                                                                             getenv
LOW   fs/blkid                         works with block device attributes                    blkid
LOW   fs/directory/create              creates directories                                   mkdir
LOW   fs/directory/list                Uses Go functions to list a directory                 .OpenDir
                                                                                             .ReadDir
LOW   fs/directory/remove              Uses libc functions to remove directories             Rmdir
                                                                                             rmdir
LOW   fs/fifo/create                   make a FIFO special file (a named pipe)               mkfifo
LOW   fs/file/delete                   deletes files                                         unlinkat
LOW   fs/file/read                     reads files                                           ioutil.ReadFile
                                                                                             os.(*File).Read
LOW   fs/file/stat                     access filesystem information                         fs.statDirEntry
LOW   fs/file/truncate                 truncate a file to a specified length                 ftruncate
LOW   fs/link/create                   May create hard file links                            _link
LOW   fs/link/read                     read value of a symbolic link                         readlink
LOW   fs/lock/update                   apply or remove an advisory lock on a file            flock
LOW   fs/mount                         mounts file systems                                   -o
                                                                                             mount
LOW   fs/node/create                   create device files                                   mknod
LOW   fs/swap/off                      stop swapping to a file/device                        swapoff
LOW   fs/swap/on                       start swapping to a file/device                       swapon
LOW   fs/symlink/resolve               resolves symbolic links                               realpath
LOW   fs/tempfile/create               Uses mktemp to create temporary files                 mktemp
                                                                                             temp file
LOW   fs/unmount                       unmount file system                                   umount
LOW   hash/blake2b                     Uses blake2b encryption algorithm                     blake2b
LOW   kernel/cpu/info                  gets number of processors                             nproc
LOW   kernel/pivot_root                change the root mount location                        pivot_root
LOW   net/dns                          Uses DNS (Domain Name Service)                        CNAMEResource
                                                                                             SetEDNS0
                                                                                             dnsmessage
LOW   net/dns/txt                      Uses DNS TXT (text) records                           TXT
                                                                                             dns
LOW   net/hostname/resolve             resolve network host name to IP address               LookupHostIP
                                                                                             net.hostLookup
LOW   net/hostport/parse               Network address and service translation               freeaddrinfo
                                                                                             getaddrinfo
LOW   net/http/accept/encoding         set HTTP response encoding format (example: gzip)     Accept-Encoding
LOW   net/http/auth                    makes HTTP requests with basic authentication         WWW-Authenticate
                                                                                             Www-Authenticate
                                                                                             www-authenticate
LOW   net/http/request                 makes HTTP requests                                   HTTP/1.
                                                                                             Referer
                                                                                             User-Agent
LOW   net/http2                        Uses the HTTP/2 protocol                              HTTP/2
LOW   net/http_proxy                   use HTTP proxy that requires authentication           Proxy-Authorization
LOW   net/ip                           access the internet                                   invalid packet
LOW   net/ip/multicast/send            send data to multiple nodes simultaneously            multicast
LOW   net/sendfile                     transfer data between file descriptors                sendfile
                                                                                             syscall.Sendfile
LOW   net/socket/listen                listen on a socket                                    accept
                                                                                             listen
                                                                                             socket
LOW   net/socket/local/address         get local address of connected socket                 getsockname
LOW   net/socket/peer/address          get peer address of connected socket                  getpeername
LOW   net/socket/receive               receive a message from a socket                       recvfrom
                                                                                             recvmsg
LOW   net/socket/send                  send a message to a socket                            sendmsg
                                                                                             sendto
LOW   net/udp/receive                  Listens for UDP responses                             ReadFromUDP
                                                                                             listenUDP
LOW   net/udp/send                     Sends UDP packets                                     DialUDP
                                                                                             WriteMsgUDP
LOW   net/url                          Handles URL strings                                   RequestURI
LOW   process/chroot                   change the location of root for the process           chroot
LOW   process/create                   create child process                                  _fork
LOW   process/multithreaded            creates pthreads                                      pthread_create
LOW   process/unshare                  disassociate parts of the process execution context   unshare
LOW   ref/path/bin/su                  Calls /bin/su                                         /bin/su
LOW   ref/path/etc                     path reference within /etc                            /etc/apache/mime.typeshpack
                                                                                             /etc/apk/keys/etc/apk/archcached
                                                                                             /etc/apk/lib/apk
                                                                                             /etc/apk/repositories/lib/apk/db/inst
                                                                                             /etc/apk/world
                                                                                             /etc/bash
                                                                                             /etc/busybox-paths.d/usr/bin/setkeyco
                                                                                             /etc/default/motd-newsformat
                                                                                             …
LOW   ref/path/etc/resolv.conf         accesses DNS resolver configuration                   /etc/resolv.conf
LOW   ref/path/home/config             path reference within ~/.config                       ~/.config/fish/completions/
LOW   ref/path/home_library            path reference within ~/Library                       /System/Library/Frameworks/CoreFoundation
                                                                                             /System/Library/Frameworks/Security
                                                                                             offset/Library/Caches is not definedwrite heap dump…
LOW   ref/path/usr/bin                 path reference within /usr/bin                        /usr/bin/ar/usr/bin/bc/usr/bin/dc/usr/bin/du/usr/bi…
                                                                                             /usr/bin/ascii/usr/bin/crc32/usr/bin/tsortVERSION_ID
                                                                                             /usr/bin/awk/usr/bin/cal/usr/bin/cmp/usr/bin/cut/us…
                                                                                             /usr/bin/basename/usr/bin/dos2unix/usr/bin/dpkg-deb…
                                                                                             /usr/bin/bc/usr/bin/dc/usr/bin/du/usr/bin/hd/usr/bi…
                                                                                             /usr/bin/beep/usr/bin/chrt/usr/bin/chvt/usr/bin/com…
                                                                                             /usr/bin/blkdiscard/usr/bin/dumpleases/usr/bin/ssl_…
                                                                                             /usr/bin/bunzip2/usr/bin/crontab/usr/bin/cryptpw/us…
                                                                                             …
LOW   ref/path/usr/sbin                path reference within /usr/sbin                       /usr/sbin/add-shell/usr/sbin/dhcprelay/usr/sbin/get…
                                                                                             /usr/sbin/addgroup/usr/sbin/chpasswd/usr/sbin/delgr…
                                                                                             /usr/sbin/adduser/usr/sbin/deluser/usr/sbin/flashcp…
                                                                                             /usr/sbin/arping/usr/sbin/chroot/usr/sbin/i2cget/us…
                                                                                             /usr/sbin/brctl/usr/sbin/crond/usr/sbin/fbset/usr/s…
                                                                                             /usr/sbin/chat/usr/sbin/dnsd/usr/sbin/ftpd/usr/sbin…
                                                                                             /usr/sbin/chpasswd/usr/sbin/delgroup/usr/sbin/fdfor…
                                                                                             /usr/sbin/chroot/usr/sbin/i2cget/usr/sbin/i2cset/us…
                                                                                             …
LOW   ref/path/var                     path reference within /var                            /var/cache%s
                                                                                             /var/cache/apk/etc/apk/worldCalculateWorldcache
                                                                                             /var/cache/miscAPKINDEX.tar.gzfetchAlpineKeyscfg.Ma…
                                                                                             /var/lib/db/sbomSPDXRef-Package-remote
                                                                                             /var/run/docker.sockopen
LOW   ref/site/url                     contains embedded HTTPS URLs                          https://GoString01234567beEfFgGvsignal
                                                                                             https://alpinelinux.org/releases.jsondid
                                                                                             https://github.com/chainguard-dev/apkocould
                                                                                             https://github.com/google/go-containerregistry/issu…
                                                                                             https://github.com/spf13/cobra/issues/1279
                                                                                             https://github.com/spf13/cobra/issues/1508
                                                                                             https://index.docker.io/v1/Path
                                                                                             https://index.docker.io/v2/library/ubuntu/tags/list
                                                                                             …
LOW   ref/words/password               references a 'password'                               IncorrectPasswordError
                                                                                             Password from
                                                                                             PasswordHashIterations
                                                                                             UserPassword
                                                                                             and password requiredreading
                                                                                             bson bytes as PasswordGODEBUG sys
                                                                                             passwordSet
                                                                                             passwordStdin
                                                                                             …
LOW   secrets/private_key              References private keys                               privateKey
                                                                                             private_key
LOW   time/clock/set                   set time via system clock                             adjtimex
MED   archives/zip                     Works with zip files                                  archive/zip
MED   combo/dropper/bash               may fetch file, make it executable, and run it        ./b
                                                                                             ./c
                                                                                             ./jb
                                                                                             ./line
                                                                                             ./pipe/docker
                                                                                             ./q
                                                                                             ./r
                                                                                             ./v
                                                                                             …
MED   combo/stealer/ssh                possible SSH stealer                                  .ssh
                                                                                             curl
                                                                                             socket
                                                                                             tar
                                                                                             wget
                                                                                             zip
MED   data/embedded/zstd               Contains compressed content in ZStandard format       (�/�
MED   evasion/content/length/0         Sets HTTP content length to zero                      Content-Length: 0
MED   exec/program                     executes external programs                            ).CombinedOutput
                                                                                             exec.(*Cmd).Run
MED   fs/permission/chown              Changes file ownership                                Chown
MED   fs/permission/modify             modifies file permissions                             Chmod
                                                                                             chmod
MED   kernel/ptrace                    trace or modify system calls                          ptrace
MED   kernel/uname/get                 system identification (uname)                         uname
MED   net/dns/reverse                  looks up the reverse hostname for an IP               .in-addr.arpa
                                                                                             ip6.arpa
MED   net/download                     download files                                        DownloadLocation
                                                                                             downloadLocation
                                                                                             to registrySkip downloading
MED   net/fetch                        Invokes curl                                          curl -H "
MED   net/http/cookies                 access HTTP resources using cookies                   Cookie
                                                                                             HTTP
MED   net/http/post                    submit content to websites                            HTTP
                                                                                             POST
                                                                                             http
MED   net/interface/list               list network interfaces                               ifconfig
MED   net/ip/parse                     parses IP address (IPv4 or IPv6)                      IsLinkLocalUnicast
                                                                                             IsSingleIP
MED   net/mac/address                  Retrieves network MAC address                         MAC address
MED   net/socket/connect               initiate a connection on a socket                     _connect
MED   net/socks5                       Supports SOCK5 proxies                                SOCKS5
                                                                                             socks5
MED   net/ssh                          Uses crypto/ssh to connect to the SSH (secure shell)  crypto/ssh
                                       service
MED   net/stat                         Uses 'netstat' for network information                netstat
MED   net/upload                       uploads files                                         UPLOAD
                                                                                             Upload
                                                                                             upload
MED   net/url/encode                   encodes URL, likely to pass GET variables             urlencode
MED   net/url/request                  requests resources via URL                            http.request
                                                                                             net/url
MED   process/find                     Finds program in process table                        pgrep
MED   process/username/get             returns the user name running this process            whoami
MED   ref/path/etc/hosts               references /etc/hosts                                 /etc/hosts
MED   ref/path/hidden                  hidden path generated dynamically                     %s/.ssh
MED   ref/path/home                    peferences path within /home                          /home/sha2561.32.11.33.01.33.11.33.21.34.01.34.11.3…
MED   ref/path/relative                references and possibly executes relative path        ./jb
                                                                                             ./line
                                                                                             ./pipe
MED   ref/path/root                    path reference within /root                           /root/linuxrc/sbin/hwclock/sbin/ipneigh/sbin/iprout…
MED   ref/path/usr/local               path reference within /usr/local/bin                  /usr/local/bin
MED   ref/words/server_address         references a 'server address', possible C2 client     serverAddress
MED   secrets/keychain                 May access the macOS keychain                         Keychain
                                                                                             keychain
MED   secrets/ssh                      accesses SSH configuration and/or keys                /.ssh/known_hosts
                                                                                             found.ssh
                                                                                             plumbing/object.sshSignatureFormat
                                                                                             repository.ssh
                                                                                             ssh.sshConn
                                                                                             ssh_config.sshLexStateFn
                                                                                             ssh_config.sshLexer
                                                                                             ssh_config.sshParser
                                                                                             …
MED   security_controls/linux/selinux  selinux                                               setenforce
MED   shell/background/sleep           calls sleep and runs shell code in the background     #!
                                                                                             2>&1 &
                                                                                             nohup
MED   shell/exec                       executes shell                                        /bin/bash
                                                                                             /bin/sh
CRIT  third_party/                     Detection patterns for the tool 'RDPassSpray' taken   netcat
      mthcht_thk_yara_rules            from the ThreatHunting-Keywords github project, by
                                       @mthcht
---------------------------------------------------------------------------------------------------------------------------------------------------

Markdown (screenshot of the headers for brevity):

egibs · 2024-05-03T01:07:50Z

I can work on appending the file name to the original path.

Edit: Added in 2b61953 (#180).

egibs · 2024-05-03T01:38:26Z

Improved the diff output in 0a275a7 (#180). It would be nice to store both original paths, though. 🤔

❯ go run . --diff ~/Downloads/apko_tar_gzs/apko_0.13.2_darwin_amd64.tar.gz ~/Downloads/apko_tar_gzs_2/apko_0.13.2_darwin_arm64.tar.gz
Moved: ../../../../../var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_amd64.tar.gz2511095900/apko_0.13.2_darwin_amd64/apko -> ../../../../../var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_arm64.tar.gz178042758/apko_0.13.2_darwin_arm64/apko (score: 0.941791)

Original Path: /Users/egibs/Downloads/apko_tar_gzs_2/apko_0.13.2_darwin_arm64.tar.gz > apko

+++ ADDED: 3 behavior(s) +++

------------------------------------------------------------------------------
RISK  KEY                          DESCRIPTION                      EVIDENCE
------------------------------------------------------------------------------
+LOW  process/chdir                changes working directory        cd H2l
+MED  net/bpf                      BPF (Berkeley Packet Filter)     bpf
+MED  security_controls/linux/ufw  interacts with the ufw firewall  ufw
------------------------------------------------------------------------------

Edit: even better output added in 532088f (#180) (full Markdown rendered as a showcase):

Moved: ../../../../../var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_amd64.tar.gz2494270430/apko_0.13.2_darwin_amd64/apko -> ../../../../../var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_arm64.tar.gz2377260856/apko_0.13.2_darwin_arm64/apko (similarity: 0.95)

Original Path (From): /Users/egibs/Downloads/apko_tar_gzs/apko_0.13.2_darwin_amd64.tar.gz > apko

Original Path (To): /Users/egibs/Downloads/apko_tar_gzs_2/apko_0.13.2_darwin_arm64.tar.gz > apko

3 new behaviors

RISK	KEY	DESCRIPTION	EVIDENCE
+MEDIUM	net/bpf	BPF (Berkeley Packet Filter)	bpf
+MEDIUM	security_controls/linux/ufw	interacts with the ufw firewall	ufw
+LOW	process/chdir	changes working directory	cd H2l

tstromberg · 2024-05-03T12:03:53Z

pkg/bincapz/bincapz.go

-	Path   string
-	SHA256 string
+	Path          string
+	AlternatePath string


Is there a more specific name we can use there?

It isn't readily apparent to readers here what the difference between Path and AlternativePath is.

Addressed in bf5a1eb (#180).

tstromberg · 2024-05-03T12:08:46Z

samples/Windows/2024.Sharp/sharpil_RAT.exe.md

@@ -1,4 +1,4 @@
-## Windows/2024.Sharp/sharpil_RAT.exe [🚨 CRITICAL]
+## Scanned Path: Windows/2024.Sharp/sharpil_RAT.exe [🚨 CRITICAL]


Can we go back to the original output here?

Addressed in bf5a1eb (#180).

…uard-dev#169) * improve heartbeat detection * Update testdata, include 3rd party rules * shorter ref link

* Make --ignore-self more precise * update flag description

tstromberg · 2024-05-07T20:17:33Z

pkg/bincapz/bincapz.go

+
+	// The original path for scanned archive files
+	// When not scanning archives, this will be empty
+	OriginalAbsPath string


Add json:",omitempty" yaml:",omitempty" so that the empty field doesn't show up in JSON output.

I'm unclear on the relationship between Path and OriginalAbsPath. If Path is the original path requested by the user, is this just the absolute version of it?

On a related note - we should make sure that we're not storing any temporary file paths within the struct, as they aren't useful to the reader, as that file location no longer exists when they read the report.

We should however store the relative location within an archive somewhere so that they can investigate further.

egibs · 2024-05-08T22:23:20Z

I'm going to sit on this and ideate a bit more.

egibs requested a review from tstromberg May 3, 2024 00:34

tstromberg requested changes May 5, 2024

View reviewed changes

egibs requested a review from tstromberg May 5, 2024 15:55

egibs and others added 9 commits May 7, 2024 08:09

Preserve original path for archives; add to output

914d5f3

Display scanned file next to original path

dcfdfd8

Add original path of moved file to diff output

66637ea

Add from/to original paths to diff output

7e16599

Fix tests

bf2e861

Formatting fix

8cb6eff

Address PR comments

b8eb7eb

Improve rules based on LightSpy + add Huntress to third_party (chaing…

ad7f455

…uard-dev#169) * improve heartbeat detection * Update testdata, include 3rd party rules * shorter ref link

Make --ignore-self more precise (chainguard-dev#194)

3e5926c

* Make --ignore-self more precise * update flag description

egibs force-pushed the archive-original-path branch from bde3ff2 to 3e5926c Compare May 7, 2024 13:11

egibs added 2 commits May 7, 2024 08:12

Fix errant change

c7ca09d

More rebase fixes

cd28a28

tstromberg requested changes May 7, 2024

View reviewed changes

egibs closed this May 8, 2024

egibs mentioned this pull request May 11, 2024

Clean up displayed paths when encountering archives #217

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preserve original path for archives; add to output #180

Preserve original path for archives; add to output #180

egibs commented May 3, 2024 •

edited

egibs commented May 3, 2024 •

edited

egibs commented May 3, 2024 •

edited

tstromberg May 3, 2024

egibs May 5, 2024

tstromberg May 3, 2024

egibs May 5, 2024

tstromberg May 7, 2024

tstromberg May 7, 2024

egibs commented May 8, 2024

		@@ -1,4 +1,4 @@
		## Windows/2024.Sharp/sharpil_RAT.exe [🚨 CRITICAL]
		## Scanned Path: Windows/2024.Sharp/sharpil_RAT.exe [🚨 CRITICAL]

Preserve original path for archives; add to output #180

Preserve original path for archives; add to output #180

Conversation

egibs commented May 3, 2024 • edited

egibs commented May 3, 2024 • edited

egibs commented May 3, 2024 • edited

Moved: ../../../../../var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_amd64.tar.gz2494270430/apko_0.13.2_darwin_amd64/apko -> ../../../../../var/folders/3g/88131l9j11x995ppjbxsvhbh0000gn/T/apko_0.13.2_darwin_arm64.tar.gz2377260856/apko_0.13.2_darwin_arm64/apko (similarity: 0.95)

Original Path (From): /Users/egibs/Downloads/apko_tar_gzs/apko_0.13.2_darwin_amd64.tar.gz > apko

Original Path (To): /Users/egibs/Downloads/apko_tar_gzs_2/apko_0.13.2_darwin_arm64.tar.gz > apko

3 new behaviors

tstromberg May 3, 2024

Choose a reason for hiding this comment

egibs May 5, 2024

Choose a reason for hiding this comment

tstromberg May 3, 2024

Choose a reason for hiding this comment

egibs May 5, 2024

Choose a reason for hiding this comment

tstromberg May 7, 2024

Choose a reason for hiding this comment

tstromberg May 7, 2024

Choose a reason for hiding this comment

egibs commented May 8, 2024

egibs commented May 3, 2024 •

edited

egibs commented May 3, 2024 •

edited

egibs commented May 3, 2024 •

edited