Packages not being found in 1.0.1

[devinrsmith]

We've got a workflow that involves creating an SBOM from a tarball. It was working as of 0.105.1; on 1.0.1, there are no packages found. I'm not sure if the specifics of this tarball matter; but it can be found at https://github.com/deephaven/deephaven-core/releases/download/v0.33.2/server-jetty-0.33.2.tar

$ syft version
Application: syft
Version:    0.105.1
BuildDate:  2024-02-26T16:52:59Z
GitCommit:  928511ea0f1449e057e8057e38743d258b22476b
GitDescription: v0.105.1
Platform:   linux/amd64
GoVersion:  go1.21.7
Compiler:   gc

$ syft server-jetty-0.33.2.tar -o json > server-jetty-0.33.2.tar.syft.json             
 ✔ Indexed file system                                                                                                                                             /tmp/syft-archive-contents-3263322906
 ✔ Cataloged contents                                                                                                                   69a7e22bf88f653f45afa68c064d124ea8bea34d2d4675870851d2f812ace651
   ├── ✔ Packages                        [326 packages]  
   └── ✔ Executables                     [0 executables]

$ syft version
Application: syft
Version:    1.0.1
BuildDate:  2024-03-06T19:51:27Z
GitCommit:  1b121ac3f4d589060ddf1fac0bcd6871ea4731e3
GitDescription: v1.0.1
Platform:   linux/amd64
GoVersion:  go1.21.7
Compiler:   gc

$ syft server-jetty-0.33.2.tar -o json > server-jetty-0.33.2.tar.syft.json                                                                                                                 
 ✔ Indexed file system                                                                                                                                             /tmp/syft-archive-contents-2039841950  
 ✔ Cataloged contents                                                                                                                   69a7e22bf88f653f45afa68c064d124ea8bea34d2d4675870851d2f812ace651  
   ├── ✔ Packages                        [0 packages]                                                                                                                                                     
   └── ✔ Executables                     [0 executables]                                                                                                                                                  

Notice 326 packages on the former, 0 packages on the latter. Happy to provide more details if it would help.

[kzantow]

Hi @devinrsmith ,

I gave this a try, and it seems to be working properly for me:

$ syft version
Application: syft
Version:    1.0.1
BuildDate:  2024-03-06T19:25:56Z
GitCommit:  Homebrew
GitDescription: [not provided]
Platform:   darwin/amd64
GoVersion:  go1.22.1
Compiler:   gc

$ syft server-jetty-0.33.2.tar -o json
 ✔ Indexed file system                                                /private/var/folders/jc/fxkytj5j5sj8rwxsv7k5x1b80000gn/T/syft-archive-contents-1938077057
 ✔ Cataloged contents                                                                          69a7e22bf88f653f45afa68c064d124ea8bea34d2d4675870851d2f812ace651
   ├── ✔ Packages                        [326 packages]  
   └── ✔ Executables                     [0 executables]  
...

Do you have a syft configuration file that's overriding some default settings? Are you able to send the output of syft -vv server-jetty-0.33.2.tar?

[devinrsmith]

I don't think I have any custom configuration; I've managed my installs by installing the am64 rpm releases on Fedora.
syft-debug.txt attached.

$ dnf install syft_1.0.1_linux_amd64.rpm
...

$ dnf info syft
Name         : syft
Version      : 1.0.1
Release      : 1
Architecture : x86_64
Size         : 40 M
Source       : syft-1.0.1-1.src.rpm
Repository   : @System
From repo    : @commandline
Summary      : A tool that generates a Software Bill Of Materials (SBOM) from container images and filesystems
URL          : https://github.com/anchore/syft
License      : Apache 2.0
Description  : A tool that generates a Software Bill Of Materials (SBOM) from container images and filesystems

[kzantow]

@devinrsmith something is wrong trying to download the attachment (404), but I gave this a shot with a fedora:latest container installing with dnf and this seems to also work fine:

% docker run --rm -it fedora:latest

[root@47cbb8ddf179 /]# cat /etc/os-release 
NAME="Fedora Linux"
VERSION="39 (Container Image)"
ID=fedora
...

[root@47cbb8ddf179 /]# curl -LO https://github.com/anchore/syft/releases/download/v1.0.1/syft_1.0.1_linux_amd64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 15.6M  100 15.6M    0     0  17.2M      0 --:--:-- --:--:-- --:--:-- 17.2M

[root@47cbb8ddf179 /]# dnf install syft_1.0.1_linux_amd64.rpm 
Last metadata expiration check: 0:00:26 ago on Tue Mar 12 22:05:42 2024.
Dependencies resolved.
=================================================================================================================================================================
 Package                            Architecture                         Version                                Repository                                  Size
=================================================================================================================================================================
Installing:
 syft                               x86_64                               1.0.1-1                                @commandline                                16 M

Transaction Summary
=================================================================================================================================================================
Install  1 Package

Total size: 16 M
Installed size: 40 M
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                         1/1 
  Installing       : syft-1.0.1-1.x86_64                                                                                                                     1/1 
  Verifying        : syft-1.0.1-1.x86_64                                                                                                                     1/1 

Installed:
  syft-1.0.1-1.x86_64                                                                                                                                            

Complete!

[root@47cbb8ddf179 /]# curl -LO https://github.com/deephaven/deephaven-core/releases/download/v0.33.2/server-jetty-0.33.2.tar
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  262M  100  262M    0     0  59.8M      0  0:00:04  0:00:04 --:--:-- 64.6M

[root@47cbb8ddf179 /]# syft version
Application: syft
Version:    1.0.1
BuildDate:  2024-03-06T19:51:27Z
GitCommit:  1b121ac3f4d589060ddf1fac0bcd6871ea4731e3
GitDescription: v1.0.1
Platform:   linux/amd64
GoVersion:  go1.21.7
Compiler:   gc

[root@47cbb8ddf179 /]# syft server-jetty-0.33.2.tar -o json > server-jetty-0.33.2.tar.syft.json  
 ✔ Indexed file system                                                                                                    /tmp/syft-archive-contents-3010994469
 ✔ Cataloged contents                                                                          69a7e22bf88f653f45afa68c064d124ea8bea34d2d4675870851d2f812ace651
   ├── ✔ Packages                        [326 packages]  
   └── ✔ Executables                     [0 executables]  

Are you able to run this on a clean fedora container? If you re-run this a few times, does it happen to eventually succeed?

[devinrsmith]

I guess the upload didn't work for some reason. I'll paste it here:

[0000]  INFO syft version: 1.0.1
[0000] DEBUG config:
�[35m  log:
      quiet: false
      level: debug
      file: ""
  dev:
      profile: none
  config: ""
  output:
      - syft-table
  format:
      pretty: null
      template:
          path: ""
          legacy: false
      json:
          legacy: false
          pretty: false
      spdx-json:
          pretty: false
      cyclonedx-json:
          pretty: false
      cyclonedx-xml:
          pretty: false
  check-for-app-update: true
  default-catalogers: []
  select-catalogers: []
  package:
      search-unindexed-archives: false
      search-indexed-archives: true
      exclude-binary-overlap-by-ownership: true
  file:
      metadata:
          selection: owned-by-package
          digests:
              - sha1
              - sha256
      content:
          skip-files-above-size: 256000
          globs: []
      executable:
          globs: []
  scope: squashed
  parallelism: 1
  relationships:
      package-file-ownership: true
      package-file-ownership-overlap: true
  golang:
      search-local-mod-cache-licenses: false
      local-mod-cache-dir: /home/devin/go/pkg/mod
      search-remote-licenses: false
      proxy: https://proxy.golang.org,direct
      no-proxy: ""
      main-module-version:
          from-ld-flags: true
          from-contents: true
          from-build-settings: true
  java:
      use-network: false
      maven-url: ""
      max-parent-recursive-depth: 0
  javascript:
      search-remote-licenses: false
      npm-base-url: ""
  linux-kernel:
      catalog-modules: true
  python:
      guess-unpinned-requirements: false
  registry:
      insecure-skip-tls-verify: false
      insecure-use-http: false
      auth: []
      ca-cert: ""
  from: []
  platform: ""
  source:
      name: ""
      version: ""
      base-path: ""
      file:
          digests:
              - SHA-256
      image:
          default-pull-source: ""
  exclude: []�[0m
[0000] DEBUG checking if a new version of syft is available
[0000] DEBUG no new syft update available
[0000] DEBUG source path is an archive
[0000] DEBUG ignoring system mountpoint mountpoint=/dev
[0000] DEBUG ignoring system mountpoint mountpoint=/dev/shm
[0000] DEBUG ignoring system mountpoint mountpoint=/sys
[0000] DEBUG ignoring system mountpoint mountpoint=/proc
[0000] DEBUG ignoring system mountpoint mountpoint=/run
[0000] DEBUG ignoring system mountpoint mountpoint=/tmp
[0000] DEBUG ignoring system mountpoint mountpoint=/run/user/1000
[0000] DEBUG discovered 0 packages cataloger=alpm-db-cataloger
[0000] DEBUG discovered 0 packages cataloger=apk-db-cataloger
[0000] DEBUG discovered 0 packages cataloger=dpkg-db-cataloger
[0000] DEBUG discovered 0 packages cataloger=portage-cataloger
[0000] DEBUG discovered 0 packages cataloger=rpm-db-cataloger
[0000] DEBUG discovered 0 packages cataloger=rpm-archive-cataloger
[0000] DEBUG discovered 0 packages cataloger=conan-cataloger
[0000] DEBUG discovered 0 packages cataloger=dart-pubspec-lock-cataloger
[0000] DEBUG discovered 0 packages cataloger=dotnet-deps-cataloger
[0000] DEBUG discovered 0 packages cataloger=elixir-mix-lock-cataloger
[0000] DEBUG discovered 0 packages cataloger=erlang-rebar-lock-cataloger
[0000] DEBUG discovered 0 packages cataloger=erlang-otp-application-cataloger
[0000] DEBUG discovered 0 packages cataloger=haskell-cataloger
[0000] DEBUG discovered 0 packages cataloger=go-module-file-cataloger
[0000] DEBUG discovered 0 packages cataloger=java-gradle-lockfile-cataloger
[0000] DEBUG discovered 0 packages cataloger=java-pom-cataloger
[0000] DEBUG discovered 0 packages cataloger=javascript-lock-cataloger
[0000] DEBUG discovered 0 packages cataloger=php-composer-lock-cataloger
[0000] DEBUG discovered 0 packages cataloger=python-package-cataloger
[0000] DEBUG discovered 0 packages cataloger=ruby-gemfile-cataloger
[0000] DEBUG discovered 0 packages cataloger=ruby-gemspec-cataloger
[0000] DEBUG discovered 0 packages cataloger=rust-cargo-lock-cataloger
[0000] DEBUG discovered 0 packages cataloger=cocoapods-cataloger
[0000] DEBUG discovered 0 packages cataloger=swift-package-manager-cataloger
[0000] DEBUG discovered 0 packages cataloger=dotnet-portable-executable-cataloger
[0000] DEBUG discovered 0 packages cataloger=python-installed-package-cataloger
[0000] DEBUG discovered 0 packages cataloger=go-module-binary-cataloger
[0000] DEBUG discovered 0 packages cataloger=java-archive-cataloger
[0000] DEBUG discovered 0 packages cataloger=graalvm-native-image-cataloger
[0000] DEBUG discovered 0 packages cataloger=nix-store-cataloger
[0000] DEBUG discovered 0 packages cataloger=binary-classifier-cataloger
[0000] DEBUG discovered 0 packages cataloger=github-actions-usage-cataloger
[0000] DEBUG discovered 0 packages cataloger=github-action-workflow-usage-cataloger
[0000] DEBUG discovered 0 packages cataloger=linux-kernel-cataloger
[0000] DEBUG discovered 0 packages cataloger=wordpress-plugins-cataloger
[0000] DEBUG executable cataloger processed 0 files

I'll try in a container soon.

[kzantow]

Developer notes: if someone is able to pick this up to try to reproduce it, a possibility why this isn't working for Devin could be due to handling of /dev folders, note the home directory of /home/devin/.

[tgerla]

Hey @devinrsmith, have you had a chance to try to reproduce this in your container yet? We have so far been unable to reproduce the problem here but we would be happy to keep looking. Thanks!

[devinrsmith]

I'm unable to reproduce the issue using a fedora container; it seems to work. There must be something about my main workstation that is different. Using 1.1.1 now from my workstation I get the same issue though. With -vvv:

[0000] TRACE starting package cataloger name=java-archive-cataloger
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-1286526638/etc/os-release
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-1286526638/usr/lib/os-release
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-1286526638/etc/system-release-cpe
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-1286526638/etc/redhat-release
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-1286526638/bin/busybox
[0000] TRACE searching for paths matching glob glob=**/*.jar
[0000] TRACE searching filetree by glob glob=**/*.jar
[0000] TRACE searching for paths matching glob glob=**/*.war
[0000] TRACE searching filetree by glob glob=**/*.war
[0000] TRACE searching for paths matching glob glob=**/*.ear
[0000] TRACE searching filetree by glob glob=**/*.ear
[0000] TRACE searching for paths matching glob glob=**/*.par
[0000] TRACE searching filetree by glob glob=**/*.par
[0000] TRACE searching for paths matching glob glob=**/*.sar
[0000] TRACE searching filetree by glob glob=**/*.sar
[0000] TRACE searching for paths matching glob glob=**/*.nar
[0000] TRACE searching filetree by glob glob=**/*.nar
[0000] TRACE searching for paths matching glob glob=**/*.jpi
[0000] TRACE searching filetree by glob glob=**/*.jpi
[0000] TRACE searching for paths matching glob glob=**/*.hpi
[0000] TRACE searching filetree by glob glob=**/*.hpi
[0000] TRACE searching for paths matching glob glob=**/*.lpkg
[0000] TRACE searching filetree by glob glob=**/*.lpkg
[0000] TRACE searching for paths matching glob glob=**/*.zip
[0000] TRACE searching filetree by glob glob=**/*.zip
[0000] DEBUG discovered 0 packages cataloger=java-archive-cataloger
[0000] TRACE package cataloger completed name=java-archive-cataloger

from the container:

[0000] TRACE starting package cataloger name=java-archive-cataloger
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-3704229090/etc/os-release
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-3704229090/usr/lib/os-release
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-3704229090/etc/system-release-cpe
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-3704229090/etc/redhat-release
[0000] TRACE searching filetree by path path=/tmp/syft-archive-contents-3704229090/bin/busybox
[0000] TRACE searching for paths matching glob glob=**/*.jar
[0000] TRACE searching filetree by glob glob=**/*.jar
[0000] TRACE searching for paths matching glob glob=**/*.war
[0000] TRACE searching filetree by glob glob=**/*.war
[0000] TRACE searching for paths matching glob glob=**/*.ear
[0000] TRACE searching filetree by glob glob=**/*.ear
[0000] TRACE searching for paths matching glob glob=**/*.par
[0000] TRACE searching filetree by glob glob=**/*.par
[0000] TRACE searching for paths matching glob glob=**/*.sar
[0000] TRACE searching filetree by glob glob=**/*.sar
[0000] TRACE searching for paths matching glob glob=**/*.nar
[0000] TRACE searching filetree by glob glob=**/*.nar
[0000] TRACE searching for paths matching glob glob=**/*.jpi
[0000] TRACE searching filetree by glob glob=**/*.jpi
[0000] TRACE searching for paths matching glob glob=**/*.hpi
[0000] TRACE searching filetree by glob glob=**/*.hpi
[0000] TRACE searching for paths matching glob glob=**/*.lpkg
[0000] TRACE searching filetree by glob glob=**/*.lpkg
[0000] TRACE searching for paths matching glob glob=**/*.zip
[0000] TRACE searching filetree by glob glob=**/*.zip
[0000] TRACE parsing file contents path=server-jetty-0.33.2/lib/HdrHistogram-2.1.12.jar
[0000] TRACE unable to parse license expression: %winvalid SPDX expression: http://creativecommons.org/publicdomain/zero/1.0/, https://opensource.org/licenses/BSD-2-Clause
[0000] TRACE unable to parse license expression: %winvalid SPDX expression: Public Domain, per Creative Commons CC0
[0000] TRACE parsing pom.xml artifactID=HdrHistogram name=HdrHistogram path=META-INF/maven/org.hdrhistogram/HdrHistogram/pom.xml projectURL=http://hdrhistogram.github.io/HdrHistogram/
[0000] TRACE resolving property existingPropertyValue=org.hdrhistogram propertyName=groupId
[0000] TRACE resolving property existingPropertyValue=2.1.12 propertyName=version
[0000] TRACE unable to parse license expression: %winvalid SPDX expression: Public Domain, per Creative Commons CC0
[0000] TRACE parsing pom.xml artifactID=HdrHistogram name=HdrHistogram path=META-INF/maven/org.hdrhistogram/HdrHistogram/pom.xml projectURL=http://hdrhistogram.github.io/HdrHistogram/
[0000] TRACE resolving property existingPropertyValue=org.hdrhistogram propertyName=groupId
[0000] TRACE resolving property existingPropertyValue=2.1.12 propertyName=version
...
[0005] DEBUG discovered 326 packages cataloger=java-archive-cataloger
[0005] TRACE package cataloger completed name=java-archive-cataloger
[0005] TRACE starting package cataloger name=graalvm-native-image-cataloger

I'm not sure how to continue debugging the issues; is there some sort of tar-level logging I could enable?

[tgerla]

Thanks @devinrsmith, this is helpful. On a hunch, can you try moving the tar file to /tmp and scanning it from there, and/or creating a new system user (without "dev" in the username) and scanning from there?

(Please also double check your tar file to make sure it is a real tar file -- during testing we thought we had reproduced the problem but we were scanning something that wasn't a real tar file because we had not followed redirects with curl.)

Thank you!