Issue building 9.14.1 with 7zip or p7zip in Linux (Ubuntu/Debian)

[xthursdayx]

Hey guys, I'm currently updating my YACReaderLibraryServer Docker image to include QT6 and package the recent YACReader 9.14.1 release. So far, the unarr version is working well with QT6 and 9.14.1, however I'm running into some issues when I try to build the version packaged with 7zip or p7zip.

I'm finding that when I build my QT6 image with YACReader 9.14.1 compiled with either the newer 7zip libraries hosted in your 7z deps repo or the previously-used p7zip libraries as /src/git/compressed_archive/lib7zip or /src/git/compressed_archive/libp7zip, respectively, I get the following error:
ERROR: You'll need 7zip source code to compile YACReader. Please check the compressed_archive folder for further instructions.

Obviously, I am including 7zip source, so I'm not sure what is triggering this error. Any ideas? Compiling YACReader 9.13.1 with p7zip has always worked without issue in my previous QT5 YACReaderLibraryServer image.

Here are the relevant sections of my Dockerfile, for 7zip:

# clone yacreaderlibraryserver
    git clone -b master --single-branch https://github.com/YACReader/yacreader.git /src/git && \
    cd /src/git && \
    git checkout $YACR_VERSION && \
# install 7zip libraries
    cd /src/git/compressed_archive && \
    wget "https://github.com/YACReader/yacreader-7z-deps/blob/main/7z2301-src.7z?raw=true" -O /src/git/compressed_archive/7z2301-src.7z && \
    7z x /src/git/compressed_archive/7z2301-src.7z -o/src/git/compressed_archive/lib7zip && \
# build yacreaderlibraryserver
    cd /src/git/YACReaderLibraryServer && \
    qtchooser -list-versions && \
    qmake6 PREFIX=/app CONFIG+="7zip no_pdf server_standalone" YACReaderLibraryServer.pro && \
    qmake6 -v && \
    make  && \
    make install && \

and for p7zip:

# clone yacreaderlibraryserver
    git clone -b master --single-branch https://github.com/YACReader/yacreader.git /src/git && \
    cd /src/git && \
    git checkout $YACR_VERSION && \
# install p7zip libraries
    cd /src/git/compressed_archive && \
    wget "https://sourceforge.net/projects/p7zip/files/p7zip/16.02/p7zip_16.02_src_all.tar.bz2" && \
    tar xjf /src/git/compressed_archive/p7zip_16.02_src_all.tar.bz2 -C /src/git/compressed_archive && \
    mv /src/git/compressed_archive/p7zip_16.02 /src/git/compressed_archive/libp7zip && \
# build yacreaderlibraryserver
    cd /src/git/YACReaderLibraryServer && \
    qmake PREFIX=/app "CONFIG+=7zip server_standalone" YACReaderLibraryServer.pro && \
    make  && \
    make install && \

[luisangelsm]

Hi,

I sent you an email (mailbox.org account) when I migrated YACReader to use 7zip 23.01 to see if it would cause you any troubles, p7zip is abandoned and now 7zip supports linux so this is the right path to keep things up to date. p7zip is no longer supported.

In compressed_archive/wrapper.pri there is this check:

exists (../compressed_archive/lib7zip) {
    message(Using 7zip)
} else {
    error(You\'ll need 7zip source code to compile YACReader. \
          Please check the compressed_archive folder for further instructions.)
}

So it seems that the check is failing for some reason, can you double check that lib7zip is there?

You are going to need the right 7z.so binaries, there are some instructions to build 7z.so with rar support here if you can't find prebuilt ones: https://github.com/YACReader/yacreader/blob/develop/dependencies/7zip/linux/Steps%20for%20building%207z.so%20with%20rar%20support.txt

Thanks for keeping the docker image updated, it is really appreciated, maybe it would be a good idea if you join the discord server so this type of changes don't take you by surprise.

[xthursdayx]

Hi Luis,

Apologies, I must have missed your email... I'm up for joining the discord server, if that's the best place to join the discussion.

In the mean time, I'll check into this and report back once I figure out what's going on. Thanks for pointing me in the right direction. Cheers!

[luisangelsm]

I run some tests in Ubuntu and I was able to compile from develop with 7zip support just fine. Note that there is a job in azure pipelines that builds in Ubuntu + qt6 + 7zip and it has been passing CI/CD (in case you find any of the setup done there useful). Let me know if you need help with anything.

[xthursdayx]

Yeah, I based my updated qt6 + 7zip image (built on Ubuntu 22.04) on the job in your azure pipelines. There may be something else in baseimage that is causing the issue. I've come across a bug with qtchooser + qt6 + Ubuntu 22.04 in an arm64 environment, so that may be part of the issue. I'm still troubleshooting, but will report back soon.

[xthursdayx]

Okay, I spun up a docker container of my baseimage and followed my Dockerfile up the the stage of building yacreaderlibraryserver. Everything worked fine again up until the qmake stage when once again the compiler couldn't find the lib7zip directory for some reason.

This my terminal output from the point of downloading and installing the 7zip libraries up to the error:

root@6b3bfb12bbab:/src/7zip/7z# cd /src/git/compressed_archive
root@6b3bfb12bbab:/src/git/compressed_archive# wget "https://github.com/YACReader/yacreader-7z-deps/blob/main/7z2301-src.7z?raw=true" -O /src/git/compressed_archive/7z2301-src.7z
--2024-02-09 11:11:18--  https://github.com/YACReader/yacreader-7z-deps/blob/main/7z2301-src.7z?raw=true
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/YACReader/yacreader-7z-deps/raw/main/7z2301-src.7z [following]
--2024-02-09 11:11:18--  https://github.com/YACReader/yacreader-7z-deps/raw/main/7z2301-src.7z
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/YACReader/yacreader-7z-deps/main/7z2301-src.7z [following]
--2024-02-09 11:11:19--  https://raw.githubusercontent.com/YACReader/yacreader-7z-deps/main/7z2301-src.7z
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.110.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1365807 (1.3M) [application/octet-stream]
Saving to: ‘/src/git/compressed_archive/7z2301-src.7z’

/src/git/compressed_archive/7z2301-src.7z     100%[===============================================================================================>]   1.30M  --.-KB/s    in 0.05s

2024-02-09 11:11:19 (28.2 MB/s) - ‘/src/git/compressed_archive/7z2301-src.7z’ saved [1365807/1365807]

root@6b3bfb12bbab:/src/git/compressed_archive# 7z x /src/git/compressed_archive/7z2301-src.7z -o/src/git/compressed_archive/lib7zip

7-Zip (z) 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
 64-bit locale=en_US.UTF-8 Threads:24 OPEN_MAX:40960, ASM

Scanning the drive for archives:
1 file, 1365807 bytes (1334 KiB)

Extracting archive: /src/git/compressed_archive/7z2301-src.7z
--
Path = /src/git/compressed_archive/7z2301-src.7z
Type = 7z
Physical Size = 1365807
Headers Size = 16806
Method = LZMA:9m
Solid = +
Blocks = 1

Everything is Ok

Files: 1256
Size:       9123132
Compressed: 1365807
root@6b3bfb12bbab:/src/git/compressed_archive# cd /src/git/YACReaderLibraryServer
root@6b3bfb12bbab:/src/git/YACReaderLibraryServer# qtchooser -install qt6 $(which qmake6)
root@6b3bfb12bbab:/src/git/YACReaderLibraryServer# export QT_SELECT=qt
root@6b3bfb12bbab:/src/git/YACReaderLibraryServer# qtchooser -list-versions
4
5
qt4-x86_64-linux-gnu
qt4
qt5-x86_64-linux-gnu
qt5
qt6
root@6b3bfb12bbab:/src/git/YACReaderLibraryServer# qmake6 PREFIX=/app CONFIG+="7zip no_pdf server_standalone" YACReaderLibraryServer.pro
Info: creating stash file /src/git/YACReaderLibraryServer/.qmake.stash
Project ERROR: You'll need 7zip source code to compile YACReader. Please check the compressed_archive folder for further instructions.

And here is the context, ownership and permissions of /src/git/compressed_archive inside of my running container, which shows that the lib7zip directory is present and accessible:

root@6b3bfb12bbab:/src/git/compressed_archive# ls -l
total 1420
-rw-r--r-- 1 root root 1365807 Feb  9 11:11 7z2301-src.7z
-rw-r--r-- 1 root root    1839 Feb  9 11:04 7z_includes.h
-rw-r--r-- 1 root root     178 Feb  9 11:04 7z_precompiled_header.h
-rw-r--r-- 1 root root   21176 Feb  9 11:04 compressed_archive.cpp
-rw-r--r-- 1 root root    1695 Feb  9 11:04 compressed_archive.h
-rw-r--r-- 1 root root    9728 Feb  9 11:04 extract_callbacks.h
-rw-r--r-- 1 root root     304 Feb  9 11:04 extract_delegate.h
drwxr-xr-x 1 root root      20 Feb  9 11:11 lib7zip
drwxr-xr-x 1 root root     182 Feb  9 11:04 libarchive
-rw-r--r-- 1 root root    8416 Feb  9 11:04 libp7zip.patch
-rw-r--r-- 1 root root    1381 Feb  9 11:04 open_callbacks.h
-rw-r--r-- 1 root root     407 Feb  9 11:04 README_7zip.txt
-rw-r--r-- 1 root root     100 Feb  9 11:04 StdAfx.h
-rw-r--r-- 1 root root     476 Feb  9 11:04 StdAfx.h.cpp
drwxr-xr-x 1 root root     192 Feb  9 11:04 unarr
-rw-r--r-- 1 root root    3041 Feb  9 11:04 wrapper.pri

For now, I'm going to try building the 7z.so binaries with rar support within the container, rather than using the prebuilt ones, so hopefully that makes a difference. Any additional suggestions for troubleshooting from here, or do you think it's possibly a bug in the compressed_archive/wrapper.pri check?

[luisangelsm]

The prebuilt 7z.so ones are there more as a proof of concept, ideally they should be built for the exact platform/system they are going to run in. But sadly at that point in the build system the binaries aren't checked at all, so it won't make a difference.

The check has been there years, the only change was that before lib7zip was used in windows and libp7zip in unix systems to differentiate between 7zip and p7zip.

I was wondering if the folder you start the build from could had something to do with the error but...

~/yacreader/YACReaderLibraryServer# qmake6 PREFIX=/app CONFIG+="7zip no_pdf server_standalone" YACReaderLibraryServer.pro
Project MESSAGE: Using 7zip

So it is really weird that lib7zip is not found in your case. Just having the folder there (even with nothing inside) should be enough to pass that check, obviously you'll get later other errors.

What happens if you remove the check? does it compile?

[xthursdayx]

I've made some changes to the Docker image and I'm now able to get the QT6 version to build with both 7zip and unarr, so I'm about ready to release it. However, I've been trying to build 7z.so for these specific images and am running into another issue (I can open a new issue is you would prefer that).

As per the directions here I started by downloading the the 7zip source. For my image, based on Ubuntu 22.04 (the current LTS), the latest version of 7zip available is 21.07, not 23.01, which appears to be an issue, since I believe YACReader only works with 7zip 23.01, correct? After removing the DISABLE_RAR* line in 7zip-21.07+dfsg/debian/rules (there is a type in the instructions here btw) and running dpkg-buildpackage -us -uc -b from within 7zip-21.07+dfsg, I get the following error, which I assume is because I'm building 21.07, not 23.01:

make[2]: *** No rule to make target '../../Compress/Rar1Decoder.cpp', needed by 'b/g/Rar1Decoder.o'.  Stop.
make[2]: Leaving directory '/7zip-21.07+dfsg/CPP/7zip/Bundles/Alone2'
make[1]: *** [debian/rules:13: override_dh_auto_build] Error 2
make[1]: Leaving directory '/7zip-21.07+dfsg'
make: *** [debian/rules:10: binary] Error 2

I'm currently attempting to figure out if it's possible to build 7zip 23.01 for Ubuntu 22.04. If not, the only alternative may be be for me to either wait to see if a 7zip 23.01 package is released for Ubuntu 22.04 or to change my Docker baseimage, which will necessitate completely reworking the image.

[luisangelsm]

No, I think this issue is ok.

Hmmm, Igor likes to change things around from version to version, so compilation + integration is always tricky. I didn't try to compile 21.07, the binaries should be compatible though.

About that error, it sounds like Rar1Decoder.cpp is not found, check if it is in the code and if so, maybe some of the lines with DISABLE_RAR remain ?? Not sure if they have the rar code in a separate package or something...but at least for 23.01 I didn't have to do anything.

But yeah, if building 23.01 for ubuntu 22.04 is possible I would do that.

[selmf]

It is possible to ship a different version of 7zip with YACReader than what is provided by the system. Just place 7z.so and the codecs in /usr/lib/yacreader

[luisangelsm]

actually it may be a problem with the source package: https://bugs.launchpad.net/ubuntu/+source/7zip/+bug/2051615

I think I have around the code I got originally, let me check it.

[luisangelsm]

Yep, the package I got originally had the rar code in it.

~/7zip-deb-src-untouched# find . | grep Rar1Decoder.cpp
./7zip-23.01+dfsg/CPP/7zip/Compress/Rar1Decoder.cpp

But if a got the code again (same setup and system) it doesn't have it.

I guess they realized that it could potentially build the 7z.so with rar suport in it, without the need of external codecs....

[xthursdayx]

I've been able to build 7zip 23.01 for Ubuntu 22.04 and compiled YACReader with it, seemingly without any issues. I'm continuing to test at the moment, but if everything continues to work I'll push my updated docker image later today.

[xthursdayx]

Crap... Well, my updated docker images seem to work fine and both the amd64 and aarch64 versions compile without issue on those respective systems. However, there is apparently a bug in QT6 (since 6.2) that causes qmake to fail when being run as part of a cross-environment docker build process (in this instance my GitHub docker image build workflow emulating aarch64 from their x86-64 host) using docker buildx with qemu. See these bug reports: 1, 2, 3.

This means that for now, I'm either going to have to drop support for aarch64 or roll the aarch64 versions back to QT5 (there are not QT6 version before 6.2 that I can pin the packages to), since I don't have the time to figure out how to completely change my build workflow (and aarch64 runners/VMs don't exist for GitHub actions yet).

Any thoughts?

[luisangelsm]

I guess users will prefer a Qt5 image rather than no image at all...

[xthursdayx]

Yeah, I went ahead and pushed a release with the aarch64 version rolled back to Qt5. It seems to still be working with 7zip 23.01, so we'll see if I get any bug reports. Cheers!

[xthursdayx]

Unfortunately it seems that via iOS (from iPads), 7zip is not working and comics are failing to open. The unarr version of the docker image is working without issue, but comics fail to open with the 7zip version. I'm not sure exactly how to go about troubleshooting what is going on at the moment, and don't have a lot of time to do so at the moment, but if you have any suggestions please let me know.

[luisangelsm]

Did you check the type of files that fail? zip? rar? rar5? all of them?

[xthursdayx]

It's not actually opening the files that fail, it's the database creation itself I think. Rather than trying to add my existing YACReader comic library to the YACReaderLibraryServer running in my 7zip container, I created a new container mounted with a small comic test library at /comics. When I run the create-library it appears to work, creating the .yacreaderlibrary directory in /comics, however, when I try to browse the library by going to <docker_IP>:8080 and clicking on the library name it does not open. When I run the update-library from within the container I get the response Processing comics and Segmentation fault. My docker log isn't really any help in that it just shows something like: WARN 2024-03-07T00:23:00.678 QSqlQuery::prepare: database not open when I try to browse the libraries via the IP or update the library.

[luisangelsm]

Could it be a missing Qt6 package?

[luisangelsm]

Here there is a user saying that the Qt6 build has some .so missing: https://www.yacreader.com/forum/support-linux/2067-how-2-create-a-yacreader-server-on-synology-nas?start=30#8243

Not sure if it is related to this though.

[luisangelsm]

@xthursdayx did you try to use libarchive? #290

That could be an alternative to 7zip, it supports rar5 and that would be good enough for an image, or at least better that only having support for rar4. Sadly rar5 is everywhere these days.