Failing to fetch dependencies when cross compiling focal #71

amartinz · 2022-07-07T17:01:10Z

I have tried to build repowerd and lomiri-system-settings using crossbuilder.
However it fails everytime when fetching dependencies:

...

Reading package lists... Done
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.36 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.39 80]

My current workaround is to manually enter the created lxc container and edit /etc/apt/sources.list, replacing all instances of deb with deb [arch=amd64,i386].

The text was updated successfully, but these errors were encountered:

JamiKettunen · 2022-09-01T11:57:03Z

Having the same problem trying to crossbuild lomiri-system-settings from amd64 to arm64 on my Void Linux host: https://termbin.com/zm8f
@amartinz What was/is your host OS where you saw this? I doubt that matters but @gber wasn't able to reproduce the issue

amartinz · 2022-09-01T12:01:56Z

I am using Fedora Rawhide (branched f37).

gber · 2022-09-01T12:03:08Z

Can any of you please provide some additional information:

What revision of crossbuilder are you using?
What OS are you running?
@JamiKettunen Do you use a wrapper around crossbuilder? Please provide the contents, particularly the cli options used.
Please enter the container via crossbuilder --arch=arm64 shell and post the output of head -10000 /etc/apt/sources.list /etc/apt/sources.list.d/*.list

gber · 2022-09-01T12:06:08Z

I can ideed build both repowerd as well as lomiri-system-settings (with https://gitlab.com/ubports/development/core/lomiri-system-settings/-/merge_requests/346 applied) just fine on Ubuntu 22.04 by running the crossbuilder@9f452fe using crossbuilder --lxd-image=ubuntu:20.04 --architecture=arm64 build.

JamiKettunen · 2022-09-01T13:08:29Z

@gber

Can any of you please provide some additional information:
* What revision of crossbuilder are you using?

https://github.com/JamiKettunen/crossbuilder/tree/void-linux @ 06470ec

* What OS are you running?

Void Linux

* @JamiKettunen Do you use a wrapper around crossbuilder? Please provide the contents, particularly the cli options used.

No wrapper, storage setup without ZFS.

* Please enter the container via `crossbuilder --arch=arm64 shell` and post the output of `head -10000 /etc/apt/sources.list /etc/apt/sources.list.d/*.list`

That gave Error: unknown option: --arch, but crossbuilder --architecture=arm64 shell worked: https://termbin.com/kt3z

gber · 2022-09-01T13:35:21Z

OK, now the problem is clear, from your /etc/apt/sources.list:

## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
## if you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
##     or do the same in user-data
## b.) add sources in /etc/apt/sources.list.d
## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu focal main restricted
# deb-src http://archive.ubuntu.com/ubuntu focal main restricted

…

Note the comment on top, the file is create by cloud-init which is why it's missing [arch=amd64], now the question is if you're using a different container than me. Either it is overwritten by cloud-init or crossbuilder thinks you're running a sdk image and doesn't execute nonsdk_container_setup which fixes up /etc/apt/sources.list.

gber · 2022-09-01T13:39:43Z

According to the logs you're using ubuntu:20.04 image what's the remote of that, i.e. lxc remote ls?

JamiKettunen · 2022-09-01T20:03:49Z

According to the logs you're using ubuntu:20.04 image what's the remote of that, i.e. lxc remote ls?

https://cloud-images.ubuntu.com/releases, relevant script for focal here: https://github.com/ubports/crossbuilder/blob/9f452fe/crossbuilder#L1158-L1166

$ lxc remote ls
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubports-sdk     | https://sdk-images.ubports.com           | simplestreams | none        | YES    | NO     | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+--------+

gber · 2022-09-05T06:48:51Z

According to the logs you're using ubuntu:20.04 image what's the remote of that, i.e. lxc remote ls?

https://cloud-images.ubuntu.com/releases, relevant script for focal here: https://github.com/ubports/crossbuilder/blob/9f452fe/crossbuilder#L1158-L1166

Yep, but the name is configurable hence my question. But as I see you're using the default

| ubuntu | https://cloud-images.ubuntu.com/releases | simplestreams | none | YES | YES | NO |

so something else (configuration?) must explain why cloud-init overwrites the /etc/apt/sources.list prepared by crossbuilder.

JamiKettunen · 2022-09-05T17:08:46Z

so something else (configuration?) must explain why cloud-init overwrites the /etc/apt/sources.list prepared by crossbuilder.

@gber exec_container_root apt update (in create_container()) runs the cloud-init stuff overwriting the changes made earlier, so it looks like nonsdk_container_setup() call needs to be moved after that.

gber · 2022-09-06T12:25:48Z

@gber exec_container_root apt update (in create_container()) runs the cloud-init stuff overwriting the changes made earlier, so it looks like nonsdk_container_setup() call needs to be moved after that.

cloud-init runs on boot when the container is first started, apt update then only downloads the packaging metadata from the configured repositories, it needs to be run after nonsdk_container_setup() because of the changes made there.

To me it rather looks like nonsdk_container_setup() itself or the sed commands changing sources.list are not being run, can you check with set -x what's going on there?

JamiKettunen · 2023-01-05T20:19:18Z

@amartinz What was the fix? I can test this soon again but I don't believe this got fixed?

maciek134 · 2023-02-25T15:21:17Z

It didn't,

E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/main/binary-arm64/Packages  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.38 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

crossbuilder 59592bd
os Artix with OpenRC, fixed cgroups for running systemd in lxc
no wrapper

head -10000 /etc/apt/sources.list /etc/apt/sources.list.d/*.list

klh@lomiri-keyboard-20-04-amd64-arm64:~$ head -10000 /etc/apt/sources.list /etc/apt/sources.list.d/*.list
==> /etc/apt/sources.list <==
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
## if you wish to make changes you can:
## a.) add 'apt_preserve_sources_list: true' to /etc/cloud/cloud.cfg
##     or do the same in user-data
## b.) add sources in /etc/apt/sources.list.d
## c.) make changes to template file /etc/cloud/templates/sources.list.tmpl

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://archive.ubuntu.com/ubuntu focal main restricted
# deb-src http://archive.ubuntu.com/ubuntu focal main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://archive.ubuntu.com/ubuntu focal-updates main restricted
# deb-src http://archive.ubuntu.com/ubuntu focal-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu focal universe
# deb-src http://archive.ubuntu.com/ubuntu focal universe
deb http://archive.ubuntu.com/ubuntu focal-updates universe
# deb-src http://archive.ubuntu.com/ubuntu focal-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://archive.ubuntu.com/ubuntu focal multiverse
# deb-src http://archive.ubuntu.com/ubuntu focal multiverse
deb http://archive.ubuntu.com/ubuntu focal-updates multiverse
# deb-src http://archive.ubuntu.com/ubuntu focal-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse
# deb-src http://archive.ubuntu.com/ubuntu focal-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu focal partner
# deb-src http://archive.canonical.com/ubuntu focal partner

deb http://security.ubuntu.com/ubuntu focal-security main restricted
# deb-src http://security.ubuntu.com/ubuntu focal-security main restricted
deb http://security.ubuntu.com/ubuntu focal-security universe
# deb-src http://security.ubuntu.com/ubuntu focal-security universe
deb http://security.ubuntu.com/ubuntu focal-security multiverse
# deb-src http://security.ubuntu.com/ubuntu focal-security multiverse

==> /etc/apt/sources.list.d/ports.list <==
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal main restricted
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-updates main restricted
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal universe
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal universe
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-updates universe
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal multiverse
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal multiverse
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-updates multiverse
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-backports main restricted universe multiverse
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is offered by Canonical and the
## respective vendors as a service to Ubuntu users.
# deb http://archive.canonical.com/ubuntu focal partner
# deb-src http://archive.canonical.com/ubuntu focal partner

deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-security main restricted
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-security main restricted
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-security universe
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-security universe
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports/ focal-security multiverse
# deb-src http://ports.ubuntu.com/ubuntu-ports/ focal-security multiverse

==> /etc/apt/sources.list.d/ubports.list <==
deb http://repo2.ubports.com/ focal main

peat-psuwit · 2023-02-25T19:12:10Z

The code here is responsible for setting up sources.list for crossbuilding.

crossbuilder/crossbuilder

Lines 429 to 442 in 59592bd

    
               case "${HOST_ARCH}-${TARGET_ARCH}" in 
        
                   i386-amd64|amd64-i386) 
        
                       # Both on archive, do nothing. 
        
                       ;; 
        
                   i386-*|amd64-*) 
        
                       # Host is on archive, target is on ports. 
        
                       exec_container_root "sed -E \ 
        
                           -e 's:(archive|security)\.ubuntu\.com/ubuntu/:ports.ubuntu.com/ubuntu-ports/:' \ 
        
                           -e 's:^deb :deb [arch=${TARGET_ARCH}] :' \ 
        
                           /etc/apt/sources.list >/etc/apt/sources.list.d/ports.list" 
        
                       exec_container_root "sed -i -E \ 
        
                           -e 's:^deb :deb [arch=${HOST_ARCH}] :' \ 
        
                           /etc/apt/sources.list" 
        
                   ;;

My feeling is that HOST_ARCH is wrong?

maciek134 · 2023-02-26T11:07:55Z

Hmm, gcc -print-multiarch returns nothing, but

$ gcc -dumpmachine | sed 's/^\([^-]\+\)-\([^-]\+\)-\([^-]\+\)-\([^-]\+\)$/\1-\3-\4/'
x86_64-linux-gnu

so that looks fine since it would be mapped to amd64. I'll try to debug later, for some reason I thought all of that happens in the image, not in crossbuilder.

ElectrodeYT · 2024-02-07T13:41:00Z

I can confirm this still seems to be a problem:

E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.81 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.81 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.81 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/main/binary-arm64/Packages  404  Not Found [IP: 91.189.91.82 80]
E: Some index files failed to download. They have been ignored, or old ones used instead.

Running this script in the container seems to mitigate this problem:

#!/bin/bash
sed -i 's/deb http/deb [arch=amd64\,i386] http/g' /etc/apt/sources.list
sed -e 's/amd64\,i386/arm64/g' -e 's/archive\.ubuntu/ports\.ubuntu/g' -e 's/security\.ubuntu/ports\.ubuntu/g' -e 's/\.com\/ubuntu/\.com\/ubuntu-ports/g' /etc/apt/sources.list | tee /etc/apt/sources.list.d/arm64.list

New version of Ubuntu LXD images seem to have a slightly different sources line. Account for those by making the trailing slash optional. Fixes: #71

peat-psuwit · 2024-02-07T18:27:22Z

Could you please test if #78 fixes the issue?

ElectrodeYT · 2024-02-08T21:31:25Z

Could you please test if #78 fixes the issue?

Still seems to be broken even with that PR for me; will look into it as well at some point, but I have some other projects as well

amartinz closed this as completed Jan 5, 2023

maciek134 reopened this Feb 25, 2023

peat-psuwit added a commit that referenced this issue Feb 7, 2024

nonsdk_container_setup: correct sources line changing

c23bb02

New version of Ubuntu LXD images seem to have a slightly different sources line. Account for those by making the trailing slash optional. Fixes: #71

peat-psuwit linked a pull request Feb 7, 2024 that will close this issue

nonsdk_container_setup: correct sources line changing #78

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing to fetch dependencies when cross compiling focal #71

Failing to fetch dependencies when cross compiling focal #71

amartinz commented Jul 7, 2022

JamiKettunen commented Sep 1, 2022

amartinz commented Sep 1, 2022

gber commented Sep 1, 2022

gber commented Sep 1, 2022

JamiKettunen commented Sep 1, 2022

gber commented Sep 1, 2022

gber commented Sep 1, 2022

JamiKettunen commented Sep 1, 2022

gber commented Sep 5, 2022

JamiKettunen commented Sep 5, 2022

gber commented Sep 6, 2022

JamiKettunen commented Jan 5, 2023

maciek134 commented Feb 25, 2023

peat-psuwit commented Feb 25, 2023

maciek134 commented Feb 26, 2023

ElectrodeYT commented Feb 7, 2024

peat-psuwit commented Feb 7, 2024

ElectrodeYT commented Feb 8, 2024

Failing to fetch dependencies when cross compiling focal #71

Failing to fetch dependencies when cross compiling focal #71

Comments

amartinz commented Jul 7, 2022

JamiKettunen commented Sep 1, 2022

amartinz commented Sep 1, 2022

gber commented Sep 1, 2022

gber commented Sep 1, 2022

JamiKettunen commented Sep 1, 2022

gber commented Sep 1, 2022

gber commented Sep 1, 2022

JamiKettunen commented Sep 1, 2022

gber commented Sep 5, 2022

JamiKettunen commented Sep 5, 2022

gber commented Sep 6, 2022

JamiKettunen commented Jan 5, 2023

maciek134 commented Feb 25, 2023

peat-psuwit commented Feb 25, 2023

maciek134 commented Feb 26, 2023

ElectrodeYT commented Feb 7, 2024

peat-psuwit commented Feb 7, 2024

ElectrodeYT commented Feb 8, 2024