Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to retireve image info for instance for non-admin user #62

Closed
keestux opened this issue May 6, 2024 · 23 comments
Closed

Failed to retireve image info for instance for non-admin user #62

keestux opened this issue May 6, 2024 · 23 comments
Labels
Incomplete Waiting on more information from reporter

Comments

@keestux
Copy link

keestux commented May 6, 2024 •
edited

OK. The typo is one thing (occurs four times).

The real issue is that I'm getting this error for a simple example. I have added an alias to a local image. Then
this example:

resource "incus_instance" "c9" {
  name  = "c9"
  image = "local:centos/9-Stream/cloud/vm"
  type = "virtual-machine"

  config = {
    "boot.autostart" = true
    "security.secureboot" = false
  }

  limits = {
    cpu = 2
  }
  profiles = ["default", "user-kees-centos", "config-centos"]
}

The user is member of the incus group, not incus-admin.

incus_instance.c9: Creating...
╷
│ Error: Failed to retireve image info for instance "c9"
│ 
│   with incus_instance.c9,
│   on main.tf line 1, in resource "incus_instance" "c9":
│    1: resource "incus_instance" "c9" {
│ 
│ Image not found
╵

If I run this with an incus-admin user it succeeds.
@keestux
Copy link
Author

keestux commented May 7, 2024

Here is a more complete log.

$ incus project ls
+----------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
|         NAME         | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |                 DESCRIPTION                 | USED BY |
+----------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
| user-60001 (current) | YES    | YES      | YES             | YES             | NO       | YES           | User restricted project for "keesb" (60001) | 55      |
+----------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
$ incus image ls local:ubuntu/focal/cloud
+-----------------------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+-----------------------+
|         ALIAS         | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |      TYPE       |   SIZE    |      UPLOAD DATE      |
+-----------------------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+-----------------------+
| ubuntu/focal/cloud    | 85e9afc8d65b | no     | Ubuntu focal amd64 (cloud) (20240423_07:42) | x86_64       | CONTAINER       | 128.90MiB | 2024/04/29 16:46 CEST |
+-----------------------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+-----------------------+
| ubuntu/focal/cloud/vm | ba068bd18db5 | no     | Ubuntu focal amd64 (cloud) (20240423_07:42) | x86_64       | VIRTUAL-MACHINE | 266.59MiB | 2024/04/29 16:45 CEST |
+-----------------------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+-----------------------+
$ cat main.tf
resource "incus_instance" "u22" {
  name  = "u22"
  image = "local:ubuntu/focal/cloud"

  profiles  = ["default"]
}
$ INCUS_SOCKET=/var/lib/incus/unix.socket.user tofu apply

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

OpenTofu will perform the following actions:

  # incus_instance.u22 will be created
  + resource "incus_instance" "u22" {
      + config           = {}
      + ephemeral        = false
      + image            = "local:ubuntu/focal/cloud"
      + ipv4_address     = (known after apply)
      + ipv6_address     = (known after apply)
      + limits           = {}
      + mac_address      = (known after apply)
      + name             = "u22"
      + profiles         = [
          + "default",
        ]
      + running          = true
      + status           = (known after apply)
      + target           = (known after apply)
      + type             = "container"
      + wait_for_network = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

incus_instance.u22: Creating...
╷
│ Error: Failed to retireve image info for instance "u22"
│
│   with incus_instance.u22,
│   on main.tf line 1, in resource "incus_instance" "u22":
│    1: resource "incus_instance" "u22" {
│
│ Image not found
╵
$ incus launch local:ubuntu/focal/cloud u22
Launching u22
$

@maveonair
Copy link
Member

Could you please try to use the image's fingerprint and post the output:

resource "incus_instance" "u22" {
  name  = "u22"
  image = "local:85e9afc8d65b"

  profiles  = ["default"]
}

$ TF_LOG=debug INCUS_SOCKET=/var/lib/incus/unix.socket.user tofu apply

@keestux
Copy link
Author

keestux commented May 21, 2024

Still failing

$ INCUS_SOCKET=/var/lib/incus/unix.socket.user tofu apply

OpenTofu used the selected providers to generate the following execution plan. Resource actions are indicated with the
following symbols:
  + create

OpenTofu will perform the following actions:

  # incus_instance.u22 will be created
  + resource "incus_instance" "u22" {
      + config           = {}
      + ephemeral        = false
      + image            = "local:85e9afc8d65b"
      + ipv4_address     = (known after apply)
      + ipv6_address     = (known after apply)
      + limits           = {}
      + mac_address      = (known after apply)
      + name             = "u22"
      + profiles         = [
          + "default",
        ]
      + running          = true
      + status           = (known after apply)
      + target           = (known after apply)
      + type             = "container"
      + wait_for_network = true
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  OpenTofu will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

incus_instance.u22: Creating...
╷
│ Error: Failed to retireve image info for instance "u22"
│ 
│   with incus_instance.u22,
│   on main.tf line 1, in resource "incus_instance" "u22":
│    1: resource "incus_instance" "u22" {
│ 
│ Image "85e9afc8d65bc996e464f7df1a8a974634124089c3dabf0e78656dd30f63a8fb" not found
╵

@maveonair
Copy link
Member

maveonair commented May 27, 2024 •
edited

I tried to reconstruct your error, and I was able to create an instance by using local:<image-id> with a user-managed project.

Could you please use the following command so that I can see all the debug information:

TF_LOG=debug INCUS_SOCKET=/var/lib/incus/unix.socket.user tofu apply

@maveonair maveonair added the Incomplete Waiting on more information from reporter label May 27, 2024
@keestux
Copy link
Author

keestux commented May 28, 2024

Here is the debug log

debug.log

@maveonair
Copy link
Member

maveonair commented May 30, 2024 •
edited

I think you forgot to set the correct p

Here is the debug log

debug.log

I think the problem is that you are not setting the project in your Terraform/Opentofu resources. It is designed to use the default project to create resources when the project for a resource is not set.

However, in your case this fails because you can only create resources for the user-60001 and not the default project. The Incus CLI works because it recognizes that you only have user rights and therefore can only create resources for your "user" isolated project.

Can you please try this?

resource "incus_instance" "u22" {
  name  = "u22"
  image = "local:85e9afc8d65b"

  project  = "user-60001"
  profiles = ["default"]
}

@keestux
Copy link
Author

keestux commented May 31, 2024

Here is the new debug log with the addition of project. Notice that there is a new image, new fingerprint. The same image is present in project default and in project user-60001. That's probably why it prints the full fingerprint.

debug2.log

And here is another debug log. This time I have first deleted the image with that fingerprint from the default project.

debug3.log

@maveonair
Copy link
Member

maveonair commented May 31, 2024 •
edited

I am having a hard time reproducing your problem. I have tested your original Terraform code on a fresh installation of Debian 12 and Ubuntu 24.04 with a user-only project (user only is part of the incus group, no incus-admin assigned) and it worked without any problems.

Could you please provide me the output of:

$ incus info

@keestux
Copy link
Author

keestux commented May 31, 2024

Can it be related to the fact that my user is a FreeIPA user? I'll try to test it with a "local" user account.

@keestux
Copy link
Author

keestux commented May 31, 2024

Confirmed. For a local user works fine. For a FreeIPA user does not.

So, now we have to figure out why that is. FreeIPA uid/gid comes from sssd. But why would that matter?

@maveonair
Copy link
Member

@stgraber Do you have any idea why there is this problem with a FreeIPA user when interacting with Incus?

@stgraber
Copy link
Member

@maveonair When running as an unprivileged user, you don't deal with the default project but with a user-specific project (user-UID).

It's mostly transparent on the CLI as the incus tool will just detect it only has access to a single project and then use that for all requests. The Terraform provider however doesn't know that, so the user would need to specify the correct project in their TF files.

incus project list will show the project.

@maveonair
Copy link
Member

@maveonair When running as an unprivileged user, you don't deal with the default project but with a user-specific project (user-UID).

It's mostly transparent on the CLI as the incus tool will just detect it only has access to a single project and then use that for all requests. The Terraform provider however doesn't know that, so the user would need to specify the correct project in their TF files.

incus project list will show the project.

Yes. That's what I said a few comments earlier. @keestux tested it with a local user where specifying the user-specific project works, but it doesn't work when he uses his FreeIPA user.

@keestux
Copy link
Author

keestux commented May 31, 2024

@stgraber I already do that (as earlier suggested by @maveonair). That certainly helps with a "local" user, but not for a FreeIPA user.

@stgraber
Copy link
Member

What do you get for the FreeIPA user when you do incus project list and id?
It may also be useful to run incus monitor --pretty --all-projects (as root) to see what requests Incus is getting.

@keestux
Copy link
Author

keestux commented Jun 1, 2024 

$ incus project ls
+----------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
|         NAME         | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |                 DESCRIPTION                 | USED BY |
+----------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
| user-60001 (current) | YES    | YES      | YES             | YES             | NO       | YES           | User restricted project for "keesb" (60001) | 64      |
+----------------------+--------+----------+-----------------+-----------------+----------+---------------+---------------------------------------------+---------+
$ sudo incus project ls
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+----------------------------------------------+---------+
|       NAME        | IMAGES | PROFILES | STORAGE VOLUMES | STORAGE BUCKETS | NETWORKS | NETWORK ZONES |                 DESCRIPTION                  | USED BY |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+----------------------------------------------+---------+
| default (current) | YES    | YES      | YES             | YES             | YES      | YES           | Default Incus project                        | 16      |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+----------------------------------------------+---------+
| user-1001         | YES    | YES      | YES             | YES             | NO       | YES           | User restricted project for "local01" (1001) | 2       |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+----------------------------------------------+---------+
| user-60001        | YES    | YES      | YES             | YES             | NO       | YES           | User restricted project for "keesb" (60001)  | 64      |
+-------------------+--------+----------+-----------------+-----------------+----------+---------------+----------------------------------------------+---------+

@keestux
Copy link
Author

keestux commented Jun 1, 2024

First, monitor logging when the FreeIPA user (uid 60001) does the tofu apply

DEBUG  [2024-06-01T14:16:27+02:00] Event listener server handler started         id=51d3def8-113b-4743-a748-108b2963a742 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/networks username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/storage-pools username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/profiles/default username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/storage-pools?recursion=1" username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/networks?recursion=1" username=root
DEBUG  [2024-06-01T14:17:08+02:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/profiles/default username=root
INFO   [2024-06-01T14:17:08+02:00] Action: profile-updated, Source: /1.0/profiles/default, Requestor: unix/root (@) 
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Handling API request                          ip=@ method=GET protocol=tls url=/1.0 username=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Handling API request                          ip=@ method=GET protocol=tls url=/1.0 username=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf
DEBUG  [2024-06-01T14:17:08+02:00] Handling API request                          ip=@ method=GET protocol=tls url=/1.0/images/aliases/f58b52a245ef username=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Matched trusted cert                          fingerprint=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:17:08+02:00] Handling API request                          ip=@ method=GET protocol=tls url=/1.0/images/f58b52a245ef username=ec4af657b0be602bf6b5f1aa588b0ca0d17bb390b4c3744ec20fe8f3188b2fbf

@keestux
Copy link
Author

keestux commented Jun 1, 2024

Next, the monitoring when a local user (uid 1001) does the tofu apply

DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/networks username=root
DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/storage-pools username=root
DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/profiles/default username=root
DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/storage-pools?recursion=1" username=root
DEBUG  [2024-06-01T14:18:53+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/networks?recursion=1" username=root
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/profiles/default username=root
INFO   [2024-06-01T14:18:54+02:00] Action: profile-updated, Source: /1.0/profiles/default, Requestor: unix/root (@) 
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url=/1.0 username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/images/aliases/f58b52a245ef?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/images/f58b52a245ef?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/events?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Event listener server handler started         id=7c0bf00d-2f2c-44df-9003-ee89cb6d484e local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=POST protocol=tls url="/1.0/instances?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Responding to instance create                
DEBUG  [2024-06-01T14:18:54+02:00] New operation                                 class=task description="Creating instance" operation=3e41fb56-df1e-436a-97d1-4396117b0800 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Lock acquired for image                       fingerprint=f58b52a245efc700780b12046aa21741f5113d3e92dab20d77ebd28b629dc0ce
DEBUG  [2024-06-01T14:18:54+02:00] Started operation                             class=task description="Creating instance" operation=3e41fb56-df1e-436a-97d1-4396117b0800 project=user-1001
INFO   [2024-06-01T14:18:54+02:00] ID: 3e41fb56-df1e-436a-97d1-4396117b0800, Class: task, Description: Creating instance  CreatedAt="2024-06-01 14:18:54.204139609 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[containers:[/1.0/instances/u22?project=user-1001] instances:[/1.0/instances/u22?project=user-1001]]" Status=Pending StatusCode=Pending UpdatedAt="2024-06-01 14:18:54.204139609 +0200 CEST"
INFO   [2024-06-01T14:18:54+02:00] ID: 3e41fb56-df1e-436a-97d1-4396117b0800, Class: task, Description: Creating instance  CreatedAt="2024-06-01 14:18:54.204139609 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[containers:[/1.0/instances/u22?project=user-1001] instances:[/1.0/instances/u22?project=user-1001]]" Status=Running StatusCode=Running UpdatedAt="2024-06-01 14:18:54.204139609 +0200 CEST"
DEBUG  [2024-06-01T14:18:54+02:00] Acquiring lock for image                      fingerprint=f58b52a245efc700780b12046aa21741f5113d3e92dab20d77ebd28b629dc0ce
DEBUG  [2024-06-01T14:18:54+02:00] Instance operation lock created               action=create instance=u22 project=user-1001 reusable=false
INFO   [2024-06-01T14:18:54+02:00] Creating instance                             ephemeral=false instance=u22 instanceType=container project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/operations/3e41fb56-df1e-436a-97d1-4396117b0800?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Adding device                                 device=eth0 instance=u22 instanceType=container project=user-1001 type=nic
INFO   [2024-06-01T14:18:54+02:00] Created instance                              ephemeral=false instance=u22 instanceType=container project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Adding device                                 device=root instance=u22 instanceType=container project=user-1001 type=disk
INFO   [2024-06-01T14:18:54+02:00] Action: instance-created, Source: /1.0/instances/u22?project=user-1001  location=none storage-pool=pool2 type=container
DEBUG  [2024-06-01T14:18:54+02:00] CreateInstanceFromImage started               driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] EnsureImage started                           driver=btrfs fingerprint=f58b52a245efc700780b12046aa21741f5113d3e92dab20d77ebd28b629dc0ce pool=pool2
DEBUG  [2024-06-01T14:18:54+02:00] Setting image volume size                     driver=btrfs fingerprint=f58b52a245efc700780b12046aa21741f5113d3e92dab20d77ebd28b629dc0ce pool=pool2 size=
DEBUG  [2024-06-01T14:18:54+02:00] Checking image volume size                    driver=btrfs fingerprint=f58b52a245efc700780b12046aa21741f5113d3e92dab20d77ebd28b629dc0ce pool=pool2
DEBUG  [2024-06-01T14:18:54+02:00] EnsureImage finished                          driver=btrfs fingerprint=f58b52a245efc700780b12046aa21741f5113d3e92dab20d77ebd28b629dc0ce pool=pool2
DEBUG  [2024-06-01T14:18:54+02:00] Checking volume size                          driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Set new volume size                           driver=btrfs instance=u22 pool=pool2 project=user-1001 size=
DEBUG  [2024-06-01T14:18:54+02:00] UpdateInstanceBackupFile started              driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] CreateInstanceFromImage finished              driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Success for operation                         class=task description="Creating instance" operation=3e41fb56-df1e-436a-97d1-4396117b0800 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] UpdateInstanceBackupFile finished             driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Instance operation lock finished              action=create err="<nil>" instance=u22 project=user-1001 reusable=false
INFO   [2024-06-01T14:18:54+02:00] ID: 3e41fb56-df1e-436a-97d1-4396117b0800, Class: task, Description: Creating instance  CreatedAt="2024-06-01 14:18:54.204139609 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[containers:[/1.0/instances/u22?project=user-1001] instances:[/1.0/instances/u22?project=user-1001]]" Status=Success StatusCode=Success UpdatedAt="2024-06-01 14:18:54.204139609 +0200 CEST"
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/instances/u22/state?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] GetInstanceUsage started                      driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] GetInstanceUsage finished                     driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=PUT protocol=tls url="/1.0/instances/u22/state?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] New operation                                 class=task description="Starting instance" operation=1784bf2b-a189-4632-956c-93cbae3aef37 project=user-1001
INFO   [2024-06-01T14:18:54+02:00] ID: 1784bf2b-a189-4632-956c-93cbae3aef37, Class: task, Description: Starting instance  CreatedAt="2024-06-01 14:18:54.383641647 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/u22?project=user-1001]]" Status=Running StatusCode=Running UpdatedAt="2024-06-01 14:18:54.383641647 +0200 CEST"
DEBUG  [2024-06-01T14:18:54+02:00] Started operation                             class=task description="Starting instance" operation=1784bf2b-a189-4632-956c-93cbae3aef37 project=user-1001
INFO   [2024-06-01T14:18:54+02:00] ID: 1784bf2b-a189-4632-956c-93cbae3aef37, Class: task, Description: Starting instance  CreatedAt="2024-06-01 14:18:54.383641647 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/u22?project=user-1001]]" Status=Pending StatusCode=Pending UpdatedAt="2024-06-01 14:18:54.383641647 +0200 CEST"
DEBUG  [2024-06-01T14:18:54+02:00] Start started                                 instance=u22 instanceType=container project=user-1001 stateful=false
INFO   [2024-06-01T14:18:54+02:00] Starting instance                             action=start created="2024-06-01 12:18:54.219910737 +0000 UTC" ephemeral=false instance=u22 instanceType=container project=user-1001 stateful=false used="1970-01-01 00:00:00 +0000 UTC"
DEBUG  [2024-06-01T14:18:54+02:00] Instance operation lock created               action=start instance=u22 project=user-1001 reusable=false
DEBUG  [2024-06-01T14:18:54+02:00] MountInstance started                         driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] MountInstance finished                        driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/operations/1784bf2b-a189-4632-956c-93cbae3aef37?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:54+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:54+02:00] Starting device                               device=eth0 instance=u22 instanceType=container project=user-1001 type=nic
DEBUG  [2024-06-01T14:18:54+02:00] UpdateInstanceBackupFile started              driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Starting device                               device=root instance=u22 instanceType=container project=user-1001 type=disk
DEBUG  [2024-06-01T14:18:54+02:00] UpdateInstanceBackupFile finished             driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Skipping unmount as in use                    driver=btrfs pool=pool2 refCount=1 volName=user-1001_u22
DEBUG  [2024-06-01T14:18:54+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/internal/containers/u22/onstart?project=user-1001" username=root
DEBUG  [2024-06-01T14:18:54+02:00] Scheduler: container u22 started: re-balancing 
DEBUG  [2024-06-01T14:18:54+02:00] Success for operation                         class=task description="Starting instance" operation=1784bf2b-a189-4632-956c-93cbae3aef37 project=user-1001
DEBUG  [2024-06-01T14:18:54+02:00] Start finished                                instance=u22 instanceType=container project=user-1001 stateful=false
DEBUG  [2024-06-01T14:18:54+02:00] Instance operation lock finished              action=start err="<nil>" instance=u22 project=user-1001 reusable=false
INFO   [2024-06-01T14:18:54+02:00] Started instance                              action=start created="2024-06-01 12:18:54.219910737 +0000 UTC" ephemeral=false instance=u22 instanceType=container project=user-1001 stateful=false used="1970-01-01 00:00:00 +0000 UTC"
INFO   [2024-06-01T14:18:54+02:00] Action: instance-started, Source: /1.0/instances/u22?project=user-1001, Requestor: tls/4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 (@) 
INFO   [2024-06-01T14:18:54+02:00] ID: 1784bf2b-a189-4632-956c-93cbae3aef37, Class: task, Description: Starting instance  CreatedAt="2024-06-01 14:18:54.383641647 +0200 CEST" Err= Location=none MayCancel=false Metadata="map[]" Resources="map[instances:[/1.0/instances/u22?project=user-1001]]" Status=Success StatusCode=Success UpdatedAt="2024-06-01 14:18:54.383641647 +0200 CEST"
DEBUG  [2024-06-01T14:18:56+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:56+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/instances/u22/state?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:56+02:00] GetInstanceUsage started                      driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:56+02:00] GetInstanceUsage finished                     driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:58+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:58+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/instances/u22/state?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:58+02:00] GetInstanceUsage started                      driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:58+02:00] GetInstanceUsage finished                     driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:58+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/instances/u22?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:58+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:58+02:00] Handling API request                          ip=@ method=GET protocol=tls url="/1.0/instances/u22/state?project=user-1001" username=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2
DEBUG  [2024-06-01T14:18:58+02:00] Matched trusted cert                          fingerprint=4dbc389f43c7a50ca55048528bde4bbe24577049407fea00aee723eaa03642d2 subject="CN=root@winkel,O=Linux Containers"
DEBUG  [2024-06-01T14:18:58+02:00] GetInstanceUsage started                      driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:58+02:00] GetInstanceUsage finished                     driver=btrfs instance=u22 pool=pool2 project=user-1001
DEBUG  [2024-06-01T14:18:58+02:00] Event listener server handler stopped         listener=7c0bf00d-2f2c-44df-9003-ee89cb6d484e local=/var/lib/incus/unix.socket remote=@

@stgraber
Copy link
Member

stgraber commented Jun 3, 2024

So the different is pretty clear, in both cases we can see the checks that incus-user does on first run:

DEBUG  [2024-06-01T14:16:27+02:00] Event listener server handler started         id=51d3def8-113b-4743-a748-108b2963a742 local=/var/lib/incus/unix.socket remote=@
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/networks username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/storage-pools username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0 username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url=/1.0/profiles/default username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/storage-pools?recursion=1" username=root
DEBUG  [2024-06-01T14:17:07+02:00] Handling API request                          ip=@ method=GET protocol=unix url="/1.0/networks?recursion=1" username=root
DEBUG  [2024-06-01T14:17:08+02:00] Handling API request                          ip=@ method=PUT protocol=unix url=/1.0/profiles/default username=root
INFO   [2024-06-01T14:17:08+02:00] Action: profile-updated, Source: /1.0/profiles/default, Requestor: unix/root (@)

(I'm sending a fix for the profile update as that shouldn't happen every time).

After that initial chunk is where we see a clear difference.

In the first case (not working), we see it hitting all the URLs without the ?project=user-60001 which causes the Terraform failure.

In the second case (working), we see it hitting all the URLs with ?project=user-1001 and so things work properly.

I'm not seeing any Incus issue here, it seems to really be a client issue, that is, Terraform not specifying the project that it's supposed to be acting on.

Can you post the .tf you're working with in the failing case?

@keestux
Copy link
Author

keestux commented Jun 3, 2024 •
edited

Can you post the .tf you're working with in the failing case?

A very simple .tf

resource "incus_instance" "u22" {
  name  = "u22"
  image = "local:85e9afc8d65b"

  project  = "user-60001"
  profiles = ["default"]
}

Of course with user-1001 for the local user. The use of image fingerprint was a suggestion earlier in this issue, but I guess that it not needed. I was using local:ubuntu/focal/cloud. That's local: because I'm in an air-gapped environment.

@stgraber
Copy link
Member

stgraber commented Jun 6, 2024

Okay, so that .tf would certainly tell Terraform to use the user-60001 project for the instance creation, but because image doesn't refer to a Terraform object, nothing is telling it to use the user-60001 project for that image.

So it'd just use the default configuration for the local remote.

Can you do incus project switch local:user-60001 and then run terraform apply again, see if that sorts it out?

@keestux
Copy link
Author

keestux commented Jun 6, 2024

Yes that worked.

But how? I don't understand, because that user only has one project, namely its own user-60001.

Unfortunately I forgot to test it once more before entering the switch command, so I don't know if the problem solved itself in the last couple of days..

@keestux
Copy link
Author

keestux commented Jun 7, 2024

Strange. I can't make it fail anymore. I'll close the issue.

@keestux keestux closed this as completed Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Incomplete Waiting on more information from reporter
Milestone
No milestone
Development

No branches or pull requests
3 participants
@keestux @maveonair @stgraber