Question: Remote debugging of swift code with LLDB?

[haikusw]

I'm not seeing support for remote debugging with LLDB (use case 1: from Nova on my Mac to a Docker container on the same Mac running Linux and bound to a folder on the Mac that has the source code so Nova can edit it but it can be built and run inside the docker container).

Looking inside Icarus UI and source I don't see any specific support for remote debugging with LLDB but the Readme file says:

If you are Swift or C-family language developer, Icarus can provide you with first-class support for building client- and server-side applications and frameworks.

and the "server-side applications" part made me think it might.

I was able to use a generic task (not the LLDB) task and get Nova editing and able to invoke a remote build and run command via Run Script Tasks like docker exec -i -w /app {ContainerID} swift build and docker exec -i -w /app {ContainerID} swift run but that doesn't give you debugging inside Nova via LLDB.

---

I started looking at what it would take to add support for this to Icarus and I'd have a lot to learn before I could do it. I'm working though that as time permits, but it seemed worth asking if:

1. It's already there and I'm missing it (?)
2. You are already working on adding this so there's no need for me to work on it.

TIA

[logancollins]

(Sorry for the long delay on replying to this.)

Correct, at the moment remote debugging isn't supported by the adapter, but it could be, like you suspected. By "server-side applications" here, we only meant "those being built and debugged on the Mac" for the time being.

I haven't specifically started work on this, only really because nobody had really asked until now. I don't think it would be too much work. Icarus's debug adapter essentially sits on top of LLDB.framework, which does most of the heavy lifting with attaching to and driving the debuggee. I believe LLDB.framework exposes LLDB's remote debugging functionality, as well, so it would presumably be a matter of exposing that upwards to Swift and adding support to the adapter (and lots of testing, I suppose. 😄)

I'm happy to take a look into this soon. We've been busy with the current Nova development milestone but I could use a break on something else.

[haikusw]

Hi Logan. No worries - thanks for the reply.

Since Xcode doesn't support this (remote / source code debugging a server side swift app in a local docker container (or other VM?), or remote debugging a server side swift app on a remote server), the only choice now is VS Code which is not a nice native app like Nova :)

One can use something like BBedit to edit files and then terminal logged into the container to launch builds but debugging source code debugging in an app/IDE.

As for interest: I asked a question about how to do this on one of the iOS Slack instances I'm on and got a couple of folks saying they didn't know if Nova could do this but would be very interested in finding out if it did.

So there are at least a few people who'd appreciate it :)

---

Some things I ran into thinking about this:

1. One thing that will want to be taken into account is that the cross-compilation story on macOS to build for Linux isn't great and it's not officially supported (as far as I saw). So that makes this slightly harder; have to use a swift development docker image and do the building on the "remote" server (could be a local docker container).
2. I'm new to docker and lldb remote, but I couldn't figure out how to tell lldb to launch an existing binary on the remote server without it wanting to copy the binary from the local machine (where the source was). This would require cross compilation support to be able to build linux binaries on macOS. (I consider using the bind docker command to bi-directionally share a Mac folder with the source; then the linux server could build the binary and then the lldb on the Mac could potentially copy that binary back to a different location on the server, but that seems like a bit of a hack).
3. There is another group of developers that I know would have interest in this, though I'm less familiar with that community, and that's the grou who wants to build server side swift code to run on smaller devices like a Raspberry Pi. In some cases they have insufficient memory to build on device and would want to figure out the cross compilation setup and use Nova and lldb that way. Not sure how large this group is, but given the complicated work setup they otherwise have to use, they're likely to share the joy with others and do some word-of-mouth marketing for Nova.

---

That's good to hear that the plumbing might be in lldbLib and that we maybe don't have to write the socket communication stuff entirely from scratch.

I'm dealing with some health challenges and so am only intermittently available for the next couple of months, but I'd be willing to test dev versions of this and give feedback if that would be helpful. I also might be able to contribute more once I'm done with treatment.

[logancollins]

Good news: After spending a day or two messing around, I think I've managed to get an initial implementation of remote debugging working.

I've placed it on a separate branch for the time being: https://github.com/panicinc/icarus/tree/remote-debugging.

The debug adapter has been modified to support LLDB's remote debugging features, and there's a new task template for Nova named "LLDB Remote Debug" which wires into this. It's the same adapter, just with different configuration options for remote debugging. There's configuration options for how to connect and how paths map between the local and remote file systems (for proper breakpoint and stack frame translation in Nova).

At the moment, this is only working for remote Linux systems, as I've hard-coded the LLDB platform driver name (remote-linux). I'm not sure yet if this can be autoconfigured (as I hope) or if LLDB always requires a specific platform name and it might need to be exposed in the task configuration.

The remote system needs to be running lldb-server listening on a specific port which you set into your debug task configuration in Nova.

For Docker containers, that port needs to be exposed as well as a few other options so that LLDB can connect and remotely drive properly. Here's a copy of the Docker Compose file I'm using for testing this:

services:
    lldb-server:
        image: swift
        entrypoint: /bin/bash
        command: "-c 'lldb-server p --server --listen *:8000 --min-gdbserver-port 31200 --max-gdbserver-port 31300'"
        stdin_open: true
        tty: true
        ports:
            - 8000:8000
            - 31200-31300:31200-31300
        working_dir: /app
        volumes:
            - ./:/app
        cap_add:
            - SYS_PTRACE
        security_opt:
            - seccomp=unconfined

I'll keep working on this as I can figure things out, but this is really promising. It required far less work than I'd imagined, as most of it was just figuring out the exact APIs and steps with LLDB that need to be wrapped and used in the adapter.

[haikusw]

Super Cool! Great News! I'm really looking forward to trying this out. Thanks so much for taking a look at it. I think a lot of folks will be excited about this.

[haikusw]

Related: Great that this pitch got approved. Will hopefully make the build-on-macOS_and_run-and-debug-on-Linux(etc) easier in the future.

https://forums.swift.org/t/pitch-cross-compilation-destination-bundles/61777

[logancollins]

Indeed! I was happy to see that work start.

[haikusw]

Finally getting to this (sorry for the delay) but I'm getting a build error on this 13" M1 MBP (arm64):

DebugAdapter % swift build --product LLDBAdapter --configuration release 
Building for production...
error: link command failed with exit code 1 (use -v to see invocation)
Undefined symbols for architecture arm64:
  "__ZN4lldb10SBPlatform10SetSDKRootEPKc", referenced from:
    -[LLDBPlatform setSDKRoot:] in LLDBPlatform.mm.o
ld: symbol(s) not found for architecture arm64
[3/4] Linking LLDBAdapter

• Tried with Xcode 14.2 (via xcode-select) and Xcode-beta 15.0b4 - same result.
• Tried building the main branch just to make sure my dev setup wasn't borked and that builds successfully.
• Tried building with swift build -v --product LLDBAdapter --configuration debug in case it was a debug vs release config issue. No dice.

Dumped symbols for LLDB framework and I'm not seeing setSDKRoot exported there.

Since I didn't see anyone calling setSDKRoot I commented it out and this branch then builds. #6 is a PR with this change if this is the right thing to do.

[haikusw]

Build steps in the readme don't seem quite right so I made a patch for what seems right #7.

(edited: removed comments about issues getting it to run).

Ok. So I got this branch build, loaded into Nova and activated as an extension.
It now builds and runs my little trivial test package, but it doesn't stop on breakpoint set in the source in Nova, and it's not clear it's being run in the debugger.
So I guess I'm doing something wrong.

Do I need to add a specific Run action to the "LLDB Remote Debug" Task?

For the "LLDB Remote Debug" Task I added a Build script step like this:
docker exec -i -w /app swift-dev swift build --configuration debug

where "swift-dev" is the name in the docker-compose.yml file so I could reference it easily:

services:
  lldbServer:
    image: swift
    container_name: swift-dev
  ...

That compose file is otherwise identical to what you posted above.

Perhaps I don't have the LLDB Remote Debug Task setup properly. Here's what I have:

[haikusw]

Ah, So it does look like it connected according to the debugger output, but the breakpoint wasn't hit.

[haikusw]

Happy to spend more time exercising this but I think I need a hint on how to get the debugger working (local/remote Path Mappings entry(ies) needed? Something else?).

[logancollins]

I swear this is still on my mind, sorry for the delays again. Panic was just at PAX West in Seattle at the start of September and most of my time for engineering work has been taken up by other things. I'm hoping to look into this more in the coming days, if not today!

[haikusw]

:) No problem! I saw you scrambling to complete your cool cosplay costume via Mastodon, so I knew you had important stuff you were dealing with :). That was one heck of a costume - super impressive.

Thank you. Let me know if there's anything I can do to help with testing or whatever.

I feel like it might work better than I was able to get it to work if I understood what settings to put into the LLDB Remote Debug configuration window better.

[logancollins]

Okay. After finally coming back around to this, I think the issue might be that you need to establish one or more Path Mappings in your task configuration so that Nova can map paths on the remote side to the local project. I do not have this well documented right now, so I will try and improve that.

For your case, I think it'd be as easy as something like I have here:

This maps . / the local project root to /app/ on the remote side, as that is where the project is mounted in Docker for me. I think yours is as well from the Docker commands you listed.

If you set this up, does it by chance start working? 😅

[logancollins]

I also see that Nova 11 appears to have a bug where it duplicates the fields for the Path Mapping below as a single item (see screenshot below). We've already fixed this for the upcoming Nova 12, but for now I guess just ignore the individual field for this and use the list / table above. 😓

Ignore this:

How I didn't catch this earlier, I don't know. I've just been glossing over it visually, apparently.