Command line utility

[Anjali05]

Hi,

Can I invoke clangd from the command line? I am working on a static analyzer and was wondering if I can use it like cscope from the command line. I have been using cscope but it has some limitations. For example, for making a function call graph from a start symbol cscope fails to work for certain functions with complex definitions such as (https://elixir.bootlin.com/linux/v6.1/source/drivers/tty/tty_io.c#L957 ) and (https://elixir.bootlin.com/linux/v6.1/source/kernel/cgroup/cgroup.c#L6565). I have looking for a capability similar to cscope -dL2 <function symbol> that lists all the function calls made from the passed function symbol.

Any help/insight into this will be appreciated. Thanks in advance!

[HighCommander4]

Clangd wasn't really designed with this use case in mind, but one way of achieving this that seems relatively low effort is to write a script that talks to clangd over the Language Server Protocol to query some information and then presents it in whatever way you want (e.g. printing the results to the command line).

For an example of this written in python, have a look at https://github.com/simark/ls-interact/blob/master/test_clangd.py. (This file exercises the "go to definition" request, but there's no reason you couldn't do something similar for other requests like "find references" or "call hierarchy".)

[Anjali05]

@HighCommander4 Thanks! Where can I find a list of all possible requests available?

[HighCommander4]

You can find them in the Language Server Protocol specification.

The ls_interact tool has python wrapper classes for some of them but by no means all, but it should be straightforward to look at an existing one like Hover and get an idea of how to write one for other requests.

[Anjali05]

@HighCommander4 Thanks! I will look into this. I am planning to use pylspclient library for this.

[HighCommander4]

I am planning to use pylspclient library for this.

Ah, nice, yeah that seems more fully-featured and better maintained than ls_interact which was a utility mostly written for testing purposes.

[Anjali05]

@HighCommander4 I ended up reusing the code that you pointed me to. I have been able to get the setup working for my project (Linux kernel). So I have added textDocument/documentSymbol, textDocument/prepareCallHierarchy and allHierarchy/outgoingCalls. I was testing this on a file. So these are the steps I followed:

1. I get all the symbols info by making a call to textDocument/documentSymbol to get the correct line and character information for all the symbols. This is working fine.
2. I pass the line and character to textDocument/prepareCallHierarchy. This works only for functions that do not start with a comments like these:https://elixir.bootlin.com/linux/v6.1/source/fs/inode.c#L2016 vs the one that do like these: https://elixir.bootlin.com/linux/v6.1/source/fs/inode.c#L1885
3. And then pass the result I get from textDocument/prepareCallHierarchy like range, name, selectionRange etc to allHierarchy/outgoingCalls in order to get all the function calls made by the symbol of name. I have not been able to get this work at all. I was going through the doc and it said "It is only issued if a server registers for the textDocument/prepareCallHierarchy request." (https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#callHierarchy_outgoingCalls). I am not sure what that statement means.

Let me know if you have any thoughts on this.

[Anjali05]

@HighCommander4 I did test for this example and it seems to be working. I am going to test a couple of more and let you know if I have any issues.

[Anjali05]

For iget5_locked, it does not seem to resolve unlikely to any function. There are multiple macros of the same name and I think some of them resolve

1. to a variable such as this: https://elixir.bootlin.com/linux/v6.1/source/arch/x86/tools/insn_decoder_test.c#L14.
2. But for some other such as this, https://elixir.bootlin.com/linux/v6.1/source/include/linux/compiler.h#L48 shouldn't it return functions like ftrace_likely_update.

Basically, I am trying to understand how OutGoingcall is working. I understand if iget5_locked is using unlikely that resolves to a variable then I won't see ftrace_likely_update. But shouldn't it return all the possible ways unlikely is resolved? I do not care about it being resolved to all possible cases cause that's not accurate but I want to understand how indexing works.

[HighCommander4]

shouldn't it return all the possible ways unlikely is resolved?

Clangd sees the code the way a compiler would when compiling using the command from compile_commands.json. So, for a preprocessor if/else, only one branch will be visible and parsed.

For example, suppose you have:

#ifdef FOO
#define foo_wrapper(x) foo_function(x)
#else
#define foo_wrapper(x)
#endif

I the file's compile command in compile_commands.json contains -DFOO, then only the first branch is parsed and a use of foo_wrapper will be recorded as a call to foo_function. But if the compile command does not contain -DFOO, then only the second branch is parsed and a use of foo_wrapper will not be recorded as a call to anything.

[HighCommander4]

It's probably instructive to open the codebase in an editor using clangd (for example, vscode), using the same compile_commands.json and command-line flags to clangd (if applicable) as your script, to get a picture of how clangd sees the code. You can e.g. use the editor's go-to-definition feature to see which definition clangd is resolving a particular use of a macro to.

[Anjali05]

Thank you! I will take a look.

[Anjali05]

@HighCommander4 I have a clarifying question about textDocument/references. I have been testing this feature to get references for a symbol. My goal is to get all the references for a given lock name such as inode_hash_lock here. There are two things thats happening:

• First thing that I need is the correct position for inode_hash_lock from textDocument/documentSymbol but I am not getting it. It only returns line and character number for DEFINE_SPINLOCK. I can get the position for inode_hash_lock by adding the characters from DEFINE_SPINLOCK and get references but that returns an empty result.
• The other thing is when I run textDocument/references for the position of DEFINE_SPINLOCK it just returns references in the given path and not across all files under the project dir.

Let me know if I am missing anything in making these calls. Currently, I use a combination of ctags and cscope to get the lock and unlock calls for this lock which is one of the goals for the analysis but since I have decided to move to clangd which seems more accurate I want to port this feature as well.

[HighCommander4]

• First thing that I need is the correct position for inode_hash_lock from textDocument/documentSymbol but I am not getting it. It only returns line and character number for DEFINE_SPINLOCK.

I could reproduce this with a simple example, and I agree that we could do better here. I filed #1941 about it.

I can get the position for inode_hash_lock by adding the characters from DEFINE_SPINLOCK and get references but that returns an empty result.

Can you reproduce this with a simple example like this one?

struct mutex_t {};

#define DEFINE_SPINLOCK(x)	mutex_t x = mutex_t{}

static DEFINE_SPINLOCK(inode_hash_lock);

void lock(struct mutex_t*);
void unlock(struct mutex_t*);

int main() {
  lock(&inode_hash_lock);
  unlock(&inode_hash_lock);
}

In the editor at least, invoking find-refs at the inode_hash_lock token does produce the two references in main() as results.

• The other thing is when I run textDocument/references for the position of DEFINE_SPINLOCK it just returns references in the given path and not across all files under the project dir.

The reason for this may be that clangd has not finished indexing the project yet. If you open the project directory in an editor like vscode, it should indicate the indexing status in the status bar at the bottom. If it just says "clangd: idle", that indicates the indexing has completed. If you see something like "indexing: 1/100" beside it, that indicates it has more files to index, in which case you can wait for that to complete.

(I don't know if there is an easy way to progammatically wait for the indexing to complete. In vscode, the status bar messages come from $/progress notifications sent by the server; you could conceivably listen to those from your script and parse their contents.)

[Anjali05]

@HighCommander4 This is happening for callHierarchy/incomingCalls as well. I am trying to generate for this function https://elixir.bootlin.com/linux/v6.1/source/kernel/locking/mutex.c#L797, but it only returns incoming calls from the same file and not project wide.

server --> client: {
    "id": 125,
    "jsonrpc": "2.0",
    "result": [
        {
            "from": {
                "data": "2C32EFB79FA0A44A",
                "detail": "atomic_dec_and_mutex_lock",
                "kind": 12,
                "name": "atomic_dec_and_mutex_lock",
                "range": {
                    "end": {
                        "character": 29,
                        "line": 1135
                    },
                    "start": {
                        "character": 4,
                        "line": 1135
                    }
                },
                "selectionRange": {
                    "end": {
                        "character": 29,
                        "line": 1135
                    },
                    "start": {
                        "character": 4,
                        "line": 1135
                    }
                },
                "uri": "file:///root/linux-6.1-clangd/kernel/locking/mutex.c"
            },
            "fromRanges": [
                {
                    "end": {
                        "character": 11,
                        "line": 1141
                    },
                    "start": {
                        "character": 1,
                        "line": 1141
                    }
                }
            ]
        },
        {
            "from": {
                "data": "127C6163CF227CE4",
                "detail": "mutex_lock_nested",
                "kind": 12,
                "name": "mutex_lock_nested",
                "range": {
                    "end": {
                        "character": 17,
                        "line": 796
                    },
                    "start": {
                        "character": 0,
                        "line": 796
                    }
                },
                "selectionRange": {
                    "end": {
                        "character": 17,
                        "line": 796
                    },
                    "start": {
                        "character": 0,
                        "line": 796
                    }
                },
                "uri": "file:///root/linux-6.1-clangd/kernel/locking/mutex.c"
            },
            "fromRanges": [
                {
                    "end": {
                        "character": 35,
                        "line": 801
                    },
                    "start": {
                        "character": 18,
                        "line": 801
                    }
                }
            ]
        }
    ]
}

I have implemented the same way I did outgoingCalls. Will checking out this PR fix it?

[HighCommander4]

This is happening for callHierarchy/incomingCalls as well. I am trying to generate for this function https://elixir.bootlin.com/linux/v6.1/source/kernel/locking/mutex.c#L797, but it only returns incoming calls from the same file and not project wide.

Incoming calls is similar to outgoing calls in that it depends on clangd's index for results in other files, so the hypothesis from the previous comment (that clangd does not have the time to index the project) applies here as well.

Will checking out this PR fix it?

No, this fix only affects the issue of selectionRange being the location of the macro rather than the location of the variable name for variables (or functions etc.) defined by macro calls, such as DEFINE_SPINLOCK(inode_hash_lock).

[Anjali05]

I will try to increase the sleep time, I was trying with 15 seconds as it worked for outgoingCalls.

[HighCommander4]

I will try to increase the sleep time, I was trying with 15 seconds as it worked for outgoingCalls.

Forgot to mention one thing: there are two different relevant time scales here:

1. When first encountering the codebase (or new changes in the codebase such as after doing git pull), clangd needs to build the index (or update the index to reflect changes since the last time it was built). This can take (much) longer than 15 seconds on a codebase the size of the Linux kernel, but it only needs to be done once (or once per git pull of the codebase, with incremental updates being faster than the initial build), as the results are persisted to disk.
2. On every subsequent startup, clangd needs a bit of time to load the already-built index into memory before it's ready to use its contents when answering requests like incomingCalls.

A 15-second sleep sounds fine for (2), but for (1) you presumably need a different solution than sleeping, e.g. opening the codebase in an editor and waiting for clangd's initial index build to complete.

[Anjali05]

@HighCommander4 I need some suggestion on something I am thinking of implementing and wanted to check if its even possible. So I want to generate a mapping of all the locks in the linux kernel with the data structure they are associated with. For example: https://elixir.bootlin.com/linux/v6.1/source/include/net/af_unix.h#L55, in this I want to get to point where I know spinlock_t lock belongs to struct unix_sock. So far I have been able to use the children information that is there in the documentSymbol for class type and then in the children I can match with lock data types (such as spinlock_t, struct mutex) of any variable for that struct and get the lock names belonging to a data structure.

However, there are times when the lock name is generic like the one in example I have given. So I was hoping to get the data structure by statically analyzing the function body that acquires the lock from the traces I have like this one: https://elixir.bootlin.com/linux/v6.1/source/net/unix/af_unix.c#L472. I was hoping to get a similar children structure for functions as well but it is not generated in the documentSymbol. Let me know if you think of any way I can achieve this. Thanks!

[Anjali05]

@HighCommander4 Simplifying this question, I want to know if it's possible to get the type of variables and their names accessed from within a function.

[HighCommander4]

A way that occurs to me is to use the textDocument/ast request to ask the server to send a representation of the AST for a fragment of code.

To take an example:

struct lock_t {};

void lock(struct lock_t);

struct bar {
  struct lock_t lock;
};

void foo(struct bar* b) {
  lock(b->lock);
}

if I select lines 9-11 (i.e. the entire definition of the function foo) in vscode and perform the command clangd: show AST, it sends the server a request like this:

{
  "id": 10,
  "jsonrpc": "2.0",
  "method": "textDocument/ast",
  "params": {
    "range": {
      "end": {
        "character": 1,
        "line": 10
      },
      "start": {
        "character": 0,
        "line": 8
      }
    },
    "textDocument": {
      "uri": "file:///path/to/test.c"
    }
  }
}

and the server responds with a JSON representation of the AST for the selected code.

VSCode visualizes the returned AST like this:

but the JSON response also contains source ranges for each node.

You can tell from this AST that a function named lock is called with an argument which is a member reference expression accessing a member named lock of a variable named b.

This AST fragment does not directly contain information about the types of these variables, but you could e.g. then call textDocument/definition on the source range of the field name lock to get to its definition and examine its type there.

[Anjali05]

@HighCommander4 I will try it. Thanks!

[Anjali05]

@HighCommander4 Can you please confirm if the textDocument/implementation is not broken? I just get empty results from this request and I have tried a couple of times.

For context, I made a call to textDocument/getAST and then called textDocument/definition for the data structure associated with the lock eg: https://elixir.bootlin.com/linux/v6.1/source/kernel/bpf/percpu_freelist.c#L67. This returns the definition of the data structure within the function like this:

[
    {
        "range": {
            "end": {
                "character": 34,
                "line": 63
            },
            "start": {
                "character": 30,
                "line": 63
            }
        },
        "uri": "file:///mydata/clangd-linux-6.1/kernel/bpf/percpu_freelist.c"
    }
]

And I was hoping to use the location returned in textDocument/definition to textDocument/implementation to extract the name of the struct in some way.

[Anjali05]

Do you mean for this example, line = 66 and character = 32 from arcana at the top?

[HighCommander4]

I was thinking of line = 66 and character = 35 from range.end.

[Anjali05]

@HighCommander4 So I have tried your suggestion on a couple of examples and for most cases (except one that I have encountered so far), sending the character and line of symbol lock from AST goes here (using this as an example :https://elixir.bootlin.com/linux/v6.1/source/fs/exec.c#L1573):

client --> server: {
    "jsonrpc": "2.0",
    "id": 125,
    "method": "textDocument/definition",
    "params": {
        "textDocument": {
            "uri": "file:///mydata/clangd-linux-6.1/fs/exec.c"
        },
        "position": {
            "line": 1572,
            "character": 23
        }
    }
}

I[13:44:28.579] <-- textDocument/definition(125)
I[13:44:28.580] --> reply:textDocument/definition(125) 0 ms
server --> client: {
    "id": 125,
    "jsonrpc": "2.0",
    "result": [
        {
            "range": {
                "end": {
                    "character": 37,
                    "line": 347
                },
                "start": {
                    "character": 28,
                    "line": 347
                }
            },
            "uri": "file:///mydata/clangd-linux-6.1/include/linux/spinlock.h"
        }
    ]
}

This is the AST node for lock in this example:

{
    "arcana": "MemberExpr 0x7f2a0c0e0430 </mydata/clangd-linux-6.1/fs/exec.c:1573:13, col:20> 'spinlock_t':'struct spinlock' lvalue ->lock 0x7f2a0c0e03d0",
    "children": [
        {
            "arcana": "ImplicitCastExpr 0x7f2a0c0e03b0 </mydata/clangd-linux-6.1/fs/exec.c:1573:13, col:16> 'struct fs_struct *' <LValueToRValue>",
            "children": [
                {
                    "arcana": "MemberExpr 0x7f2a0c0e0380 </mydata/clangd-linux-6.1/fs/exec.c:1573:13, col:16> 'struct fs_struct *' lvalue ->fs 0x7f2a3fe7afa8",
                    "children": [
                        {
                            "arcana": "ImplicitCastExpr 0x7f2a0c0e0368 </mydata/clangd-linux-6.1/fs/exec.c:1573:13> 'struct task_struct *' <LValueToRValue>",
                            "children": [
                                {
                                    "arcana": "DeclRefExpr 0x7f2a0c0e0348 </mydata/clangd-linux-6.1/fs/exec.c:1573:13> 'struct task_struct *' lvalue Var 0x7f2a0c0dfa70 'p' 'struct task_struct *'",
                                    "detail": "p",
                                    "kind": "DeclRef",
                                    "range": {
                                        "end": {
                                            "character": 13,
                                            "line": 1572
                                        },
                                        "start": {
                                            "character": 12,
                                            "line": 1572
                                        }
                                    },
                                    "role": "expression"
                                }
                            ],
                            "detail": "LValueToRValue",
                            "kind": "ImplicitCast",
                            "range": {
                                "end": {
                                    "character": 13,
                                    "line": 1572
                                },
                                "start": {
                                    "character": 12,
                                    "line": 1572
                                }
                            },
                            "role": "expression"
                        }
                    ],
                    "detail": "fs",
                    "kind": "Member",
                    "range": {
                        "end": {
                            "character": 17,
                            "line": 1572
                        },
                        "start": {
                            "character": 12,
                            "line": 1572
                        }
                    },
                    "role": "expression"
                }
            ],
            "detail": "LValueToRValue",
            "kind": "ImplicitCast",
            "range": {
                "end": {
                    "character": 17,
                    "line": 1572
                },
                "start": {
                    "character": 12,
                    "line": 1572
                }
            },
            "role": "expression"
        }
    ],
    "detail": "lock",
    "kind": "Member",
    "range": {
        "end": {
            "character": 23,
            "line": 1572
        },
        "start": {
            "character": 12,
            "line": 1572
        }
    },
    "role": "expression"
}

I was wondering if it would be correct to extract the struct name from the second arcana from the top (the first one in the first inner children) and somehow get to the struct definition from there. Getting the definition location for the struct along with the name is also important as I might run into struct with the same name and then it would be difficult to do further analysis on the struct.

[Anjali05]

Adding to this, calling textDocument/definition from the line and col in the second arcana returns this:https://elixir.bootlin.com/linux/v6.1/source/include/linux/sched.h#L1091

client --> server: {
    "jsonrpc": "2.0",
    "id": 125,
    "method": "textDocument/definition",
    "params": {
        "textDocument": {
            "uri": "file:///mydata/clangd-linux-6.1/fs/exec.c"
        },
        "position": {
            "line": 1572,
            "character": 15
        }
    }
}

I[13:57:04.170] <-- textDocument/definition(125)
I[13:57:04.172] --> reply:textDocument/definition(125) 1 ms
server --> client: {
    "id": 125,
    "jsonrpc": "2.0",
    "result": [
        {
            "range": {
                "end": {
                    "character": 22,
                    "line": 1090
                },
                "start": {
                    "character": 20,
                    "line": 1090
                }
            },
            "uri": "file:///mydata/clangd-linux-6.1/include/linux/sched.h"
        }
    ]
}

[HighCommander4]

for most cases (except one that I have encountered so far), sending the character and line of symbol lock from AST goes here [...]

I think I know why. Let's consider this simplified example:

struct lock_t {};

struct foo {
  struct lock_t* lock;
};

void spin_lock(lock_t *);

void bar(struct foo *f) {
  spin_lock(f->lock);
}

If you open this in an editor like vscode, where the cursor is in between characters (as opposed to e.g. vim where the cursor is on a character), and place the cursor between lock and ), clangd needs to disambiguate whether you meant to navigate to the location associated with the lock token, or the location associated with the ) token.

Clangd chooses the token to the right of the cursor first, i.e. the ) token, which is associated with the spin_lock function, so that's where go-to-def takes you.

To navigate to the location associated with lock, you have to put the cursor at least one character earlier (e.g. between c and k).

So, rather than using the end of the range for the MemberExpr representing f->lock, it will probably work better to use one character earlier.

(The case that worked was probably a case where the next token was ; rather than ). ; tokens do not have a go-to-def target, to clangd falls back on the token to the left of the cursor.)