Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpptools crash - The language server crashed. Restarting... #10651

Open
Colengms opened this issue Mar 10, 2023 · 51 comments
Open

cpptools crash - The language server crashed. Restarting... #10651

Colengms opened this issue Mar 10, 2023 · 51 comments
Assignees
Milestone

Comments

@Colengms
Copy link
Collaborator

image

We're very interested in fixing crashes. Information that could help us fix crashes include:

  • Crash stacks.
  • Specific guidance on how to repro a crash.
  • Isolated examples that reproduce a crash and can be shared with us directly.

If you're experiencing a crash, any of the above information could be extremely useful to us to investigate and fix it. Please consider opening a new issue in this repo. Or, add a comment on this issue.

I'm pinning this issue, to make this more obvious to users who might be browsing our repo due to experiencing a crash.

@Colengms Colengms added this to the Backlog milestone Mar 10, 2023
@Colengms Colengms self-assigned this Mar 10, 2023
@Colengms Colengms pinned this issue Mar 10, 2023
@H-G-Hristov
Copy link

Just to remind that on macOS: #9810

@sean-mcmanus
Copy link
Collaborator

sean-mcmanus commented Mar 10, 2023

@H-G-Hristov Yes, but that issue mentions a work around via running lldb -p <processId>, then continue (assume it gets into a stopped state), and then repro the crash, and use bt to get the full stack of the crashing thread (maybe bt all if bt isn't sufficient for some reason).

@sean-mcmanus
Copy link
Collaborator

I've reproed multiple crashes. I'm tracking it with #10636 .

We may not need this issue anymore.

@sean-mcmanus sean-mcmanus unpinned this issue Mar 17, 2023
@bobbrow bobbrow closed this as not planned Won't fix, can't repro, duplicate, stale Apr 25, 2023
@Colengms
Copy link
Collaborator Author

Colengms commented Apr 27, 2023

Reopening. The intent of this issue was to be pinned so users experiencing a crash can find the issue and easily find instructions. I'd like to keep this open and pinned for now.

@sean-mcmanus
Copy link
Collaborator

@Colengms Yeah, that seems fine, but user should know that we don't currently have a widespread crashing issue (as of https://github.com/microsoft/vscode-cpptools/releases/tag/v1.14.5 or later).

@OnielN14
Copy link

OnielN14 commented Aug 9, 2023

I also encountered similar issue.
OS: Windows 11
VS Code version : 1.81.0
C/C++ Extension version: 1.16.3

here is what I found when I check the log in developer tools :
image

ERR [Extension Host] (node:20304) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `Code --trace-deprecation ...` to show where the warning was created)

@OnielN14
Copy link

OnielN14 commented Aug 9, 2023

I also encountered similar issue. OS: Windows 11 VS Code version : 1.81.0 C/C++ Extension version: 1.16.3

here is what I found when I check the log in developer tools : image

ERR [Extension Host] (node:20304) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `Code --trace-deprecation ...` to show where the warning was created)

I just try to manually install the extension into fresh VSCodium v1.81.0.. the extension works

@chris0306
Copy link

image
大佬,这个问题怎么破?

@doxxx
Copy link

doxxx commented Aug 14, 2023

Working with a large Visual Studio solution of ~70 C# projects and one C++/CMake project. Also had the same solution open in Visual Studio at the same time. Performed a Rebuild in Visual Studio and then came back to 10000+ duplicate errors in VS Code like this:

[Error - 9:24:53 AM] Sending notification cpptools/fileDeleted failed.
Error: Cannot call write after a stream was destroyed
	at new NodeError (node:internal/errors:387:5)
	at _write (node:internal/streams/writable:323:11)
	at Writable.write (node:internal/streams/writable:336:10)
	at c:\Users\gordon.tyler\.vscode\extensions\ms-vscode.cpptools-1.16.3-win32-x64\dist\main.js:77222:29
	at new Promise (<anonymous>)
	at WritableStreamWrapper.write (c:\Users\gordon.tyler\.vscode\extensions\ms-vscode.cpptools-1.16.3-win32-x64\dist\main.js:77212:16)
	at StreamMessageWriter.doWrite (c:\Users\gordon.tyler\.vscode\extensions\ms-vscode.cpptools-1.16.3-win32-x64\dist\main.js:76424:33)
	at c:\Users\gordon.tyler\.vscode\extensions\ms-vscode.cpptools-1.16.3-win32-x64\dist\main.js:76415:29
	at runNextTicks (node:internal/process/task_queues:61:5)
	at process.processImmediate (node:internal/timers:437:9)

@sean-mcmanus
Copy link
Collaborator

@OnielN14 @chris0306 @doxxx That info is insufficient. We more info, particular the info that's described in the description for this issue, such as a call stack of cpptools or a repro.

@sean-mcmanus
Copy link
Collaborator

@OnielN14 @chris0306 @doxxx Also, 1.17.1 and 1.17.2 of our extension has some crash fixes, so that may be worth trying.

@vineetsoni
Copy link

vineetsoni commented Aug 17, 2023

I start to have the same issue since the extension auto-updated to 1.17.3 today. I'm running AlmaLinux 8.8 with kernel 4.18.0-477.15.1.el8_8.x86_64. The VSCode version is 1.81

Not sure if this is any useful, but the output message is:

[Error - 2:14:08 PM] The language server crashed. Restarting...
[Error - 2:14:08 PM] Sending document notification textDocument/didOpen failed
Error: Attempting to use languageClient before initialized
	at DefaultClient.get languageClient [as languageClient] (/u/vinson3z/.vscode-server/extensions/ms-vscode.cpptools-1.17.3-linux-x64/dist/main.js:47283:19)
	at DefaultClient.sendDidOpen (/u/vinson3z/.vscode-server/extensions/ms-vscode.cpptools-1.17.3-linux-x64/dist/main.js:48450:20)
	at DefaultClient.takeOwnership (/u/vinson3z/.vscode-server/extensions/ms-vscode.cpptools-1.17.3-linux-x64/dist/main.js:48439:20)
	at processDelayedDidOpen (/u/vinson3z/.vscode-server/extensions/ms-vscode.cpptools-1.17.3-linux-x64/dist/main.js:53325:30)
	at runMicrotasks (<anonymous>)
	at processTicksAndRejections (node:internal/process/task_queues:96:5)
	at async Function.dispatch (/u/vinson3z/.vscode-server/extensions/ms-vscode.cpptools-1.17.3-linux-x64/dist/main.js:48476:44)

Is there any known workaround for this problem?

Fixed the problem by downgrading to 1.16.3. Downgrading from 1.17.3 to 1.17.2 didn't work.

PS: This happens only when I try to use VSCode with a very large project. For a medium-sized project, it works fine.

@bobbrow
Copy link
Member

bobbrow commented Aug 17, 2023

@vineetsoni it looks like the extension isn't recovering from a crash properly. There were some code changes in this area recently, so thank you for reporting it. I'll move your report into a new issue.

@rganascim
Copy link

Environment

OS and version: macOS 14.1.1 23B81 arm64
VS Code: 1.84.2
C/C++ extension: v1.18.2

Issue description:

With larger projects, the language server continuously crashes. It's crashing too fast, that I need an external script to collect the lldb info while the cpptools still executes.

Errors suddenly appeared 2 days ago when I open VSCode. I was using in the same project without any problems. I already did the intellisense cache clear, and even tried to unistall all extensions, reinstall visual studio, change the cpptools versions. But I'm always getting this crash.

crash_stack_cpptools_macos.txt

@comio
Copy link

comio commented Nov 18, 2023

Same error on 1.19.0. The logs look clean.

@MArKy1998
Copy link

MArKy1998 commented Nov 20, 2023 via email

@bobbrow
Copy link
Member

bobbrow commented Nov 20, 2023

Pasting the stack from @rganascim so we don't miss it.

* thread #9, stop reason = signal SIGABRT
  * frame #0: 0x00000001842b911c libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x00000001842f0cc0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x0000000184200a40 libsystem_c.dylib`abort + 180
    frame #3: 0x00000001842a86d8 libc++abi.dylib`abort_message + 132
    frame #4: 0x00000001842986a0 libc++abi.dylib`demangling_terminate_handler() + 52
    frame #5: 0x00000001842a7a9c libc++abi.dylib`std::__terminate(void (*)()) + 16
    frame #6: 0x00000001842a7a2c libc++abi.dylib`std::terminate() + 36
    frame #7: 0x00000001842aac5c libc++abi.dylib`__cxa_rethrow + 152
    frame #8: 0x0000000102d08580 cpptools`msvc::path_utf8_iterator::operator++() + 72
    frame #9: 0x00000001023665a4 cpptools`cpp_properties::parse_includes(std::__1::vector<include_path, std::__1::allocator<include_path>> const&, std::__1::vector<include_path, std::__1::allocator<include_path>>&, bool, std::__1::shared_ptr<browse_include_paths> const&) const + 4520
    frame #10: 0x00000001023600ac cpptools`cpp_properties::resolve_includes(compilation_args&) const + 144
    frame #11: 0x0000000102363d94 cpptools`cpp_properties::resolve_current_config(compilation_args&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, bool) const + 1320
    frame #12: 0x00000001023630dc cpptools`cpp_properties::provide_args(char const*, bool) const + 2044
    frame #13: 0x00000001024b2bac cpptools`create_sync_work(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, char const*, unsigned long long, std::__1::shared_ptr<browse_engine> const&, std::__1::shared_ptr<cpp_properties> const&, bool, bool, bool) + 1716
    frame #14: 0x00000001024b46d8 cpptools`create_async_work(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, char const*, unsigned long long, std::__1::shared_ptr<browse_engine> const&, std::__1::shared_ptr<cpp_properties> const&, bool) + 100
    frame #15: 0x00000001024bb1f0 cpptools`std::__1::__function::__func<void thread_pool::enqueue<intellisense_client_factory::create_async(char const*, char const*, std::__1::shared_ptr<browse_engine> const&, std::__1::shared_ptr<cpp_properties> const&, bool, std::__1::function<void (std::__1::vector<std::__1::shared_ptr<intellisense_client>, std::__1::allocator<std::__1::shared_ptr<intellisense_client>>>)>&&, std::__1::function<void ()>&&)::$_4, void>(intellisense_client_factory::create_async(char const*, char const*, std::__1::shared_ptr<browse_engine> const&, std::__1::shared_ptr<cpp_properties> const&, bool, std::__1::function<void (std::__1::vector<std::__1::shared_ptr<intellisense_client>, std::__1::allocator<std::__1::shared_ptr<intellisense_client>>>)>&&, std::__1::function<void ()>&&)::$_4&&, std::__1::future<void>*)::'lambda'(), std::__1::allocator<void thread_pool::enqueue<intellisense_client_factory::create_async(char const*, char const*, std::__1::shared_ptr<browse_engine> const&, std::__1::shared_ptr<cpp_properties> const&, bool, std::__1::function<void (std::__1::vector<std::__1::shared_ptr<intellisense_client>, std::__1::allocator<std::__1::shared_ptr<intellisense_client>>>)>&&, std::__1::function<void ()>&&)::$_4, void>(intellisense_client_factory::create_async(char const*, char const*, std::__1::shared_ptr<browse_engine> const&, std::__1::shared_ptr<cpp_properties> const&, bool, std::__1::function<void (std::__1::vector<std::__1::shared_ptr<intellisense_client>, std::__1::allocator<std::__1::shared_ptr<intellisense_client>>>)>&&, std::__1::function<void ()>&&)::$_4&&, std::__1::future<void>*)::'lambda'()>, void ()>::operator()() + 60
    frame #16: 0x00000001024eb944 cpptools`thread_pool::do_work(unsigned long) + 772
    frame #17: 0x0000000102d1c9a0 cpptools`msvc::thread_helper_t::thread_entry(void*) + 32
    frame #18: 0x00000001842f1034 libsystem_pthread.dylib`_pthread_start + 136

@sean-mcmanus
Copy link
Collaborator

sean-mcmanus commented Nov 20, 2023

@bobbrow This is already fixed in 1.19.0 -- #11674 .

@sean-mcmanus
Copy link
Collaborator

@comio Is the issue you're seeing fixed with 1.19.0? If not, it's a different issue and we'd need more repro info.

@sean-mcmanus
Copy link
Collaborator

@bobbrow If this is a frequent crash, we may want to port it to a 1.18.6 update, but my belief is that it was not a regression, unless something in 1.18.5 is causing it to repro more often.

@MArKy1998
Copy link

MArKy1998 commented Nov 21, 2023 via email

@borjamunozf
Copy link

Sadly, this is a showstopper for a full team of developers in our company. They have transitioned from QtCreator to VScode for C++ and the experience has been really bad for production grade projects.

I have tried to attach to the C++ extension, but it does not show any crash or stops anywhere.

Tested with:

  • VScode 1.84.2
  • Cpp 1.18.5

I also removed .vscode-server && .vscode folder, installed again extensions.

@rganascim
Copy link

I encountered an escalation of issues when the system build was located within the source code folder, particularly when compiling gRPC libraries. By relocating the build ecosystem outside of the folder, the problems had diminished.

You could try it.

*I don't have more troubles with C++ extension

@bobbrow
Copy link
Member

bobbrow commented Nov 28, 2023

@sean-mcmanus I'm supportive of a 1.18.6 release containing that patch if we can confirm that's the cause of these recent problems.

@borjamunozf would you be willing to try 1.19.1 and see if that reduces the crashes you're seeing?

@borjamunozf
Copy link

borjamunozf commented Nov 28, 2023

@sean-mcmanus I'm supportive of a 1.18.6 release containing that patch if we can confirm that's the cause of these recent problems.

@borjamunozf would you be willing to try 1.19.1 and see if that reduces the crashes you're seeing?

Yes, absolutely!

@sean-mcmanus
Copy link
Collaborator

@borjamunozf This "issue" is just a generic issue tracking crashes -- there could be many potential causes, but we need a repro or call stack of the crash. What OS are you using? What process are you attaching to?

@sean-mcmanus
Copy link
Collaborator

@rganascim What do you mean by "escalation of issues"? Crashes? I don't repro a crash in that scenario. Putting your build folder in the workspace folder will trigger didChange messages that will cause IntelliSense updates, so it's not recommended to put your build folder in the workspace folder (for performance reasons).

@bobbrow We haven't identified the source of 1.18.x crashes yet.

@borjamunozf
Copy link

@borjamunozf This "issue" is just a generic issue tracking crashes -- there could be many potential causes, but we need a repro or call stack of the crash. What OS are you using? What process are you attaching to?

Snippet of Developer Tools console after crash:

imagen

I cannot reproduce the crash with the Debugger. These are the processes I can see; if i attach to the first one it shows this in Debug Console:

New LWP 228881]
[New LWP 228882]
[New LWP 228883]
[New LWP 233149]
0x00007f09b7d500a9 in __stdio_read ()

If I attach to the second one in shows:

[Detaching after fork from child process 553242]
[LWP 540987 exited]
[LWP 540986 exited]
[LWP 540985 exited]
[LWP 540983 exited]
[LWP 540982 exited]
[LWP 540981 exited]
[LWP 540980 exited]
[LWP 540984 exited]
[New process 540980]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
Debugger has disconnected from the program '/home/borjamf/.vscode-server/extensions/ms-vscode.cpptools-1.18.5-linux-x64/bin/cpptools'.

imagen

We're using WSL2 - Ubuntu 22.04: 1.2.5 WSL and 2.0.9

@borjamunozf
Copy link

Update:

  • Seems that when we remove or rename the file

c_cpp_properties.json, which it contains

{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${default}"
            ],
            "defines": [],
            "compilerPath": "/usr/bin/g++",
            "cStandard": "c17",
            "cppStandard": "c++14",
            "intelliSenseMode": "linux-gcc-x64",
            "compileCommands": "${workspaceFolder}/build/Debug/compile_commands.json",
            "configurationProvider": "ms-vscode.cmake-tools"
        }
    ],
    "version": 4
}

it's not failing. The easiest to reproduce the failure in our case is to execute Show Call Hierarchy: with the custom c_cpp_properties it ends usually with Crash, without it it works (or it's not failing yet)

@bobbrow
Copy link
Member

bobbrow commented Nov 29, 2023

@rganascim What do you mean by "escalation of issues"? Crashes? I don't repro a crash in that scenario. Putting your build folder in the workspace folder will trigger didChange messages that will cause IntelliSense updates, so it's not recommended to put your build folder in the workspace folder (for performance reasons).

@sean-mcmanus it's extremely common to have the build folder in the workspace (e.g. CMake Tools and others do this by default). It is also very common to have a .git folder in the workspace and we don't filter those messages either. I wanted to reduce the noise with #11493, but I thought it was decided that since we filtered these in the message handler, it was ok.

@bobbrow
Copy link
Member

bobbrow commented Nov 29, 2023

@borjamunozf were you able to try 1.19.1 yet? Your screenshots suggest you are attempting to attach to 1.18.5. Do you happen to be working on anything open source that we can clone and test locally?

@bobbrow
Copy link
Member

bobbrow commented Nov 29, 2023

@borjamunozf, also I see that you opened microsoft/vscode-cmake-tools#3469. Not sure if these issues are related but I'm guessing you have a pretty big compile_commands.json file. Would you have time this week or next week to jump on a video call and walk me through the crash(es)?

@borjamunozf
Copy link

borjamunozf commented Nov 29, 2023

@borjamunozf were you able to try 1.19.1 yet? Your screenshots suggest you are attempting to attach to 1.18.5. Do you happen to be working on anything open source that we can clone and test locally?

Same. Attach to process, 2-3 minutes showing lot of messages

[Detaching after fork from child process 1163080]
[Detaching after fork from child process 1163081]
[Detaching after fork from child process 1163082]
[Detaching after fork from child process 1163083]
[Detaching after fork from child process 1163084]
[Detaching after fork from child process 1163085]
[Detaching after fork from child process 1163086]
[Detaching after fork from child process 1163087]

and finally the crash.

Sadly, we work on propietary code. It's all code from my company, complex and huge projects I guess.

imagen

@borjamunozf, also I see that you opened microsoft/vscode-cmake-tools#3469. Not sure if these issues are related but I'm guessing you have a pretty big compile_commands.json file. Would you have time this week or next week to jump on a video call and walk me through the crash(es)?

It's not the same project, but yes, we have big compile_commands.json for all projects I have seen in my division. In that issue we have found that disabling the loadCompileCommands at least allows to run CMake Configure & build the project, but it's only a workaround. Just as reference, for this project the compile_commands.json is 47.56MB size., ~83000 lines

In this project, as I mentioned above, removing or renaming c_cpp_properties seems to avoid the crash, but the IntelliSense misses stuff, of course. Also if comment out the "compileCommands" only it seems to not crash, or not crash so fast, but it also does not work any Show Call Hierarchy, for example.

About the video call, that would be great. However I have to check given the nature of the code. Will let you know.

@Colengms
Copy link
Collaborator Author

Hi @borjamunozf . I see a number of clues in the information you've posted so far.

In one of your images, I see that the IntelliSense process failed to be restarted. That would imply either an issue with our restart logic, or that something is immediately in a bad state when it starts back up, such as some anomolous configuration information.

The fact you were unable to detect a process crash from the debugger suggests that perhaps the process isn't actually crashing, but being terminated by another process. We've seen symptoms like this in the past with overzealous anti-malware software killing our process. To my knowledge, we do not have an influx of users reporting these symptoms on Linux, so something specific to your environment could potentially be involved.

Your C/C++ configuration refers to both use of CMake Tools as a configurationProvider, and a compile_commands.json file. Presumably, the compile_commands.json file is one being generated by CMake. This is superfluous, as use of CMake Tools as a custom configuration provider would provide configurations for all of the same files as in that compile_commands.json file. I'd suggest removing the compileCommands field from your config. (If that addresses the issue, could you help us investigate further, in case fixing this could help other compileCommands users?)

One of the most useful sources of information for investigating issues with the C/C++ extension is its debug output. Could you enable the setting "C_Cpp.loggingLevel": "Debug", monitor the C/C++ output channel leading up to the repro, and provide us with any pertinent content? Also, could you open a new issue here to track your specific repro, as this issue is more general and crashes often do not have the same cause.

I'm hoping there is something in your log output immediately before the issue that might point to what's going on here, or at least what it was in the process of doing when the problem started to occur.

Also if comment out the "compileCommands" only it seems to not crash, or not crash so fast, but it also does not work any Show Call Hierarchy, for example.

Given that this setting seems superfluous in your case, I would not expect any difference in behavior from removing it. Though, when using a compile_commands.json file, the C/C++ extension will analyze all of those command lines on startup (mainly to build a list of 'browse' paths to populate the browse database with), whereas it would only analyze those command lines as they are needed and provided for a file to configure IntelliSense for it. (Custom configuration providers provide browse paths up front, so all command lines do not need to be parsed in that scenario.) My guess would be that there is a failure parsing one of your command lines. Debug log output may refer to which one.

@sean-mcmanus
Copy link
Collaborator

@bobbrow fileChanged/Created/Deleted triggers an IntelliSense update unless the file or the file's folder is filtered from a files.exclude setting. So sure you can put the cmake build folder in your workspace folder, but you should also add it to the C_Cpp.files.exclude to avoid unnecessary IntelliSense updating.

@bobbrow
Copy link
Member

bobbrow commented Nov 30, 2023

@bobbrow fileChanged/Created/Deleted triggers an IntelliSense update unless the file or the file's folder is filtered from a files.exclude setting. So sure you can put the cmake build folder in your workspace folder, but you should also add it to the C_Cpp.files.exclude to avoid unnecessary IntelliSense updating.

Can we change that behavior? If it's obviously not a C or C++ file, I don't see why we would want to trigger an IntelliSense update. We have filters for non-C/C++ files (based on file extension) in the TypeScript code so it doesn't seem wrong to do the same thing in the language server.

@Zes-MinKey-Young
Copy link

How to get a call stack? I cannot attach a debugger because the crash arises before that. If I open cpptools.exe outside VS Code, I cannot reproduce the crash. The executable does nothing.

@sean-mcmanus
Copy link
Collaborator

@SemenAntonov Your issue is with the clangd extension and not our extension.

@godlikeue
Copy link

Please tell me which previous version did not have this issue.

  1. I install a clean Ubuntu system on a new computer.
  2. sudo snap install --classic code
  3. install cpptools in extensions

Open a cpp file , this issue will occur!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests