Segfaults in FTP downloads since 8.6.0

[jeroen]

Users of the R bindings are getting segfaults in ftp downloads after MacOS updated from libcurl 8.4.0 to libcurl 8.6.0.

We have observed the problem also in libcurl 7.8.1 but not yet been able to narrow it down to a smaller reproducer. It happens when the FTP download is performed in a multi-handle that already performed previous requests (possibly http/2) which seem to have corrupted the multi-handle, making it crash in the ftp download.

I will try to investigate more later, but wanted to open this now in case other people have observed similar issues.

curl/libcurl version

libcurl 8.7.1

operating system

MacOS

[jeroen]

Having a hard time getting a more minimal example, but here is the backtrace on Debian with libcurl 8.7.1. So strangely it crashes in http2_data_done even though this is an ftp download.

Thread 1 "R" received signal SIGSEGV, Segmentation fault.
chunk_list_free (anchor=0x55555a0b2141) at ./debian/build/lib/bufq.c:152
Download failed: Invalid argument.  Continuing without source file ./debian/build/lib/./debian/build/lib/bufq.c.
152     ./debian/build/lib/bufq.c: No such file or directory.
(gdb) bt
#0  chunk_list_free (anchor=0x55555a0b2141) at ./debian/build/lib/bufq.c:152
#1  Curl_bufq_free (q=q@entry=0x55555a0b2141) at ./debian/build/lib/bufq.c:244
#2  0x00007fffd1b7b859 in http2_data_done (cf=cf@entry=0x55555c25d620, data=data@entry=0x55555c2839d0) at ./debian/build/lib/http2.c:314
#3  0x00007fffd1b7ba2b in cf_h2_cntrl (cf=0x55555c25d620, data=0x55555c2839d0, event=<optimized out>, arg1=<optimized out>, arg2=<optimized out>) at ./debian/build/lib/http2.c:2485
#4  0x00007fffd1b56ae1 in Curl_conn_cf_cntrl (cf=0x55555c25d620, data=data@entry=0x55555c2839d0, ignore_result=ignore_result@entry=true, event=event@entry=2, arg1=arg1@entry=0, arg2=arg2@entry=0x0) at ./debian/build/lib/cfilters.c:509
#5  0x00007fffd1b56cac in cf_cntrl_all (arg2=0x0, arg1=0, event=2, ignore_result=true, data=0x55555c2839d0, conn=0x55555670b890) at ./debian/build/lib/cfilters.c:559
#6  Curl_conn_ev_data_detach (conn=conn@entry=0x55555670b890, data=data@entry=0x55555c2839d0) at ./debian/build/lib/cfilters.c:576
#7  0x00007fffd1b90c51 in Curl_detach_connection (data=data@entry=0x55555c2839d0) at ./debian/build/lib/multi.c:988
#8  0x00007fffd1bb4b4e in extract_if_dead (conn=conn@entry=0x55555670b890, data=0x55555c2839d0) at ./debian/build/lib/url.c:821
#9  0x00007fffd1bb4bbb in call_extract_if_dead (data=<optimized out>, conn=0x55555670b890, param=0x7fffffff78b0) at ./debian/build/lib/url.c:847
#10 0x00007fffd1b57b2d in Curl_conncache_foreach (data=data@entry=0x55555c2839d0, connc=<optimized out>, param=param@entry=0x7fffffff78b0, func=func@entry=0x7fffd1bb4ba0 <call_extract_if_dead>) at ./debian/build/lib/conncache.c:332
#11 0x00007fffd1bb7938 in prune_dead_connections (data=0x55555c2839d0) at ./debian/build/lib/url.c:876
#12 create_conn (async=0x7fffffff7934, in_connect=<synthetic pointer>, data=0x55555c2839d0) at ./debian/build/lib/url.c:3615
#13 Curl_connect (data=data@entry=0x55555c2839d0, asyncp=asyncp@entry=0x7fffffff7994, protocol_done=protocol_done@entry=0x7fffffff7993) at ./debian/build/lib/url.c:3846
#14 0x00007fffd1b93627 in multi_runsingle (multi=multi@entry=0x555556431b10, nowp=nowp@entry=0x7fffffff7a10, data=data@entry=0x55555c2839d0) at ./debian/build/lib/multi.c:1930
#15 0x00007fffd1b94696 in curl_multi_perform (multi=0x555556431b10, running_handles=running_handles@entry=0x7fffffff7b10) at ./debian/build/lib/multi.c:2704

[icing]

@jeroen this is supposed to be fixed via c6655f7, where we disconnected the pointers from the easy handle to the HTTP/2 internal stream state. The problem was that when the same easy handle was used for different protocols, the HTTP/2 implementation thought it needed to clean up state that did not belong to it.

Could you try running curl 8.8.0 in your setup?

[jeroen]

I'll wait for @samueloph to update 8.8.0 on Debian. Sadly the reproducer for my setup is quite complex because it mixes R packages that link against libcurl with other R packages that link against other debian libs which transitively also link against libcurl. So i don't think I can test this with curl 8.8.0 without rebuilding all those stacks (I tried above and got even weirder errors).

[samueloph]

@jeroen that should happen later today once I finish work

[jeroen]

@icing so sadly MacOS has shipped 8.6.0 and might not update for a while. What would be a workaround in my bindings for this problem? Would it be a good idea to set CURLOPT_FORBID_REUSE for all ftp connections?

[icing]

@icing so sadly MacOS has shipped 8.6.0 and might not update for a while. What would be a workaround in my bindings for this problem? Would it be a good idea to set CURLOPT_FORBID_REUSE for all ftp connections?

That would be harsh. I believe it is cheaper to not reuse the easy handles or maybe to not reuse them after a transfer failed. I do not know how that can be mapped into your bindings, though.

Update: after a glimpse at your documentation, it seems the easy handles are not exposed in your API. That should allow you to curl_easy_cleanup() it on a failed transfer and make a new one?

[jeroen]

Ahh ok now I understand why I had a hard time to find a reproducer. This framework the package uses indeed recycles easy handles for multiple requests, something I recommend against. So that reduces the problem impact.

Actually I do expose the easy handle, they are just called "handles": https://cran.r-project.org/web/packages/curl/vignettes/intro.html#Setting_handle_options

I think it would be a breaking change to automatically call curl_easy_cleanup because some folks may be deliberately re-using the easy-handle to keep the cookies. But perhaps I can do it for ftp connections only, which don't have cookies anyway, if that mitigates some of the problem.

[samueloph]

@jeroen 8.8.0 has been built and will be picked up by the repositories within the next 6 hours.

[jeroen]

I confirm that this crash no longer appears with curl 8.8.0 on Debian.