AddressSanitizer: heap-use-after-free on large dataset

[JonasKellerer]

We have tried to use the csv-parser on a large dataset (8 million lines at 9,9 GB). However when looping over all lines and exectue row[column_name].get<std::string>() we get the following error message

`
==245==ERROR: AddressSanitizer: heap-use-after-free on address 0x621003c37248 at pc 0x56492659e7ee bp 0x7ffe476e2f20 sp 0x7ffe476e2f10
READ of size 8 at 0x621003c37248 thread T0
#0 0x56492659e7ed in csv::internals::CSVFieldList::operator[](unsigned long) const /mwe/includes/csv_reader.h:7635
#1 0x56492659f298 in csv::CSVRow::get_field(unsigned long) const /mwe/includes/csv_reader.h:7694
#2 0x56492659ea9d in csv::CSVRow::operator[](unsigned long) const /mwe/includes/csv_reader.h:7656
#3 0x56492659ebea in csv::CSVRow::operator[](std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) const /mwe/includes/csv_reader.h:7672
#4 0x5649265927c2 in getColumn(std::filesystem::__cxx11::path const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) /mwe/src/main.cpp:27
#5 0x564926592eb0 in main /mwe/src/main.cpp:36
#6 0x7f66a36d0d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)
#7 0x7f66a36d0e3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f)
#8 0x564926591dc4 in _start (/mwe/build/csvMWE+0x7dc4)

0x621003c37248 is located 328 bytes inside of 4096-byte region [0x621003c37100,0x621003c38100)
freed by thread T107 here:
#0 0x7f66a3cb722f in operator delete(void*, unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:172
#1 0x5649265be954 in __gnu_cxx::new_allocatorcsv::internals::RawCSVField*::deallocate(csv::internals::RawCSVField**, unsigned long) /usr/include/c++/11/ext/new_allocator.h:145
#2 0x5649265b31d6 in std::allocatorcsv::internals::RawCSVField*::deallocate(csv::internals::RawCSVField**, unsigned long) /usr/include/c++/11/bits/allocator.h:199
#3 0x5649265b31d6 in std::allocator_traits<std::allocatorcsv::internals::RawCSVField* >::deallocate(std::allocatorcsv::internals::RawCSVField*&, csv::internals::RawCSVField**, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:496
#4 0x5649265aa73f in std::_Vector_base<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* >::_M_deallocate(csv::internals::RawCSVField**, unsigned long) /usr/include/c++/11/bits/stl_vector.h:354
#5 0x5649265b0692 in void std::vector<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* >::_M_realloc_insert<csv::internals::RawCSVField* const&>(__gnu_cxx::__normal_iterator<csv::internals::RawCSVField**, std::vector<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* > >, csv::internals::RawCSVField* const&) /usr/include/c++/11/bits/vector.tcc:500
#6 0x5649265a6ef2 in std::vector<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* >::push_back(csv::internals::RawCSVField* const&) /usr/include/c++/11/bits/stl_vector.h:1198
#7 0x56492659e97c in csv::internals::CSVFieldList::allocate() /mwe/includes/csv_reader.h:7640
#8 0x5649265a3b65 in void csv::internals::CSVFieldList::emplace_back<unsigned int, unsigned long&>(unsigned int&&, unsigned long&) /mwe/includes/csv_reader.h:5478
#9 0x564926598700 in csv::internals::IBasicCSVParser::push_field() /mwe/includes/csv_reader.h:6972
#10 0x564926598c01 in csv::internals::IBasicCSVParser::parse() /mwe/includes/csv_reader.h:6999
#11 0x5649265c8b49 in csv::internals::StreamParser<std::basic_ifstream<char, std::char_traits > >::next(unsigned long) /mwe/includes/csv_reader.h:6175
#12 0x56492659ceb9 in csv::CSVReader::read_csv(unsigned long) /mwe/includes/csv_reader.h:7496
#13 0x5649265c9335 in bool std::__invoke_impl<bool, bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long>(std::__invoke_memfun_deref, bool (csv::CSVReader::&&)(unsigned long), csv::CSVReader&&, unsigned long&&) /usr/include/c++/11/bits/invoke.h:74
#14 0x5649265c913e in std::__invoke_result<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long>::type std::__invoke<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long>(bool (csv::CSVReader::&&)(unsigned long), csv::CSVReader&&, unsigned long&&) /usr/include/c++/11/bits/invoke.h:96
#15 0x5649265c905e in bool std::thread::_Invoker<std::tuple<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/11/bits/std_thread.h:253
#16 0x5649265c8ec1 in std::thread::_Invoker<std::tuple<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long> >::operator()() /usr/include/c++/11/bits/std_thread.h:260
#17 0x5649265c8dbd in std::thread::_State_impl<std::thread::_Invoker<std::tuple<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211
#18 0x7f66a3ab22b2 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc2b2)

previously allocated by thread T107 here:
#0 0x7f66a3cb61c7 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:99
#1 0x5649265c49fd in __gnu_cxx::new_allocatorcsv::internals::RawCSVField*::allocate(unsigned long, void const*) /usr/include/c++/11/ext/new_allocator.h:127
#2 0x5649265bd7a6 in std::allocatorcsv::internals::RawCSVField*::allocate(unsigned long) /usr/include/c++/11/bits/allocator.h:185
#3 0x5649265bd7a6 in std::allocator_traits<std::allocatorcsv::internals::RawCSVField* >::allocate(std::allocatorcsv::internals::RawCSVField*&, unsigned long) /usr/include/c++/11/bits/alloc_traits.h:464
#4 0x5649265b779f in std::_Vector_base<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* >::_M_allocate(unsigned long) /usr/include/c++/11/bits/stl_vector.h:346
#5 0x5649265b0514 in void std::vector<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* >::_M_realloc_insert<csv::internals::RawCSVField* const&>(__gnu_cxx::__normal_iterator<csv::internals::RawCSVField**, std::vector<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* > >, csv::internals::RawCSVField* const&) /usr/include/c++/11/bits/vector.tcc:440
#6 0x5649265a6ef2 in std::vector<csv::internals::RawCSVField*, std::allocatorcsv::internals::RawCSVField* >::push_back(csv::internals::RawCSVField* const&) /usr/include/c++/11/bits/stl_vector.h:1198
#7 0x56492659e97c in csv::internals::CSVFieldList::allocate() /mwe/includes/csv_reader.h:7640
#8 0x5649265a3b65 in void csv::internals::CSVFieldList::emplace_back<unsigned int, unsigned long&>(unsigned int&&, unsigned long&) /mwe/includes/csv_reader.h:5478
#9 0x564926598700 in csv::internals::IBasicCSVParser::push_field() /mwe/includes/csv_reader.h:6972
#10 0x564926598c01 in csv::internals::IBasicCSVParser::parse() /mwe/includes/csv_reader.h:6999
#11 0x5649265c8b49 in csv::internals::StreamParser<std::basic_ifstream<char, std::char_traits > >::next(unsigned long) /mwe/includes/csv_reader.h:6175
#12 0x56492659ceb9 in csv::CSVReader::read_csv(unsigned long) /mwe/includes/csv_reader.h:7496
#13 0x5649265c9335 in bool std::__invoke_impl<bool, bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long>(std::__invoke_memfun_deref, bool (csv::CSVReader::&&)(unsigned long), csv::CSVReader&&, unsigned long&&) /usr/include/c++/11/bits/invoke.h:74
#14 0x5649265c913e in std::__invoke_result<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long>::type std::__invoke<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long>(bool (csv::CSVReader::&&)(unsigned long), csv::CSVReader&&, unsigned long&&) /usr/include/c++/11/bits/invoke.h:96
#15 0x5649265c905e in bool std::thread::_Invoker<std::tuple<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) /usr/include/c++/11/bits/std_thread.h:253
#16 0x5649265c8ec1 in std::thread::_Invoker<std::tuple<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long> >::operator()() /usr/include/c++/11/bits/std_thread.h:260
#17 0x5649265c8dbd in std::thread::_State_impl<std::thread::_Invoker<std::tuple<bool (csv::CSVReader::)(unsigned long), csv::CSVReader, unsigned long> > >::_M_run() /usr/include/c++/11/bits/std_thread.h:211
#18 0x7f66a3ab22b2 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc2b2)

Thread T107 created by T0 here:
#0 0x7f66a3c58685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x7f66a3ab2388 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_deletestd::thread::_State >, void (*)()) (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc388)
#2 0x56492659d23d in csv::CSVReader::read_row(csv::CSVRow&) /mwe/includes/csv_reader.h:7536
#3 0x56492659e70a in csv::CSVReader::iterator::operator++() /mwe/includes/csv_reader.h:7605
#4 0x5649265928ad in getColumn(std::filesystem::__cxx11::path const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) /mwe/src/main.cpp:25
#5 0x564926592eb0 in main /mwe/src/main.cpp:36
#6 0x7f66a36d0d8f (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f)

SUMMARY: AddressSanitizer: heap-use-after-free /mwe/includes/csv_reader.h:7635 in csv::internals::CSVFieldList::operator[](unsigned long) const
Shadow bytes around the buggy address:
0x0c428077edf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c428077ee00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c428077ee10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c428077ee20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c428077ee30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c428077ee40: fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd fd
0x0c428077ee50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c428077ee60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c428077ee70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c428077ee80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c428077ee90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
==245==ABORTING
`

The problem can be fixed, when using std::this_thread::sleep_for(std::chrono::nanoseconds(1)); in the same loop.

For reproduceability, I have put a MWE here:
https://drive.google.com/file/d/1M_PJLlhxs8JTmIGEcDNCBAeBqxqmdNBC/view?usp=drive_link

Just extract it and run docker build . --tag=mwe, then docker run -it mwe and inside the container ./runAndBuild.sh.

[vincentlaucsb]

Thanks for your report, I'll take a look