Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

udf/v8: memory hard limit is not controllable. #716

Open
yokofly opened this issue May 14, 2024 · 0 comments · May be fixed by #718
Open

udf/v8: memory hard limit is not controllable. #716

yokofly opened this issue May 14, 2024 · 0 comments · May be fixed by #718
Assignees
Labels
bug Something isn't working

Comments

@yokofly
Copy link
Contributor

yokofly commented May 14, 2024

Describe what's wrong
the problem is even if we set the soft limit to a huge value, the v8 default heap hard limit first aborts this query.

Received exception from server (version 1.5.6):
Code: 2531. DB::Exception: Received from localhost:8463. DB::Exception: Current V8 heap size used=722958960 bytes, total=782812824 bytes, javascript_max_memory_bytes=104857600000, exceed the limit=704531541 bytes: While executing AggregatingTransform. (UDF_MEMORY_THRESHOLD_EXCEEDED)



2024.05.14 03:18:40.215237 [ 33 ] {d31d7fc1-f348-44c2-96da-ae8f096c5179} <Error> TCPHandler: Code: 2531. DB::Exception: Current V8 heap size used=722958960 bytes, total=782812824 bytes, javascript_max_memory_bytes=104857600000, exceed the limit=704531541 bytes: While executing AggregatingTransform. (UDF_MEMORY_THRESHOLD_EXCEEDED), Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) @ 0x000000001aa2d3f9 in /usr/bin/proton
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int, bool) @ 0x0000000010106fba in /usr/bin/proton
2. DB::Exception::Exception<unsigned long&, unsigned long&, unsigned long&, unsigned long&>(int, fmt::v8::basic_format_string<char, fmt::v8::type_identity<unsigned long&>::type, fmt::v8::type_identity<unsigned long&>::type, fmt::v8::type_identity<unsigned long&>::type, fmt::v8::type_identity<unsigned long&>::type>, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x000000001a83154d in /usr/bin/proton
3. DB::V8::checkHeapLimit(v8::Isolate*, unsigned long) @ 0x000000001a830b7e in /usr/bin/proton
4. DB::AggregateFunctionJavaScriptAdapter::create(char*) const @ 0x0000000018898a9c in /usr/bin/proton
5. void DB::Aggregator::createAggregateStates<false>(char*&) const @ 0x000000001583e4df in /usr/bin/proton
6. void DB::Aggregator::executeImplBatch<false, false, false, DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true, true>>>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true, true>>>&, DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true, true>>>::State&, DB::Arena*, unsigned long, unsigned long, DB::Aggregator::AggregateFunctionInstruction*, char*) const @ 0x0000000015904fe1 in /usr/bin/proton
7. void DB::Aggregator::executeImpl<DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true, true>>>>(DB::AggregationMethodSerialized<HashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, HashTableGrowerWithPrecalculation<8ul>, Allocator<true, true, true>>>&, DB::Arena*, unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, DB::Aggregator::AggregateFunctionInstruction*, bool, char*) const @ 0x000000001583997e in /usr/bin/proton
8. DB::Aggregator::executeImpl(DB::AggregatedDataVariants&, unsigned long, unsigned long, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, DB::Aggregator::AggregateFunctionInstruction*, bool, char*) const @ 0x00000000157a7f5b in /usr/bin/proton
9. DB::Aggregator::executeOnBlock(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>, unsigned long, unsigned long, DB::AggregatedDataVariants&, std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>&, std::__1::vector<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>, std::__1::allocator<std::__1::vector<DB::IColumn const*, std::__1::allocator<DB::IColumn const*>>>>&, bool&) const @ 0x00000000157aa845 in /usr/bin/proton
10. DB::AggregatingTransform::consume(DB::Chunk) @ 0x0000000016a35366 in /usr/bin/proton
11. DB::AggregatingTransform::work() @ 0x0000000016a3351e in /usr/bin/proton
12. DB::ExecutionThreadContext::executeTask() @ 0x00000000168f6e97 in /usr/bin/proton
13. DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x00000000168e941d in /usr/bin/proton
14. void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long, DB::ExecuteMode)::$_0>(DB::PipelineExecutor::executeImpl(unsigned long, DB::ExecuteMode)::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) @ 0x00000000168eaca7 in /usr/bin/proton
15. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x00000000101a114f in /usr/bin/proton
16. void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x00000000101a3b8e in /usr/bin/proton
17. ? @ 0x00007f56689b9609 in ?
18. __clone @ 0x00007f56688ce353 in ?


@yokofly yokofly added the bug Something isn't working label May 14, 2024
@yokofly yokofly self-assigned this May 14, 2024
@yokofly yokofly linked a pull request May 14, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant