You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a 36GB entries file and I tried to run ~/kythe-v0.0.63/tools/write_tables --entries entries --out serving/ -experimental_beam_pipeline -experimental_beam_columnar_data -num_workers 20 to get its serving table in beam mode. However, ~200GB RAM seems not enough. I did an experiment showing ~200GB RAM barely survived from ~13GB entries file. Is this expected? ~200GB RAM works with write_tables without beam but some links in serving table are missing in this case. Is there any way to limit the RAM usage for write_tables in beam mode? I saw statsK/beamK is set to 128 so the max RAM usage should also be 128GB? Am I missing or misunderstanding something? Any feedback is appreciated.
The text was updated successfully, but these errors were encountered:
Hello,
I have a 36GB entries file and I tried to run
~/kythe-v0.0.63/tools/write_tables --entries entries --out serving/ -experimental_beam_pipeline -experimental_beam_columnar_data -num_workers 20to get its serving table in beam mode. However, ~200GB RAM seems not enough. I did an experiment showing ~200GB RAM barely survived from ~13GB entries file. Is this expected? ~200GB RAM works withwrite_tableswithout beam but some links in serving table are missing in this case. Is there any way to limit the RAM usage forwrite_tablesin beam mode? I saw statsK/beamK is set to 128 so the max RAM usage should also be 128GB? Am I missing or misunderstanding something? Any feedback is appreciated.The text was updated successfully, but these errors were encountered: