When running in AWS Lambda, I receive an error. Willing to figure it out, but not sure exactly where to start.

[tylerjgarland]

This works in my local dev environment.

Execution code:

    grouped = df.swifter.groupby(['lat_vehicle_categorized', 'lon_vehicle_categorized'])
    val = grouped.apply(sub, ct=call_table, labels=labels, hotspot_thresholds=hotspot_thresholds)

Error message:

[ERROR] SubprocessError: Exception occurred in preexec_fn.
Traceback (most recent call last):
  File "/var/task/app.py", line 155, in handler
    return asyncio.run(handler_async(event, context))
  File "/var/lang/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
    return future.result()
  File "/var/task/app.py", line 141, in handler_async
    final_csv, hzd_csv = excavator.main(
  File "/var/task/egocentric/excavator.py", line 174, in main
    call_tables = compute_hotspot_calltable([df, hotspot_thresholds])
  File "/var/task/utils/hotspot_utils.py", line 141, in compute_hotspot_calltable
    val = grouped.apply(sub, ct=call_table, labels=labels, hotspot_thresholds=hotspot_thresholds)
  File "/var/task/swifter/swifter.py", line 638, in apply
    return self._ray_apply(func, *args, **kwds)
  File "/var/task/swifter/swifter.py", line 625, in _ray_apply
    self._ray_progress_apply(ray_submit_apply, len(chunks)) if self._progress_bar else ray_submit_apply()
  File "/var/task/swifter/swifter.py", line 607, in _ray_submit_apply
    return [self._ray_groupby_apply_chunk.remote(self, ray.put(chunk), func, *args, **kwds) for chunk in chunks]
  File "/var/task/swifter/swifter.py", line 607, in <listcomp>
    return [self._ray_groupby_apply_chunk.remote(self, ray.put(chunk), func, *args, **kwds) for chunk in chunks]
  File "/var/task/ray/_private/auto_init_hook.py", line 17, in auto_init_wrapper
    auto_init_ray()
  File "/var/task/ray/_private/auto_init_hook.py", line 11, in auto_init_ray
    ray.init()
  File "/var/task/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/var/task/ray/_private/worker.py", line 1534, in init
    _global_node = ray._private.node.Node(
  File "/var/task/ray/_private/node.py", line 287, in __init__
    self.start_head_processes()
  File "/var/task/ray/_private/node.py", line 1159, in start_head_processes
    self.start_gcs_server()
  File "/var/task/ray/_private/node.py", line 969, in start_gcs_server
    process_info = ray._private.services.start_gcs_server(
  File "/var/task/ray/_private/services.py", line 1343, in start_gcs_server
    process_info = start_ray_process(
  File "/var/task/ray/_private/services.py", line 891, in start_ray_process
    process = ConsolePopen(
  File "/var/lang/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/var/lang/lib/python3.9/subprocess.py", line 1838, in _execute_child
    raise child_exception_type(err_msg)

[jmcarpenter2]

Hey @tylerjgarland , this is a very interesting issue... it looks like a ray.init() issue, which is the tech swifter uses under-the-hood for groupby-apply. This means that you might wanna jump over to the ray repo to ask about running in a lambda. Quite frankly, i dont know if ray supports running in a lambda... but they would know!

[tylerjgarland]

Hey @jmcarpenter2 , totally a ray issue and I'm familiar with AWS Lambda not liking processes, but I thought using the thread scheduler would solve that problem. All good though. I actually just posted this just in case someone else had the same issue, but I figured it wouldn't be compatible. Thanks!

[tranvuluan]

I have the same issue when I run Lambda with Container, How we can fix it?