Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] mars storage fetct data key error #3175

Open
chaokunyang opened this issue Jun 29, 2022 · 0 comments
Open

[BUG] mars storage fetct data key error #3175

chaokunyang opened this issue Jun 29, 2022 · 0 comments

Comments

@chaokunyang
Copy link
Contributor

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
To help us reproducing this bug, please provide information below:

  1. Your Python version
  2. The version of Mars you use
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/brain_predict/lib/sql_laundry.py", line 140, in download_marsdata_from_odps
    result_df = mars_o.to_mars_dataframe(tmp_table_name, chunk_size=chunk_size,).execute()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/core/entity/tileables.py", line 462, in execute
    result = self.data.execute(session=session, **kw)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/core/entity/executable.py", line 142, in execute
    return execute(self, session=session, **kw)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1864, in execute
    **kwargs,
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1653, in execute
    timeout=self._isolated_session.timeout
  File "/usr/local/python3/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/python3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1838, in _execute
    await execution_info
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 107, in wait
    return await self._aio_task
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 957, in _run_in_background
    raise task_result.error.with_traceback(task_result.traceback)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 359, in run
    await self._process_stage_chunk_graph(*stage_args)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 241, in _process_stage_chunk_graph
    stage_id, subtask_graph, chunk_graph, tile_context
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/executor.py", line 250, in execute_subtask_graph
    return await stage_processor.run()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/stage.py", line 252, in run
    return await self._get_stage_result()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/stage.py", line 136, in _get_stage_result
    await self._update_result_meta(execution_chunk_results)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/task/execution/mars/stage.py", line 302, in _update_result_meta
    worker_metas = await asyncio.gather(*coros)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/batch.py", line 153, in _async_batch
    return await self.batch_func(args_list, kwargs_list)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/meta/api/oscar.py", line 182, in batch_get_chunk_meta
    return await self._meta_store.get_meta.batch(*get_chunk_metas)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 196, in send
    return self._process_result_message(result)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 76, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 677, in send
    result = await self._run_coro(message.message_id, coro)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 370, in _run_coro
    return await coro
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 120, in __on_receive__
    return await super().__on_receive__(message)
  File "mars/oscar/core.pyx", line 510, in __on_receive__
  File "mars/oscar/core.pyx", line 490, in mars.oscar.core._BaseActor.__on_receive__
  File "mars/oscar/core.pyx", line 491, in mars.oscar.core._BaseActor.__on_receive__
  File "mars/oscar/core.pyx", line 499, in mars.oscar.core._BaseActor.__on_receive__
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/oscar/batch.py", line 153, in _async_batch
    return await self.batch_func(args_list, kwargs_list)
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/meta/store/dictionary.py", line 101, in batch_get_meta
    metas.append(self._get_meta(*args, **kwargs))
  File "/home/admin/ray-pack/tmp/job/4c060080/pyenv/lib/python3.7/site-packages/mars/services/meta/store/dictionary.py", line 80, in _get_meta
    meta = self._store[object_id]
KeyError: [address=ray://ray-cluster-1656468237/4/0, pid=483612] 'f955907f4409be5cb344abc1b1cec01b_0'
  1. Minimized code to reproduce the error.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant