Meaning of parameters in 'configuration.toml' #21

wenlinyao · 2024-01-26T21:46:15Z

Hi,

In the 'configuration.toml' file, I see a range of parameters, but not sure what those parameters control. Could you please provide one example config file that can produce the result in Table 1 and Table 2 of the paper? One example for each table will be great! Thank you!

mrT23 · 2024-01-27T09:41:57Z

The table shows the results of running:
https://github.com/Codium-ai/AlphaCodium?tab=readme-ov-file#solving-the-entire-dataset

With:

[dataset]
num_iterations=5

Beware - this is quite a long and expensive run...

To maximize the score, add some "randomness" between different iterations, meaning choose somewhat different params for the stages in different iterations, to get the best possible chances for solving.

I am not going to share at this point my full "recipe" for that, as it is just an ensemble detail, which is highly uncommon to share in academic implementations. even without adding randomness, you should get results close to that.

wenlinyao · 2024-01-29T19:59:21Z

Thanks for your reply!

When I run it with more iterations, I sometimes can see one error message as follows. Will it create problems in evaluation?

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-11476' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is cl
osed')>
Traceback (most recent call last):
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose
await proxy.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose
await self._pool.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
await connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
await self._connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose
await self._network_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose
await self.transport_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose
self._transport.close()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
self._check_closed()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

mrT23 · 2024-01-30T06:13:44Z

can you reproduce this error ? does it happen on specific problems ? does it happen always ?

wenlinyao · 2024-01-30T08:15:54Z

Yeah, this error is quite random. The output is something like this:

2024-01-29 23:29:54.595 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:29:54.600 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:29:54.601 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
2024-01-29 23:30:00.540 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:00.540 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------
2024-01-29 23:30:00.606 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kSuccess, passed=False
2024-01-29 23:30:00.607 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - =====================================================================
2024-01-29 23:30:00.607 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 2 attempts, d: 8.0, best_d so far: 5
2024-01-29 23:30:00.607 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:30:00.613 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:30:00.613 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-24504' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is cl
osed')>
Traceback (most recent call last):
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose
await proxy.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose
await self._pool.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
await connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
await self._connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose
await self._network_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose
await self.transport_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose
self._transport.close()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
self._check_closed()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
ERROR:asyncio:Task exception was never retrieved
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-24506' coro=<AsyncClient.aclose() done, defined at /data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_client.py", line 2006, in aclose
await proxy.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpx/_transports/default.py", line 383, in aclose
await self._pool.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
await connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
await self._connection.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_async/http11.py", line 253, in aclose
await self._network_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/httpcore/_backends/anyio.py", line 54, in aclose
await self._stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/streams/tls.py", line 202, in aclose
await self.transport_stream.aclose()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1181, in aclose
self._transport.close()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 751, in call_soon
self._check_closed()
File "/data1/wenlinyao/miniconda3/envs/alphacode/lib/python3.9/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2024-01-29 23:30:06.222 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:06.222 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------
2024-01-29 23:30:06.290 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kSuccess, passed=False
2024-01-29 23:30:06.290 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - =====================================================================
2024-01-29 23:30:06.290 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 3 attempts, d: 3.0, best_d so far: 5
2024-01-29 23:30:06.291 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:30:06.297 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:30:06.297 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
2024-01-29 23:30:12.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:12.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------
2024-01-29 23:30:12.524 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:111 - test-0 :: status=ProgramStatus.kFailed, passed=False
2024-01-29 23:30:12.525 | INFO | alpha_codium.code_contests.eval.code_test_runners:print_test_results:116 - =====================================================================
2024-01-29 23:30:12.525 | ERROR | alpha_codium.gen.stages.run_tests:run_tests:36 - failed to run solution
2024-01-29 23:30:12.525 | INFO | alpha_codium.gen.stages.run_initial_code_generation:run_initial_code_generation:61 - Failed to pass tests after 4 attempts, d: 5, best_d so far: 3.0
2024-01-29 23:30:12.526 | INFO | alpha_codium.gen.stages.run_initial_solve:run_initial_solve:14 - --initial solve stage--
2024-01-29 23:30:12.531 | INFO | alpha_codium.llm.ai_handler:chat_completion:86 - -----------------
2024-01-29 23:30:12.532 | INFO | alpha_codium.llm.ai_handler:chat_completion:87 - Running inference ...
2024-01-29 23:30:19.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:133 - done
2024-01-29 23:30:19.458 | INFO | alpha_codium.llm.ai_handler:chat_completion:134 - -----------------

Another interesting thing I found is that when I run gpt-3.5-turbo-0613 on the valid split (iterations=1) twice, the total amount of data instances after evaluation is quite different. One is 9+87=96 instances and another one is 12+ 80=92. I checked the total amount should be 177? Is this due to the above error?

First time run:
problem 8 passed all tests
problem 17 passed all tests
problem 27 passed all tests
problem 31 passed all tests
problem 60 passed all tests
problem 66 passed all tests
problem 73 passed all tests
problem 84 passed all tests
problem 96 passed all tests
total_passed: 9, total_failed: 87
pass rate: 0.09375

Second time run:
problem 4 passed all tests
problem 8 passed all tests
problem 16 passed all tests
problem 17 passed all tests
problem 34 passed all tests
problem 50 passed all tests
problem 60 passed all tests
problem 66 passed all tests
problem 73 passed all tests
problem 83 passed all tests
problem 96 passed all tests
problem 114 passed all tests
total_passed: 12, total_failed: 80
pass rate: 0.13043478260869565

mrT23 · 2024-01-30T19:48:49Z

The validation set contains 107 examples. i think 3 or 4 are invalid, because the provided generated tests contain illegal input. so you generated responses for 92 out of ~104 valid problems
i think that if you ran the script again, with the same file, it will only try to solve problems it skipped (make sure to back up the file).
i am not sure why you see the async error. its weird. the log doesn't report a problem in the run itself. maybe some memory issue ? make sure you run with 'sandbox=true'. or try with a machine with larger memory.
for 1 or 2 iteration, 13% is a reasonable solve ratio. for more iterations, think of strategies how to create some randomness between each iteration, to increase the chance of pass@5 solve. (hint - try on 1-2 iterations the 'use_direct_solutions' flag. it is somewhat orthogonal, and helps with some problems)

wenlinyao · 2024-01-31T19:25:14Z

Thanks again! that is very helpful!

I double-checked the validation set. It should be 117 original instances? Two of my "5 iterations" run returns "total_passed + total_failed = 105", so I guess the total valid problems is 105? Just want to make sure my experiment setting is correct. ;)
It seems the "async error" can be solved using the original OpenAI API call instead of litellm.
That is a good point. But I see 'use_direct_solutions' flag is defined in the 'configuration.toml' file. So how can I run 5 iterations with a partial of them applying 'use_direct_solutions=true'?

mrT23 · 2024-02-01T14:46:14Z

"So how can I run 5 iterations with a partial of them applying 'use_direct_solutions=true'?"

edit the file https://github.com/Codium-ai/AlphaCodium/blob/main/alpha_codium/gen/dataset_solver.py#L75

if iteration <2 :
...
else:
...

huanhuan6666 · 2024-02-29T14:07:14Z

@wenlinyao
Hello, I've encountered a similar issue and would greatly appreciate your guidance:

ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='Task-479' coro=<AsyncClient.aclose() done, defined at /data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py:1996> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_client.py", line 2006, in aclose
    await proxy.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpx/_transports/default.py", line 383, in aclose
    await self._pool.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/connection_pool.py", line 324, in aclose
    await connection.aclose()
  File "/data/hzhang/.conda/envs/alpha/lib/python3.10/site-packages/httpcore/_async/http_proxy.py", line 350, in aclose
    await self._connection.aclose()

Could you please provide detailed guidance on how to address this issue? I'm eager to learn and make the necessary modifications.🥰

wenlinyao · 2024-03-29T18:11:04Z

@huanhuan6666 I just saw your message. I am not sure how to solve this issue thoroughly. But you can try to replace litellm LLM call with the OpenAI official API call.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Meaning of parameters in 'configuration.toml' #21

Meaning of parameters in 'configuration.toml' #21

wenlinyao commented Jan 26, 2024

mrT23 commented Jan 27, 2024 •

edited

wenlinyao commented Jan 29, 2024

mrT23 commented Jan 30, 2024

wenlinyao commented Jan 30, 2024

mrT23 commented Jan 30, 2024 •

edited

wenlinyao commented Jan 31, 2024

mrT23 commented Feb 1, 2024

huanhuan6666 commented Feb 29, 2024

wenlinyao commented Mar 29, 2024

Meaning of parameters in 'configuration.toml' #21

Meaning of parameters in 'configuration.toml' #21

Comments

wenlinyao commented Jan 26, 2024

mrT23 commented Jan 27, 2024 • edited

wenlinyao commented Jan 29, 2024

mrT23 commented Jan 30, 2024

wenlinyao commented Jan 30, 2024

mrT23 commented Jan 30, 2024 • edited

wenlinyao commented Jan 31, 2024

mrT23 commented Feb 1, 2024

huanhuan6666 commented Feb 29, 2024

wenlinyao commented Mar 29, 2024

mrT23 commented Jan 27, 2024 •

edited

mrT23 commented Jan 30, 2024 •

edited