unstable api.tavily.com/search

[jeremi]

Over the last hour I have been trying
gpt-researcher but it seems the tavily is quite unstable. it sometimes return results, but so far, I have not been able to finish a research. There is always a search that fail. Should retry be implemented? Is there is a way to restart a research that has been interrupted?

Here is an example when it fail:

  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/tavily/tavily.py", line 47, in search
    return self._search(query, search_depth=search_depth, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/tavily/tavily.py", line 41, in _search
    response.raise_for_status()  # Raises a HTTPError if the HTTP request returned an unsuccessful status code
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

[assafelovic]

Hey @jeremi due to high volumes we had some slow response times. After scaling up it should be better now, can you check?

[jeremi]

It seems good today, no issue. Thanks!

[Tristandiscorder]

Hmm looks like its recurring on my end though

🔎 Running research for 'Current market opinion on Samsung Electronics stock December 2023'...
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/websockets/wsproto_impl.py", line 234, in run_asgi
result = await self.app(self.scope, self.receive, self.send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in call
await super().call(scope, receive, send)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 149, in call
await self.app(scope, receive, send)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call
raise e
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call
await self.app(scope, receive, send)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 341, in handle
await self.app(scope, receive, send)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 82, in app
await func(session)
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 325, in app
await dependant.call(**values)
File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/backend/server.py", line 50, in websocket_endpoint
report = await manager.start_streaming(task, report_type, websocket)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/utils/websocket_manager.py", line 51, in start_streaming
report = await run_agent(task, report_type, websocket)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/utils/websocket_manager.py", line 63, in run_agent
report = await researcher.run()
^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/master/agent.py", line 52, in run
scraped_sites = await self.scrape_sites_by_query(sub_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/master/agent.py", line 91, in scrape_sites_by_query
search_results = retriever.search(max_results=self.cfg.max_search_results_per_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/retrievers/tavily_search/tavily_search.py", line 43, in search
results = self.client.search(self.query, search_depth="advanced", max_results=max_results)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/tavily/tavily.py", line 47, in search
return self._search(query, search_depth=search_depth, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/tavily/tavily.py", line 41, in _search
response.raise_for_status() # Raises a HTTPError if the HTTP request returned an unsuccessful status code
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

[samgoodman]

EchoingTristandiscorder, this seems to be persistent

[garmanwu]

File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

I have the same problem.

[sameigen]

This is consistently happening to me as well, @assafelovic maybe worth reopening? Not sure if there is another issue open for this

[SimonB97]

Yep, same. @assafelovic this should be re-opened.

Not usable at current state.

[seanmavley]

Can't even ping the API endpoint at the moment. Not sure what's happening with tavily. Yet it was working earlier the other day. 😓😮‍💨

It's kinda unstable since it's failure breaks the AI chain

🤦🏾‍♂️

[AryanPadmanabhan]

Happening to me as well. It works once and then breaks the next time. It also uses one of my API call requests when it doesn't work @assafelovic

[evangriffiths]

Also getting requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search. Please re-open. I get this when using both a free and paid-for API key.

[nmcross]

Having the same issue.
Error: 400 Client Error: Bad Request for url: https://api.tavily.com/search
ERROR: Exception in ASGI application

[randomclicks]

HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

hey guys, u all keep getting this ?

[seanmavley]

@randomclicks Yep, randomly

[krunaln]

@randomclicks, Yes i am also facing this issue.

[ayushmore]

Same

[assafelovic]

Hey everyone, for Tavily issues please reach out at support@tavily.com.
I'll update here if I know anything