raise_errors for yf.download similar to raise_errors for yf.Ticker(symbol).history

[abrhoda]

Hey all, I am using yfinance to run an analysis on a basket of symbols daily after market close. I am scraping wikipedia for all s&p 500 symbols and make a list out of them and then pass that list into yf.download function. Occasionally, I see the download function fail to download data for a few of the symbols in the list. Is there a way to raise errors similar to the yf.Ticker(symbol).history(...) method?

Relevant code snippet where I'm using the download function:

if (datetime.utcnow().time() < time(21, 00)) and (datetime.utcnow().time() > time(14, 30)):
    end_date = datetime.now() - timedelta(days=1)
    symbols_data = yf.download(config.symbols, start=config.data_start_date, end=end_date)
else:
    symbols_data = yf.download(config.symbols, start=config.data_start_date)

I have noticed that every so often, a few of the symbols will fail to get data from the api for any sort of reason and the script needs to be rerun and it will succeed. Example of running and getting the failed and then rerunning and seeing a success from earlier today:

Looking through the code, I see that the download function calls the multi.download function which ultimately calls Ticker(symbol).history(raise_errors=True, ...) for each symbol in the list. multi.download aggregates the errors into shared._ERRORS and ends up logging via logger.error(...). The errors are then never thrown as an exception and I don't see a flag to set for this similar to how Ticker(symbol).history(...) does.

A few questions:

1. Is there a way to raise errors similar to the Ticker(symbol).history(...) function so that I can exit the script with a non-zero exit code to signal to my cronjob to retry?
2. If not, is there a reason to not have this for the download function similar to the history method?
3. Also if not, is it better for me to manually be calling yf.Ticker(symbol).history(raise_errors=True, ...) for all symbols in the list instead of using the yf.download function in order to be able to see errors?

[ValueRaider]

Looks like a simple oversight, no reason download can't also have raise_errors. But some thought will be needed if multiple errors occurred - raise them all? just raise one? Maybe raise_errors argument allows choice?

Can see what "all" would look like with yf.enable_debug_mode() at top of script.

[abrhoda]

Sure, here's a screenshot of what the error looks like when it happens with debug logging enabled:

On line 46 of the screenshot, there is a list of symbols (just QRVO in this case) that is logged as error but not then raised to the caller. A few times I have run into this, this list is properly including more than one symbol like:

 ['QRVO', 'SYMBOL2']: Exception('%ticker%: No price data found, symbol may be delisted (1d 2023-01-16 -> 2024-01-16)

So it seems that it is aggregating the symbols and messages into a single log statement. I would lean towards a single exception being raised with the list_of_symbols -> message message similar to the log message. Or perhaps follow a .net style of aggregate exceptions and throw a AggregateException with field that is a list of the individual exceptions that occurred?

Diving into the debug log statements, you can also see on line 18 that the http request was attempted but no response came back? The script then moved onto the next symbol, NSC, without issue.

[ValueRaider]

I meant multiple different errors - rare but possible.

[abrhoda]

For the situation where there are multiple different errors and therefore multiple different exceptions, could a single AggregateException be raised that contains all the individual exceptions that were raised? If this is not the python way of doing things, please let me know as I don't work with python too often. Some rough code could look like:

#! /usr/bin/env python

class AggregateException(Exception):
    def __init__(self, exceptions):
        super().__init__(f'exceptions: {exceptions}')
        self.exceptions = exceptions
    
    def __reduce__(self):
        return (AggregateExceptiion, (self.exceptions))

# If typed exceptions are wanted
class HttpClientException(Exception):
   pass

class HttpServerException(Exception):
   pass

if __name__ == '__main__':

    # plain exception example without unique types
    exception_1 = Exception("[SYMBOL]: No price data found, symbol may be delisted (1d 2023-01-17 -> 2024-01-17")
    exception_2 = Exception("[SYMBOL2, SYMBOL3]: Some other exception message that could happen.")
    exc_list = []
    exc_list.append(exception_1)
    exc_list.append(exception_2)
    aggregate_exception = AggregateException(exc_list)
    print(aggregate_exception)

    # typed exception example
    http_client_exception = HttpClientException("log client exception message")
    http_server_exception = HttpServerException("log server exception message")
    typed_exc_list = []
    typed_exc_list.append(http_client_exception)
    typed_exc_list.append(http_server_exception)
    typed_aggregate_exception = AggregateException(typed_exc_list)
    print(typed_aggregate_exception)

The AggregateException could then be raised and the caller could choose to look inside at the individual exceptions that were raised. I was thinking that the type exceptions would help with inspecting the individual exceptions to see if retrying (like in my case) with a HttpServerException would solve the case or if the symbol doesn't exist (such as a 404 http reponse code) and the type is HttpClientException so the caller knows not to retry. But with the untyped version just using the Exception class instead of HttpClientException and HttpServerException, the caller could also just iterate through the exceptions in the aggregate and check what the message is on the exception in order to decide if a retry would solve the issue or not.

Like I said, I don't work with python too much so let me know if there's a more conventional way to do this in python!

[ValueRaider]

I know nothing about aggregating exceptions. I'm fine with simply picking one to raise. Problem with raising all are the stack traces - one is enough to fill screen, never mind all. But which one?

[abrhoda]

I agree. Raising many exceptions, each with their own stacktraces, would get really messy from the terminal if you are just executing a script and they get this error. An aggregate exception is just an exception that contains many exceptions. Probably not what we're looking for here due what you brought up because the individual exceptions that the aggregate exception would retain their stacktracse and create a potentially very long message in the terminal.

I will have some time this weekend so I'll create a fork and look into getting this working for raise_errors param on the multi downloads.

[abrhoda]

Finally got some time to look into this so following up here! I added the the raise_errors option to the multi download function and just changed the shared._TRACEBACKS dict to be the traceback object instead of only the string formatted version. This means in the download function, once all exceptions that occurred have been put into the dicts, the new raise_errors option can be used to either raise one (maybe the only) exception along with its original traceback to the caller or just logged (current behavior). I noticed there aren't unit tests for the multi.py file where the changes were made. I that something I should add? Attached is a quick screen shot from my terminal window where I ran a script with the raise_errors option on the download function set to 3 different values (False, defaulting to False, and True) along with a print statement saying what the case was handling. Is there any kind of testing that I should looking into outside of these manual test cases? I didn't see anything in the contribution guide about testing steps.

[ValueRaider]

I should add?

At least one test that will fail if someone else breaks your code is a good idea, but "should" is up to you.

Are you randomly picking a stack trace to print? I prefer deterministic, easier to debug.

[abrhoda]

Sure. I'll go ahead and add a multi.py under the tests dir and test in there.

I am not just picking a random stack trace to print when multiple exceptions are thrown. They are put under shared._ERRORS{} and shared._TRACEBACKS{} and, if the raise_errors option in False (the defualt as well), then errors and tracebacks are printed as before. If raise_errors is true, I took the approach we discussed and am re-raising 1 exception with its original traceback to the caller. The exception that is raised is the first exception in the shared._ERRORS dict by just calling ticker, err = list(shared._ERRORS.items())[0]. Due to CPython ordering dicts since 3.6 (implementation detail) and officially in 3.7+, this will always return the first item added to the dict - from my understanding. So whichever the first exception that raised in the one that is re-raised to the caller. This is the best way I could think of to pick which is re-raised but open to any idea about a better/more deterministic way? Maybe the alphabetical first? That would require an additional sort though.

[ValueRaider]

Screenshot said "randomly picked", that's why I thought random.

As long as it's deterministic is fine. At some point though, it's better to submit the pull request instead of describing code.