v2.0 Ideas

[JECSand]

Some ideas I have for v2.0 of yahoofinancials:

1. Ability for user's to log into YahooFinance using their Yahoo+ credentials to get additional historical data, i.e. Download historical fundemental data #85.
2. Resolving reported differences between extracted data and data from website, i.e. Financials retrieved very different from the Yahoo Finance site #102.
3. Authentication support for proxies
4. Improvements to exception handling, i.e. class yahoofinancials.ManagedException does not have __str__ #76
5. Additional scraping methods, such as get_esg_data(), i.e. ESG data #48
6. Additional calculated methods, such as a generic get_calculated_sma(day=0), i.e. 20 sma and 100 sma #49
7. Improvements to the package's use of Requests, including investigating incorporation of 'requests_futures' for better concurrent extractions.
8. Lastly, adding logic to make YahooFinancials optionally async. This would enable it to extract in the background so the parent script can continue executing code until needing to force a resolution via a new yf.resolve() method.)

I also welcome the inclusion of any additional ideas for v2.0. Please feel free to add them to this discussion. I'd live to have v2.0 in production sometime in Q2 of this year.

[JECSand]

Pandas DF output suggestion from @SymbReprUnlim

[SymbReprUnlim]

Hi Connor, I've successfully retrieved quarterly and annual income statements using yahoofinancials v1.12 and converted them into Pandas Dataframes, (Python code shown below). I've inserted both the Python 3.7 "py" program AND console output with Dataframe results into this posting.

Actual Python code indentation likely does not show up correctly and apparently *.py programs cannot be attached to github comments, so everything had to be copied and pasted as per the below. Also "type(...).name" may not be showing up correctly - the underscores seem to disappear(e.g. underscore-underscore name underscore-underscore in actual Python code) - likely an html/Python translation/viewing quirk.
Best regards, Rob

Python program:
""" test library yahoofinancials; add functionality; convert income statements into Pandas dataframes;
Important Note: Python program correctly works with Debian Buster, Python version 3.7 and yahoofinancials v1.12 as of 2/1/2023 !!
yahoofinancials Documentation source: https://pypi.org/project/yahoofinancials/ """

import datetime, copy
import pandas as pd
from yahoofinancials import YahooFinancials as yfin
import pdb as pdb

def df_create_cols(sCSV_or_vecColLabels): # create empty pandas dataframe with column headers ["colhdr1","colhdr2",...]
svt=type(sCSV_or_vecColLabels).name
if svt=="list": df_out=pd.DataFrame(columns=sCSV_or_vecColLabels) # add list as rows: df.append(list_row); created from e.g. ["Date","Hdr1","Hdr2"]
else:
try:
df_out=pd.DataFrame(columns=(sCSV_or_vecColLabels.split(","))) # created from csv hdrs, e.g. "Date,Hdr1,Hdr2"
except:
print("Error in df_create_cols!!! sCSV_or_vecColLabels not right?")
print("varType(sCSV_or_vecColLabels,True):",svt)
print("sCSV_or_vecColLabels:",sCSV_or_vecColLabels)
pdb.set_trace()
return df_out

def df_append_row(df_in,vecVals,boolMultipleRows=False): # append a list to a pandas dataframe as new row(s)
if boolMultipleRows==True: # appending several rows
df_out=copy.deepcopy(df_in)
for row1 in vecVals: # loop thru all rows to append
df_out=df_append_row(df_out,row1) # single append row call
return df_out
df_out=copy.deepcopy(df_in)
df_out.loc[len(df_out)]=vecVals # append new row onto df_ dataframe at bottom
return df_out

def nested_list2df(listColHdrs,listNestedVals): # pprt nested list as ret'd df_; create pandas dataframe df_ from hdrs_list & nested_list_of_data_rows; if listColHdrs=[] it uses col_hdrs = listNestedVals[0]
i_beg=0
if len(listColHdrs)==0: # use topmost row in listNestedVals as col hdrs
listColHdrs=listNestedVals[0]
i_beg=1
df_out=df_create_cols(listColHdrs)
for ii in range(i_beg,len(listNestedVals)): # list1row_vals in listNestedVals:
list1row_vals=listNestedVals[ii]
df_out=df_append_row(df_out,list1row_vals)
return df_out

def dict_get_nth_key(dict_in, i_key_n=0): # get the nth key (0...n-1) in a dictionary
list_keys=list(dict_in.keys()) # retrieve all DICTionary KEYS, place them into list obj
try: key_n=list_keys[i_key_n] # attempt to retrieve nth dict key # from list of keys
except: key_n=-99999 # did not find key #
return key_n

def get_inc_stmts(sSymb,s_freq="q"): # retrieve/get quarterly or annual income statements for a ticker symbol using yahoofinancials/yfin
sSymb=sSymb.upper()
s_freq=s_freq.lower()
if s_freq=="q" or s_freq=="qtr" or s_freq=="qtrly":
s_freq="quarterly"
s_fr="Quarterly"
else:
s_freq="annual"
s_fr=""
yfin_obj=yfin(sSymb)
obj_is=yfin_obj.get_financial_stmts(s_freq,"income",reformat=True) # reformat==False returns a very large raw dictionary
obj_is1=obj_is["incomeStatementHistory"+s_fr] # retrieve inner dictionary
list_is=obj_is1[sSymb]
for ii,dict_is1 in enumerate(list_is):
is_key_str_date=dict_get_nth_key(dict_is1,0)
if ii==0: # first time thru, set up lst_line_items & lst_line_vals
lst_is_dates=["LineItem",is_key_str_date]
dict_is_x=dict_is1[is_key_str_date]
lst_line_items=list(dict_is_x.keys())
lst_line_vals=dict_is_x.values()
i_len=len(lst_line_items)
nstd_list_is_vals=[None] * i_len
for jj,val in enumerate(lst_line_vals): nstd_list_is_vals[jj]=[lst_line_items[jj],val] # place is values into ea's respective line items
else:
lst_is_dates.append(is_key_str_date)
dict_is_x=dict_is1[is_key_str_date]
lst_line_itms=dict_is_x.keys()
lst_line_vls=dict_is_x.values()
for jj,val in enumerate(lst_line_vls):
# print("jj",jj," val",val)
lst_x=nstd_list_is_vals[jj]
try: lst_x.append(val)
except:
print("Error, unable to: lst_x.append(val)")
pdb.set_trace()
nstd_list_is_vals[jj]=lst_x
df_is=nested_list2df(lst_is_dates,nstd_list_is_vals)
return df_is

#------------------------------- Main Program Starts Here -------------------------------
dt_start = datetime.datetime.now() # for elapsed time, record start time
print("----- Started "+file+" for downloading & analyzing stocks information at "+dt_start.strftime("%a %-m/%-d/%Y %H:%M:%S")+" : -----")
sSymb="msft"
df_q_inc_stmts=get_inc_stmts(sSymb,"q")
df_a_inc_stmts=get_inc_stmts(sSymb,"a")
print(sSymb.upper(),"df_q_inc_stmts:\n",df_q_inc_stmts)
print("-"*35)
print(sSymb.upper(),"df_a_inc_stmts:\n",df_a_inc_stmts)
#------------------------------------------------------------------------------------------
dt_ended = datetime.datetime.now() # for elapsed time, record ended time
dt_elapsed = dt_ended - dt_start
flt_secs = round(dt_elapsed.seconds,4)
if flt_secs == 0: flt_secs = 1
print("-"*85)
print("Ended "+file+" at "+dt_ended.strftime("%a %-m/%-d/%Y %H:%M:%S")+".")
print("Program "+file+" req'd",flt_secs,"seconds to run/fully execute")
pdb.set_trace()

---

Console output (Pandas dataframe columns shown below correctly line up within a Linux Terminal/Console):
----- Started test_yahoofinancials_inc_stmts.py for downloading & analyzing stocks information at Wed 2/1/2023 01:46:23 : -----
MSFT df_quarterly_inc_stmts:
LineItem 2022-12-31 2022-09-30 2022-06-30 2022-03-31
0 researchDevelopment 6844000000 6628000000 6849000000 6306000000
1 effectOfAccountingCharges None None None None
2 incomeBeforeTax 20339000000 21572000000 20487000000 20190000000
3 minorityInterest None None None None
4 netIncome 16425000000 17556000000 16740000000 16728000000
5 sellingGeneralAdministrative 7216000000 6524000000 8053000000 7075000000
6 grossProfit 35259000000 34670000000 35436000000 33745000000
7 ebit 21199000000 21518000000 20534000000 20364000000
8 operatingIncome 21199000000 21518000000 20534000000 20364000000
9 otherOperatingExpenses None None None None
10 interestExpense -493000000 -500000000 -492000000 -499000000
11 extraordinaryItems None None None None
12 nonRecurring None None None None
13 otherItems None None None None
14 incomeTaxExpense 3914000000 4016000000 3747000000 3462000000
15 totalRevenue 52747000000 50122000000 51865000000 49360000000
16 totalOperatingExpenses 31548000000 28604000000 31331000000 28996000000
17 costOfRevenue 17488000000 15452000000 16429000000 15615000000
18 totalOtherIncomeExpenseNet -860000000 54000000 -47000000 -174000000
19 discontinuedOperations None None None None
20 netIncomeFromContinuingOps 16425000000 17556000000 16740000000 16728000000
21 netIncomeApplicableToCommonShares 16425000000 17556000000 16740000000 16728000000

MSFT df_annual_inc_stmts:
LineItem 2022-06-30 2021-06-30 2020-06-30 2019-06-30
0 researchDevelopment 24512000000 20716000000 19269000000 16876000000
1 effectOfAccountingCharges None None None None
2 incomeBeforeTax 83716000000 71102000000 53036000000 43688000000
3 minorityInterest None None None None
4 netIncome 72738000000 61271000000 44281000000 39240000000
5 sellingGeneralAdministrative 27725000000 25224000000 24709000000 23098000000
6 grossProfit 135620000000 115856000000 96937000000 82933000000
7 ebit 83383000000 69916000000 52959000000 42959000000
8 operatingIncome 83383000000 69916000000 52959000000 42959000000
9 otherOperatingExpenses None None None None
10 interestExpense -2047000000 -2330000000 -2591000000 -2686000000
11 extraordinaryItems None None None None
12 nonRecurring None None None None
13 otherItems None None None None
14 incomeTaxExpense 10978000000 9831000000 8755000000 4448000000
15 totalRevenue 198270000000 168088000000 143015000000 125843000000
16 totalOperatingExpenses 114887000000 98172000000 90056000000 82884000000
17 costOfRevenue 62650000000 52232000000 46078000000 42910000000
18 totalOtherIncomeExpenseNet 333000000 1186000000 77000000 729000000
19 discontinuedOperations None None None None
20 netIncomeFromContinuingOps 72738000000 61271000000 44281000000 39240000000
21 netIncomeApplicableToCommonShares 72738000000 61271000000 44281000000 39240000000

Ended test_yahoofinancials_inc_stmts.py at Wed 2/1/2023 01:46:32.
Program test_yahoofinancials_inc_stmts.py req'd 8 seconds to run/fully execute
--Return--

-> pdb.set_trace()
(Pdb)

[JECSand]

Thank you for sharing that!

[JECSand]

Closing this discussion today, but here are the planned features for v2.0:

1. New output format for fundamentals. Instead of dicts in a list, a dict with datetime keys will be returned. Legacy return format will still be available via a YahooFinancials class keyword for backwards support.
2. Pandas output will be available via a YahooFinancials class keyword.
3. Authentication support for proxies.
4. Async improvements.
5. Persistent caching support.
6. Ability to use Yahoo+ credentials in requests
7. Improvements to logging and exception handling.