Future of YahooFinancials

[JECSand]

Hello Everyone,

I wanted to start by thanking all of you who have stuck by this project over the past few years. I wrote YahooFinancials early in my career as a learning experience and to offer an alternative to yahoo-finance. Overtime I became extremely busy with my professional work and other projects, which resulted in me not paying much attention to the needs of YF. During that period, folks like @sylvandb stepped in and really kept this project alive. Many of the optimizations to YF, such as the caching mechanism and some important fixes are thanks to sylvandb.

My plan for YF going forward is to maintain v1 while beginning work on a v2 release sometime during Q2 of this year. Refer to #118 for a discussion on v2.0 feature ideas, I plan on closing that discussion by mid-February. I want to really overhaul the code quality while implementing the features needed to bring YF into the current era of Python development. For v2 development, I've created a version2 branch. Please submit pull requests of approved features into that branch. I intend for v2 to be backwards compataible with v1, the only exception being the proxies parameter released in v1.9.

After the release of v2, I plan to begin working on a new successor project to YF. My hope is to have it ready for initial alpha release by end of Q3. The goal of this new project is to develop a generic extractor for financial data from many sources, not just Yahoo Finance. Builtin default data sources, such as Yahoo Finance, will be interfaced with via a "Connector" class which will also offer the added ability for users to instantiate and implement their own custom connectors to alternative data sources. Other default data sources could include EDGAR #104, Oanda, and others. I'll post more information on this project as we get closer to a v2 release. I intend for it to be more decentralized, democratic, and not dependent upon one person.

I fully intend to maintain YF after v2 for the foreseeable future even after release of the successor project. This maintenance will be in the form of fixing bugs and issues arising from changes in Yahoo Finance's data store / api.

Once again, I appreciate all of you and look forward to delivering on this. Please feel free to drop comments, questions, or concerns on this thread. Any posts regarding feature ideas for v2 should be directed towards the dedicated v2.0 ideas discussion.

Best Regards,
Connor

[ValueRaider]

Your Connector project - I have some rhetorical questions because I suspect it's a bad idea.

How is this better than a simple Wiki page linking to GitHub Python projects e.g. yahooquery and sec-edgar? Isn't that already decentralized?

Maybe what's needed is a better community/organisation, where maintainers/devs can assist each others projects?

[JECSand]

@ValueRaider My current thinking is to build an abstraction layer on top of libraries such as yahooquery and sec-edgar. That way with a single pip installed package an end-user could do something along the lines of:

from demofinance import Extractor

extractor = Extractor(default=“yfinance”)
aapl = extractor.Ticker(“AAPL”)
print(aapl.income_stmt(“sec-edgar”))
print(aapl.income_stmt(“yahooquery”))
print(aapl.income_stmt())    # would use yfinance as default

If these data sources are definable in a reusable configuration driven fashion, then end-users maybe able to eventually define their own sources:

from demofinance import Extractor

# I know this will be much more comprehensive, but just an example
source_config = {
    “type”: “rest-api”,  “host”: “http://localhost:8081”, “auth”: {“username”: “”, “password”: “”}, 
    “extracts”: {
        “income_stmt”: {
            “endpoint”: “/income/{ticker}”,
            “method”: “GET”,
            “output_type”: “JSON”,
            “output_map”: {               # {‘universalSchemaField’: ‘sourceSchemaField’}
                “net income”: “net earnings”,                            # alias mapping
                “gross income”: “net earnings + deductions”, # calculated mapping
            }
        }
    }
}

extractor = Extractor().add_source(“customSource”, source_config)
aapl = extractor.Ticker(“AAPL”)
print(aapl.income_stmt(“customSource”))

Some of the benefits are:

1. If a data source goes down / changes / paywalls there would be others available to the end-user without the need of installing a new package and/or a big refactor.
2. If the default source configurations are hosted publicly on github, the maintainers could simply update the configs in the event of a minor change without needing to release a hard fix to PyPI.
3. Ease of data source expansion and interoperability.
4. Stability

To deliver such a project would require a strong community of open source developers mutually assisting one another and would be an endeavor. However it is more than doable and would provide the most stable experience for end-users.

[cmjordan42]

If there is going to be a significant endeavor to reimagine the various financial data scrapers into one, I strongly recommend starting with a well-designed singular data model around the various domains of security data. Especially if sources beyond Yahoo are added - which I think is a great idea and will greatly decrease critical upstream dependencies. Regardless of how Yahoo Finance and Marketwatch and Google Finance may think to display them on their UI, the underlying data point has a singular truth and relation to the security.

I spent a decade working for a (the) premier global financial data provider, several years of which was building out a structured time-series data warehouse (of which a subset is the domain of data we're discussing here). I can probably help architect this if that's the path you want to go down.

[ValueRaider]

@maread99 FYI, just in case

[JECSand]

@cmjordan42 Absolutely, that’s a really good call out. Essentially define standard data models for each financial statement type, pricing data, profile data, etc. From there perhaps a configuration driven approach maybe leveraged to map data from various sources into those standard schemas.

That was absolutely the goal, my thinking is data sources may come and go, however the schema and ideally returned data will be consistent enough for big projects to confidently use long term.

I’d definitely appreciate any help you’d be willing to offer! I think this could morph into a really awesome project.

[JECSand]

@cmjordan42 I set up an org for this, open-financials, if you were still interested.

[azataiot]

It would be better for some vars if they written with more pythonic way using _ instead of shareHolder for example.