Incomplete list of tickers for stocks

[codeboot]

Hello

I have Python 3.3 (on Windows 7). Several months ago I ran this python script, and I got 14935 stock tickers by running "YahooTickerDownloader.py stocks"

I ran this same script again, but now I only get around 2700 tickers. I reran the script several times, but the number of tickers is always around this number.

Do you know what the problem is? I'd like to get an updated list of tickers for stocks, but I believe there are more than 2700.

Thank you

Tyre

[Benny-]

I have confirmed this issue. I am looking into it.

[Benny-]

The problem has been discovered. Yahoo sometimes returns zero results.

You can reproduce this in your browser:

Go to a random yahoo finance page like http://finance.yahoo.com/lookup/?s=m&r=&m=ALL&t=S&b=20 and refresh it several times. Occasionally it will show zero results.

This causes inconsistent results when running the script multiple times. It assumes yahoo has nothing more to show.

[codeboot]

Hello Benny

Thank you for investigating

Can this issue be resolved? Or does it mean that Python script cannot be rescued and will not return all stock tickers?

Sorry if it appears that I am hassling you.

Tyre

[Benny-]

The script never returned all ticker symbols from yahoo finance and never has. This is unrelated to this issue. Yahoo has always limited the number of results it will show. I seem to have neglected to mention this in the readme. I will mention this in the readme in next release.

Now about this issue, there is a crude fix now in the repo. It will simply try to download a page 20 times if yahoo returns no symbols. If all twenty download attempts still result in nothing, the script will assume yahoo will never give a result for that page and will continue. Now there is a slight chance yahoo happen to give you a bad page for all 20 attempts. This can happen and you will be missing data. Only Yahoo can solve this issue the correct way.

You will have to do a checkout from master to use the crude fix. It will take another day or so to test it before I upload it to the python packaging index.

I will keep this issue open until yahoo fixes the website.

[codeboot]

Benny

I'm currently putting the updated script through its paces.

Thank you for the updated - I appreciate it

Tyre

[Benny-]

I was handling another problem related to utf-8 csv output. This might screw up your .csv output file in python3. Python2 is likely not affected.

If this is the case, please hold on to your downloader.pickle file. The .csv file can be re-created from the downloader.pickle file. This way you don't need to download everything again if the code for outputting the .csv file is fixed.

I'm sorry for the inconvenience.

[Benny-]

The current checkout (24d0511) should work perfectly. It should output around 25495 ticker symbols for stock. And it produces the output in several formats (json, yaml, csv, xls) for convenience. Report if you have any additional issues.

Here is the output of the script (only for stock at the moment) so you don't have to re-run the script:
http://opslag.bennyjacobs.nl/market/2014-08/symbols/
All files contain the same data. Only the format is different.
This webserver might not always exist. And there is no guarantee the files will ever be updated.

[codeboot]

Benny - thanks. I'm now testing the new script, but first impressions are that it's all good.

[Bruno-Hanzen]

I was surprised not to find Solvay (SOLB.BR) in the results.

1. looking for SO: 9 results (no Solvay, though), so the algorithm does no go trigger the recursion.
2. looking for SOL: 10 results

So, the Yahoo interface is not consistent, which is not a real surprise.
I have edited the "netxtRequest" method to trigger the recursion at 6 instead of 10. The test is running.

Added on June 6th: the test has run fine (but took very long: several weeks!) I got 700,000+ symbols (including SOLB.BR).
I have now raised the recursion at 7. Is seems to run slightly faster (less recursions), we'll see the end result.

[feeley]

Hello

I have Python 3.3 (on Windows 7). Several months ago I ran this python script, and I got 14935 stock tickers by running "YahooTickerDownloader.py stocks"

I ran this same script again, but now I only get around 2700 tickers. I reran the script several times, but the number of tickers is always around this number.

Do you know what the problem is? I'd like to get an updated list of tickers for stocks, but I believe there are more than 2700.

Thank you

Tyre

Hello Tyre. I'm trying to get in touch with you. Could you email me at feeley@iro.umontreal.ca ?

Sorry for the noise on this unrelated issue!