Docker revamp #19

heyqule · 2019-09-24T02:00:06Z

Hey Chris

Here are the list of major changes. Let me know if there are things that needed to be change.

major changes

Dockerized the system. CLI scripts are retired.
- All settings are fetched from src/config.yml.
Replaced ElasticSearch 5.6 with ElasticSearch 7.3.
Added Redis for caching.
Automated requirements installation and kibana dashboard setup.
Converted original scripts to modules and classes to simplify the process of building new extensions
Data mapping have changed.
- Each Symbol has it's own set of index. One for sentiment and one for price.
- See src/Stocksight/EsMap for mapping details
Sentiment and price crawlers are spawned concurrently based on your specified stock symbols.
Improved memory footprint by spawning python instances when it's needed.

added

Added SeekingAlpha crawler
Added integration test cases
Added support for generating random proxy and random user-agent.
- may not be useful for sophisticated blockers.

current issues:

SeekingAlpha blocks frequent accesses with 403. Follow_link is disabled for it.
It doesn't have data upgrade migration.

Dashboard fix

Doc change

Add Redis support

Add redis cache Drop support for unicode() Refactor StockPrice

Twitter Stream sentiment is now supported

Add test for yahoo new listener Add sh code to listen for elastic search server Add threading support

Additional news scraper change

Fix SeekingAlpha message handler

- Move interval ticker to environment variable

- Fix Readme - add env_var to allow overwrite of kibana dashboard

- - Fix copyright

- Add copyright blocks - FIx Change Log - Move sleep timer values to config.yml

caseyjkey · 2019-09-30T22:50:47Z

src/stockprice.py


        while True:

+            if self.isNotLive(eastern_timezone):
+                #logger.info("Stock market is not live. Current time: %s" % datetime.datetime.now(timezone).strftime("%Y-%m-%d %H:%M"))


currentTime = datetime.datetime.now(timezone).strftime("%Y-%m-%d %H:%M") logger.info(f"Stock market is not live. Current time: {currentTime}")

That's from an outdated commit.

caseyjkey · 2019-09-30T22:52:50Z

src/stockprice.py

            logger.info("Grabbing stock data for symbol %s..." % symbol)

            try:

                # add stock symbol to url
-                url = re.sub("SYMBOL", symbol, url)
+                url = regex.sub("SYMBOL", symbol, url)


Nice touch, improves readability.

shaggy63 · 2019-10-08T10:26:41Z

Line 61 of src/StockSight/StockPriceListener.py
You have D = []
which produces "list indices must be integers or slices, not str"
Shouldn't line 61 be
D = {}

shaggy63 · 2019-10-11T16:17:16Z

Tweets arn't showing in Kibana.
Shouldn't line 145 in src/StockSight/TweetListener.py
"_id": redis_id,
be
"msg_id": redis_id,

Disable message body test for seeking alpha

caseyjkey

Partial review considering the amount of commits for the single pull request. Consider creating feature branches more frequently so Chris can approve them faster. Thank you for dockerizing.

caseyjkey · 2019-09-30T22:55:54Z

src/stockprice.py

-    parser.add_argument("-f", "--frequency", metavar="FREQUENCY", default=600, type=int,
-                        help="How often in seconds to retrieve stock data (default: 120 sec)")
+    parser.add_argument("-f", "--frequency", metavar="FREQUENCY", default=price_frequency, type=int,
+                        help="How often in seconds to retrieve stock data (default: %d sec)" % price_frequency)


help=f"How often in seconds to retrieve stock data (default: {price_frequency} sec)"

This file is no longer applicable in the latest commit.

caseyjkey · 2019-09-30T22:56:24Z

src/sentiment.py

-    parser.add_argument("--frequency", metavar="FREQUENCY", default=3600, type=int,
-                        help="How often in seconds to retrieve news headlines (default: 3600 sec)")
+    parser.add_argument("--frequency", metavar="FREQUENCY", default=sentiment_frequency, type=int,
+                        help="How often in seconds to retrieve news headlines (default: %d sec)" % sentiment_frequency)


consider using f string instead of old school variable replacement, like so: f"How often in seconds to retrieve news headlines (default: {sentiment_frequency} sec)"

This file is no longer applicable in the latest commit.

caseyjkey · 2019-09-30T22:56:49Z

src/sentiment.py

+
+        self.headlines = new_headline
+
+        if len(self.followedlinks) > self.max_cache:


Could the loop be replaced with a array slice like: self.headlines = self.headlines[self.max_cache / 2:]?

This file is not applicable in the latest commit.

caseyjkey · 2019-10-02T20:32:04Z

src/stockprice.py

                today = datetime.datetime.now(eastern_timezone)
-                logger.info("Stock market is not live. Current time: %s" % today.strftime('%H'))
+                logger.info("Stock market is not live. Current time: %s" % today.strftime("%Y-%m-%d %H:%M"))


logger.info(f"Stock market is not live. Current time { today.strftime('%Y-%m-%d %H:%M') }")

I didn't know about this lol.

caseyjkey · 2019-10-07T19:45:30Z

Python.Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3
+FROM python:3-alpine


Why alpine?

Small image foodprint. Should I use the regular container? Regular one is probably easier to add new features.

It's up to you. I'm reviewing to learn more about the project and also about Docker. I haven't dockerized anything before.

caseyjkey · 2019-10-07T20:18:06Z

src/delindex.py

-from Initializer.LoggerInit import *
-from Initializer.ElasticSearchInit import es
+from Sentiment.Initializer.ElasticSearchInit import es
+from Sentiment.Initializer.LoggerInit import *


Thank you for alphabetizing. Good clean code principles.

caseyjkey · 2019-10-07T20:19:19Z

src/news.sentiment.py

@@ -11,21 +11,14 @@
 """

 import argparse
-import json


Were these removed because they are imported indirectly through NewsHeadlineListener? Looks a lot cleaner without all the imports.

Ya.. They imported through NewsHeadlineListener. It's suggested to remove by PyCharm.

caseyjkey · 2019-10-07T20:20:39Z

docker-compose.yml

-      - cluster.name=docker-cluster
+      - cluster.name=elasticsearch
+      - node.name=stockdata
+      - cluster.initial_master_nodes=stockdata


What does this do?

It set the cluster name and node name. ElasticSearch wanted me to specify it.

caseyjkey · 2019-10-07T20:21:27Z

src/Sentiment/NewsHeadlineListener.py



                if self.symbol in nltk_tokens_required:
                    nltk_tokens = nltk_tokens_required[self.symbol]
                else:
                    nltk_tokens = nltk_tokens_required['default']

+                # check required tokens from config
+                tokenspass = False


Nice improvement in readability.

caseyjkey · 2019-10-07T20:22:47Z

src/news.sentiment.py

@@ -122,6 +119,7 @@

                # create instance of NewsHeadlineListener
                newslistener = NewsHeadlineListener(symbol, url)
+                time.sleep(random.randrange(2,5))


Why the addition of a random sleep interval?

I don't want it to have a consistent request pattern.

heyqule · 2019-10-17T02:18:37Z

Partial review considering the amount of commits for the single pull request. Consider creating feature branches more frequently so Chris can approve them faster. Thank you for dockerizing.

@caseykey

Thanks for the review. However, it seems you are reviewing the older commits. There were a lot of changes and experiments on earlier commits since this is my first Python project.

This pull request was submitted after I have done a lot of changes and I asked Chris' whether he's interested on dockerizing it. I initially did this as a Python practice. Submitting the pull request wasn't planned till the later stage.

aminya · 2021-02-11T09:01:39Z

python-docker/Dockerfile

+#stocksight is released under the Apache 2.0 license. See
+#LICENSE for the full license text.
+FROM python:3-alpine
+


Required for regex

Suggested change

RUN apk add build-base

heyqule added 30 commits August 22, 2019 02:31

Add docker compose for kibana and elasticsearch

b292b6d

Merge remote-tracking branch 'upstream/master'

4aef0a3

Add support for multiple set of nltk tokens. Controls by --index

8ee20b8

Fequency adjustment

9dca234

Dashboard fix

Fully automate the build with docker

74e6b36

Add support to bypass fetching stock price outside of regular hours.

53408c9

Doc change

Fix time display

d5393e7

Optimization

4dc0238

Fix hour() error

b7cda28

Fix Cache Cleaning issue

a8bf22a

Change startup.sh to startup.sample.sh

aea6a59

Add Curl to python instance for cleaning purposes.

64ddc35

Clean cache

004c17c

Change Kibana template

3cd6de0

Move news out of original sentiment script

fde9181

Break down News SA

92a9447

Add Redis support

remove exposed ports

4205c8c

Elasticsearch / Kibana 7.3 change

ff5a8cf

Add ndjson importer

e310886

Add ndjson importer

10502c8

Remove kibana 5.6 export

7675154

Fix kibana importer

c2a7010

Add redis cache Drop support for unicode() Refactor StockPrice

Update Copyright

cb42d10

Change to wt

fd9fe56

Change Mapping to 7.3 format

3a3b452

Twitter Stream sentiment is now supported

Disable twitter sentiment stream in start.sh

f3c1895

Rename original py to og.py

e6c9f1b

Change config handling

3fc49a6

Add test for yahoo new listener Add sh code to listen for elastic search server Add threading support

Fix twitter

e86efe7

Since it's single node insance, disable replica

5f1d87f

heyqule added 17 commits September 9, 2019 23:39

Kibana change

56901dc

Additional news scraper change

Kibana - remove legend

abbc740

Add kibana listener

a6002ac

Revert ndjson

c6cf17b

Attempt to fix stock price operant error

bda22a4

Fix elastic mapping

b7226d4

Fix SeekingAlpha message handler

Add delay for Seek Alpha

5cde9c9

Add delay for Seek Alpha

c3431c4

- Separate sentiment for message and title

d086bc6

- Move interval ticker to environment variable

- Kibana adjustment

f84a379

- Config adjustment

60e06fc

- Fix Readme - add env_var to allow overwrite of kibana dashboard

- Improve Kibana dashboard

0d7c7a4

- - Fix copyright

- Improve Kibana Dashboard

fb6bea1

- Add copyright blocks - FIx Change Log - Move sleep timer values to config.yml

- Additonal Readme change

097c774

- Fix kibana tmp folder issue

efc7387

- Minor change to spawn timers

175dd61

Minor Refactor

a178733

caseyjkey reviewed Sep 30, 2019

View reviewed changes

shirosaidev and others added 4 commits October 11, 2019 16:12

Merge branch 'master' into master

9c55d3d

Fix issue found by shaggy63

6f38025

Disable message body test for seeking alpha

Merge remote-tracking branch 'origin/master'

5d17a6c

Disable unnecessary exposed ports

646b0d9

caseyjkey reviewed Oct 16, 2019

View reviewed changes

Add copyright blocks to non-py files

b985410

aminya mentioned this pull request Feb 11, 2021

Fix NewsHeadlineListener in cli #28

Closed

aminya reviewed Feb 11, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker revamp #19

Docker revamp #19

heyqule commented Sep 24, 2019

caseyjkey Sep 30, 2019

heyqule Oct 12, 2019

caseyjkey Sep 30, 2019

shaggy63 commented Oct 8, 2019

shaggy63 commented Oct 11, 2019

caseyjkey left a comment

caseyjkey Sep 30, 2019

heyqule Oct 17, 2019

caseyjkey Sep 30, 2019

heyqule Oct 17, 2019

caseyjkey Sep 30, 2019

heyqule Oct 17, 2019

caseyjkey Oct 2, 2019

heyqule Oct 17, 2019

caseyjkey Oct 7, 2019

heyqule Oct 17, 2019

caseyjkey Oct 17, 2019

caseyjkey Oct 7, 2019

caseyjkey Oct 7, 2019

heyqule Oct 17, 2019

caseyjkey Oct 7, 2019

heyqule Oct 17, 2019 •

edited

caseyjkey Oct 7, 2019

caseyjkey Oct 7, 2019

heyqule Oct 17, 2019

heyqule commented Oct 17, 2019 •

edited

aminya Feb 11, 2021 •

edited


		self.headlines = new_headline

		if len(self.followedlinks) > self.max_cache:

Docker revamp #19

Are you sure you want to change the base?

Docker revamp #19

Conversation

heyqule commented Sep 24, 2019

major changes

added

current issues:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaggy63 commented Oct 8, 2019

shaggy63 commented Oct 11, 2019

caseyjkey left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heyqule Oct 17, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heyqule commented Oct 17, 2019 • edited

aminya Feb 11, 2021 • edited

Choose a reason for hiding this comment

heyqule Oct 17, 2019 •

edited

heyqule commented Oct 17, 2019 •

edited

aminya Feb 11, 2021 •

edited