Text-Anlaytics

Feature Extraction, Sentiment Analysis, Text Mining

Identifying univariate outlier detection using k-means/DBSCAN where points outside one big cluster are regarded as outliers but they tend to capture spikes in series more than shifts References: https://towardsdatascience.com/density-based-algorithm-for-outlier-detection-8f278d2f7983 https://iopscience.iop.org/article/10.1088/1755-1315/31/1/012035/pdf https://towardsdatascience.com/time-series-of-price-anomaly-detection-13586cd5ff46 Things that can potentially capture level shift are (raw tested today):

Isotonic regression: its a monotonic regression fitting a free form line. By specifying 'increasing=False' among parameters: https://scikit-learn.org/stable/modules/generated/sklearn.isotonic.IsotonicRegression.html we can fit level shifting line without making it strictly monotonic.
Rolling std deviation/variance: http://jonisalonen.com/2014/efficient-and-accurate-rolling-standard-deviation/; https://stackoverflow.com/questions/45487229/identifying-an-event-based-on-a-jump-in-rolling-variance-in-python
Decision tree regression: https://scikit-learn.org/stable/auto_examples/tree/plot_tree_regression.html
Identified method as: Step1 - Fit a Decision Tree Regressor to figure out steps/levels in the data. Step2 - Calculate rolling standard deviation in steps/levels to point out the jumps/dips. Step3 - Capture peaks in standard deviations to identify change of level dates.
One example, for motherboard counts, is shown as: (Legend - Original count: dark blue lines, steps/levels: light blue dots, rolling std dev: green dots)

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
DNN With Tensorflow.ipynb		DNN With Tensorflow.ipynb
DSBDA_code_data.7z		DSBDA_code_data.7z
Edureka DL Class.ipynb		Edureka DL Class.ipynb
HCL.ipynb		HCL.ipynb
Hanger Reviews consolidated.xlsx		Hanger Reviews consolidated.xlsx
Keywords-I-Anomaly-Detection.ipynb		Keywords-I-Anomaly-Detection.ipynb
LCDataDictionary.xlsx		LCDataDictionary.xlsx
LDA Topic Modelling.ipynb		LDA Topic Modelling.ipynb
Lending LoanData.ipynb		Lending LoanData.ipynb
ML_E04_assignment.pdf		ML_E04_assignment.pdf
Plotly Whirlwind Introduction.ipynb.txt		Plotly Whirlwind Introduction.ipynb.txt
README.md		README.md
Similarity Mapping.ipynb		Similarity Mapping.ipynb
TF Recurrent Neural Network.ipynb		TF Recurrent Neural Network.ipynb
Topic Modelling Assignments.ipynb		Topic Modelling Assignments.ipynb
anomaly_detection.py		anomaly_detection.py
iris_sample_notebook.ipynb		iris_sample_notebook.ipynb
ldavis_prepared_3.html		ldavis_prepared_3.html
main.ipynb		main.ipynb
run-Keywords-I.sh		run-Keywords-I.sh
sonar.csv		sonar.csv
sonarII.csv		sonarII.csv
t-Distributed Stochastic Neighbor Embedding(t - SNE).ipynb		t-Distributed Stochastic Neighbor Embedding(t - SNE).ipynb

singhmanish979/Text-Anlaytics