Inspired by this. Read it first: http://www.analyticsvidhya.com/blog/2013/06/art-structured-thinking-analyzing/
- Figure out the questions involved in the analytics project and decide which ones can be tackled
separately, and which ones are intertwined with others, and which ones need to be answered first
before tackling others. Then pick one.
0.5 Layout the data requirements and hypothesis before looking at what data is available
- Actually Look at the data summary(dataframe.describe()) that includes mean, mode, std, and quartiles)
- Look for patterns in the summary. Think about what each of the values mean to your question? What
do questions do they lead to? How do they modify your question?
- Figure out the ML problem use this.
- Go back to step 1 and 2 again and redo them with the ML problem .
- See if you have enough data (noise vs signal) or you need more samples or do you need more
features. (see http://scikit-learn.org/stable/modules/feature_selection.html)
First Model building time-split:
1.Descriptive analysis on the Data – 50% time
2.Data treatment (Missing value and outlier fixing) – 40% time
3.Data Modelling – 4% time
4.Estimation of performance – 6% time
Data Exploration steps:
Source Reference: https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/
Below are the steps involved to understand, clean and prepare your data for building your predictive model:
4.Missing values treatment
Missing Value Treatment:
2.Mean/ Mode/ Median Imputation
1.Data Entry Errors:
2. Measurement Error:
3. Experimental Error:
4. Intentional Outlier:
5. Data Processing Error:
6. Sampling error:
7. Natural Outlier:
- word 2 vector is a way to take a big set of text and convert into a matrix with a word at
- It is a shallow neural-network(2 layers)
- Two options/training methods (
- — a text is represented as the bag(multiset) of its words
- — disregards grammar
- — disregards word order but keeps multiplicity
- — Also used in computer vision
skip-gram() — it is a generalization
of n-grams(which is basically a markov chain model, with (n-1)-order)
* — It is a n-1 order markov model
* — Used in Protein sequencing, DNA Sequencing, Computational
linguistics(character and word)
* — Models sequences, using the statistical properties of n-grams
* — predicts based on .
* — in language modeling independence assumptions are made so that each
word depends only on n-1 previous words.(or characters in case of
character level modeling)
* — The probability of a word conditional on previous n-1 words follows a
* — In practice, the probability distributions are smoothed by assigning non-zero probabilities to unseen words or n-grams.
- — Finding the right ‘n’ for a model is based on the Bias Vs Variance tradeoff we’re wiling to make
- — Problems of balance weight between infrequent n-grams.
- — Unseen n-grams by default get 0.0 without smoothing.
— Use pseudocounts for unseen n-grams.(generally motivated by
bayesian reasoning on the sub n-grams, for n < original n)
— Skip grams also allow the possibility of skipping. So a 1-skip bi(2-)gram would create bigrams while skipping the second word in a three sequence.
- — Could be useful for languages with less strict subject-verb-object order than English.
- Depends on Distributional Hypothesis
- Vector representations of words called “word embeddings”
- Basic motivation is that compared to audio, visual domains, the word/text domain treats
them as discrete symbols, encoding them as sparse dataset. Vector based representation
works around these issues.
- Also called as vector Space models
Two ways of training: a, CBOW(Continuous-Bag-Of-Words) model predicts target words, given
a group of words, b, skip-gram is ulta. aka predicts group of words from a given word.
Trained using the Maximum Likelihood model
- Ideally, Maximizes probability of next word given the previous ‘h’ words in terms of a softmax function
- However, calculating the softmax values requires computing and normalizing each probability using score for all the other words in context at every step.
- Therefore a logistic regression aka binary classification objective functionis used.
- The way this is achieved is called negative sampling
I thought of you when I read this quote from “Seven secrets of Shiva” by DEVDUTT PATTANAIK –
“Culture by its very nature makes room for some practices and some people, and excludes others. Thieves and criminals and ghosts and goblins have no place in culture.”
Start reading this book for free: http://amzn.in/5sGSMqm
. Given the choice between the Farm and the Organization, he picked the Organization. I would have too. I have yet to meet a TPS report so onerous I would prefer to be handpicking cotton in Tennessee in August.