Improve Precision of a binary classifier - Decision Tree in Python

I guess differences in accuracies between class 0 and class 1 come from the class_weight parameter you have used. Class 1 will benefit from this overweighting towards class 0. You could try to play on this parameter to re-balance your results in class 0 and class 1.

An other idea could be to play on probabilities outputs and decision boundary threshold. Remember than when calling for method .predict(), sklearn decision tree will compare outputed probability with threshold 0.5. If it is greater than 0.5, then it assign class 1. On the contrary, if it is less than 0.5, it will assign class 0. You could try to play on this threshold by outputing probabilities first with .predict_proba()[:,1] and then test results for different thresholds decision boundaries. You can operate such as below :

model = clf.fit(df[features], df[label])
df["proba"] = model.predict_proba(df[features])[:,1]
threshold = 0.4 # You can play on this value (default is 0.5)
df["pred"] = df["proba"].apply(lambda el: 1.0 if el >= threshold else 0.0)


How to control number of branches for each split in sklearn DT classifier?

Scikit-learn uses an optimized version of the CART algorithm, which follows binary splits. What you are looking for is a multi-way split on nodes for information gain.

You should look for a C4.5 or C5.0 implementation of Decision tress in python.


Is it effective to use one-hot encoding when its dimension is as large as thousands

In neural networks applied to natural language processing normally each possible word (or sub-words) is handled as an individual token, and normally the vocabulary (the set of supported tokens) is around 32K. These tokens are typically one-hot encoded. Therefore, there is not inherent problem in having one-hot encoded vectors with thousands of components. However, the amount and variety of training data should support the dimensionality of the model for it not to overfit. Note that in NLP, these one hot encodings are used as expected outputs of the model, over which to compute maximum likelihood (categorical cross-entropy) together with the output of the model (a softmax).

Depending on how you are using this portfolio information, another option would be to have an embedding layer, which embeds discrete values into a continuous representation space (whose dimensionality is a hyperparameter of the model). This would be possible if this information is used as input to the model. In those cases, you input an integer number, which is used to index the embedded vector table.


Why there are specific number of layers with same depth in any CNN architecture?

There is no equation or theory to decide the number of layers at each depth in a neural network. The most common technique is trail and error. There can be systematic search through cross validation.
Generally more layers are added until performance stops improving.


Best approach would be to perform data preparation first:

Remove features (columns) with no variance in it (you could use: sklearn feature_selection)
one-hot-encoding of categorical features
insert a lag column of -t steps

If you have more than one explanatory variable, the process is called multiple linear regression. Instead of using a regression model you could also use other learners like XGBoost or LSTMs


The Convolution layers do the job of feature extracting. Using those feature the DNN layers try to do the classification task. The 512 layer does the job of a feature selector like which feature is relevant for a class or not while last layer is just calculate sigmoid probability.


Hyperparameter tuning and cross validation

Some of the popular ways of splitting of data that the user can validate a model:

Train-Test (Most popular)
Train-Test-Validation
Train-Test-Development
Train-Test-Dev-Val

Every way has their own pros and cons. There is no one-size-fits-all approach for getting a perfect model. Choice is typically made by the developer considering following factors:

Size of data
Diversity of data
Computation budget
Efficiency
Necessity

But I would recommend K-fold CV is the best way to go with the basic train-test split model.

Thank you.


https://en.wikipedia.org/wiki/Overfitting

This model is over-fitting. Better train accuracy (and validation accuracy that gets worse with successive iterations) indicates over-fit.

For CNN Next steps should be to reduce complexity of the model and adding droputs / batch normalization.

Few articles on this :

https://towardsdatascience.com/handling-overfitting-in-deep-learning-models-c760ee047c6e

https://machinelearningmastery.com/how-to-reduce-overfitting-with-dropout-regularization-in-keras/


Which value of output should be taken in multiple sessions of training Neural Network

This is normal behaviour of most classifiers. You are not guaranteed 100% accuracy in machine learning, and a direct consequence is that classifiers make mistakes. Different classifiers, even if trained on the same data, can make different mistakes. Neural networks with different starting weights will often converge to slightly different results each time.

Also, perhaps in your problem the classification is an artificial construct over some spectrum (e.g. "car" vs "van" or "safe" vs "dangerous") in which case the mistake in one case is entirely reasonable and expected?

You should use the value from the classifier that you trust the most. To establish which one that is, use cross-validation on a hold-out set (where you know the true labels), and use the classifier with the best accuracy, or other metric, such as logloss or area under ROC. Which metric you should prefer depends on the nature of your problem, and the consequences of making a mistake.

Alternatively, you could look at averaging the class probabilities to determine the best prediction - perhaps one classifier is really confident in the class assignment, and the other is not, so an average will go with the first classifier. Some kind of model aggregation will often boost accuracy, and is common in e.g. Kaggle competitions when you want the highest possible score and don't mind the extra effort and cost. However, if you want to use aggregation to solve your problem, again you should test your assumptions using validation and a suitable metric so you know whether or not it is really an improvement.


Similar output from random forest and neural network

I think that your neural network is probably learning the same features as your Random Forest. I do not think it is obvious because it is hard to say what the neural network is learning, but in this case based on the your model it might be working similar.


Linear Regression not working due to wrong kind of array

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

data = pd.read_csv("http://www.statsci.org/data/general/uscrime.txt", sep="\t")
x = data.loc[:, data.columns != 'Crime'].to_numpy()
y = np.squeeze(data.loc[:,'Crime'].to_numpy())

regression = LinearRegression()

regression.fit(x, y)

scikit learn expects numpy arrays and not pandas dataframes. You need to convert from one to the other on top, you need to make sure that the array for y only has one dimension which I achieved via np.squeeze.
Bonus: see above how you can directly load the csv from the website.


A cursory Google search gave me:

http://noesis.ikor.org/datasets/link-prediction
http://ial.eecs.ucf.edu/travian.php
http://konect.uni-koblenz.de/networks/

Another way would be to create your own dataset by downloading say Github user's graph, defining a network by adding edges  between user A and user B if A follows B and so on. Then you can take snapshots of this graph at different time instances and see if you can use last t snapshots to predict the edges in the t+1th snapshot. Of course that would be a lot more extra effort and the links I posted may suffice. Please let me know if it helps.


Latent Dirichlet Allocation vs Hierarchical Dirichlet Process

HDP is an extension of LDA, designed to address the case where the number of mixture components (the number of "topics" in document-modeling terms) is not known a priori.  So that's the reason why there's a difference.

Using LDA for document modeling, one treats each "topic" as a distribution of words in some known vocabulary.  For each document a mixture of topics is drawn from a Dirichlet distribution, and then each word in the document is an independent draw from that mixture (that is, selecting a topic and then using it to generate a word).

For HDP (applied to document modeling), one also uses a Dirichlet process to capture the uncertainty in the number of topics.  So a common base distribution is selected which represents the countably-infinite set of possible topics for the corpus, and then the finite distribution of topics for each document is sampled from this base distribution.

As far as pros and cons, HDP has the advantage that the maximum number of topics can be unbounded and learned from the data rather than specified in advance.  I suppose though it is more complicated to implement, and unnecessary in the case where a bounded number of topics is acceptable.


If There is a case where decision trees are getting overfitted so by using gradient boost method do we solve that problem?

If your trees are overfitting you should probably check the hyperparameters of the tree. You shouldn't let the tree have leaves with too few samples, for example, since that would mean that the tree is memorizing every possible outcome of the training set, in other words, it's overfitting.

Maybe if you share some more information of your case we could help a little further, but I think you now at least have something to check.

Hope this helps!


OPTICS gets rid of $\varepsilon$, you might want to have a look at it. Especially the reachability plot is a way to visualize what good choices of $\varepsilon$ in DBSCAN might be.

Wikipedia (article) illustrates it pretty well. The image on the top left shows the data points, the image on the bottom left is the reachability plot:

The $y$-axis are different values for $\varepsilon$, the valleys are the clusters. Each "bar" is for a single point, where the height of the bar is the minimal distance to the already printed points.


Low scale ML/statistical techniques for data poor settings

Generally, I'd pick a very simple, transparent/explainable model and use the results in a semi-automated way. That is, do not just derive a prediction but rather insights. You could, for example, use a (or multiple) decision tree(s) which you pre or post prune. The result could be a tree with, let's say, just 1-3 features to find simple rules like "if a customer is married and at least X years old, they have a high chance of making a purchase". With logistic regression you may use coefficients to identify features which influence the dependent variable the most. These (qualitative or semi-quantitative) rules should then be validated with domain experts.
Moreover, you need to be transparent about the accuracy and precision of your estimates. In the above example, leave purity would provide some intuition. If you report any quantitative measures (which I'd be careful with), you may want to consider confidence intervals (see here or chapter 5 in Tom Mitchell's "Machine Learning", for example). (With only 10 samples typical assumptions about normal distribution will not hold here though)
Regarding the time series I would start even simpler. Depending on the number of customers, I'd start by visualizing some or all historical data in a line plot (sales per customer over time) and check the min, max and mean per customer. This gives some intuition regarding potential trends. For example, if all observations remain constant over time for a given customer, whether there is an upward/downward trend or if the data has high variance with no clear trend. Also, there may be clusters of customers which show similar patterns. Obviously, this is neither Machine Learning nor a rigorous statistical analysis but rather a pragmatic approach supported by some basic data analytics.
What you need to be very careful with is the time horizon of any quantitative prediction: based on 10 observations at $t \in \{1,21,...,201\}$ you may derive some conclusions for, let's say, $t<=301$ (to make up a total out of the blue ballpark figure) but $t=621$  is very far in the future. Also, you need to keep seasonality in mind. For example: If your observations are all from October to April of a given year and assuming you have a winter/summer seasonal pattern. Then you cannot infer a lot for months May to September. To understand the limitations and forecast potential of your time series better, I'd speak to subject matter experts, e.g. in sales & marketing. It could also be helpful to understand their forecasting approach and cross check any insights you derive with their predictions.
But, as Erwan pointed out, be very careful to derive conclusions. Applying some "ML magic" will not find a useful pattern if there insufficient data to find any signal. And of course additional data collection would be reasonable if that is an option.


Recommendation model that can recommend already bought item

Usually recommendation algorithms provides the confidence that a user will like an item. Items that the user already bought should get high confidence - the user bought them since he liked them.

In most cases filtering out item the user already bough is done in the application level, not in the algorithm level.

So , you can use regular recommendation algorithms in order to know if the user will like the item.

Please note that you might face a different problem - will the user buy the item for the second/third time.
In order to cope with this problem, using some domain knowledge might be beneficial. If the products are usually bough many times (e.g., milk), just use the recommendation algorithm. If all the products have the same tendency to be bought few times, build a model for that (e.g., probability for buying another time given already buying x times) and combine the models.
If your products are very different from this aspect you might need to get into a lower level. Different domains require different solution but you might split the product by the buying behavior and train few recommendation systems, add the number of buying as a feature, etc.


Constituency vs Dependency Parsing: What is more effective for Sentiment Analysis?

Both approaches have been used for sentiment analysis in the literature. From a quick search, we can find these results:

Constituency parsing:

Deep Recursive Neural Networks for Compositionality in Language [NeurIPS'14]
Less Grammar, More Features[ACL'14]
A Statistical Parsing Framework for Sentiment Classification [Computational Linguistics journal'15]

Dependency parsing:

What's great and what's not: learning to classify the scope of negation for improved sentiment analysis [NeSp-NLP'10]
UBham: Lexical Resources and Dependency Parsing for Aspect-Based Sentiment Analysis [SemEval'14]

It seems to me that constituency parsing has been used more than dependency parsing. My sample, however, is very small and focused on the English language. I think this may change in languages with more complex (and more non-projective) syntax.
That been said, I think currently neither of them are considered to be state of the art in sentiment analysis. If we take a look at the leaderboards of sentiment analysis tasks from paperswithcode, all of the leading approaches (BERT, RoBERTa, M5) handle text as a mere sequence of tokens. While these results are from only a


XGBoost: Quantifying Feature Importances

In xgboost 0.7.post3:

XGBRegressor.feature_importances_ returns weights that sum up to one.
XGBRegressor.get_booster().get_score(importance_type='weight') returns occurrences of the features in splits. If you divide these occurrences by their sum, you'll get Item 1. Except here, features with 0 importance will be excluded.
xgboost.plot_importance(XGBRegressor.get_booster()) plots the values of Item 2: the number of occurrences in splits.
XGBRegressor.get_booster().get_fscore() is the same as XGBRegressor.get_booster().get_score(importance_type='weight')

Method get_score returns other importance scores as well. Check the argument importance_type.

In xgboost 0.81, XGBRegressor.feature_importances_ now returns gains by default, i.e., the equivalent of get_score(importance_type='gain'). See importance_type in XGBRegressor.

So, for importance scores, better stick to the function get_score with an explicit importance_type parameter.

Also, check this question for the interpretation of the importance_type parameter: "weight", "gain", and "cover".


Optimal parameter estimation for a classifier with multiple parameters

The answer depends a lot on what you want to achieve:

Please future reader's of your machine learning paper OR
Solve a use-case that matters to some business

In first case, you have to scan this audience's publications, see what they use, and use the same. There isn't a perfect metric, since actually it depends on the business' cost-ratio.

In the second case, all you need is the cost-matrix and the result follows naturally. Optimizing anything which isn't the cost-matrix is a rather academical exercise. The cost-matrix has one parameter and depends really on the business. No reference is going to solve this decision for you. The discussion between performance metrics is missing the point. There is only one answer once you manage to squeeze out a cost-matrix from your business stakeholders. In a business anything can be estimated in terms of money. It only gets tricky, if it more complex than a single class weight.

So the answers are:

Parameter performance: calculate cost; Overall performance: take max(), or to be more precise correct for testing multiple hypotheses, too
There isn't any, unless you find one which proves that optimizing a metric optimizes for all possible cost matrices simultaneously.

In that sense, ROCAUC is actually a pretty meaningless artifical metric, despite being popular. There is an interpretation which relies on the fitted model's probabilities being correctly estimated, but if you want to get real, only cost matrices matter. Model the real world impact of you model and you'll get answers!

PS: I personally consider TPR-FPR the best bet in absence of a defined cost matrix.


Kmeans cluster validation when I have labeled test data

In order to keep the same dimension for both training & testing set, you need apply your fit on the training set with you tokenization, tf-idf etc..
You need to apply same treatments as you do on training, on the testing set and then this will be okay :

from sklearn.feature_extraction.text import CountVectorizer

corpus_train = [
    'This is the first document.',
    'This document is the second document.'
]
corpus_test = [
    'And this is the third one.',
    'Is this the first document?',
]

vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(corpus_train)
print(vectorizer.get_feature_names())

# Returns
['document', 'first', 'is', 'second', 'the', 'this']

So your training set looks like this :

print(X_train.toarray())
# Returns
[[1 1 1 0 1 1]
 [2 0 1 1 1 1]]

print(X_train.shape)
# Returns
(2, 6)

Then just transform your testing set with the same vectorizer :

X_test = vectorizer.transform(corpus_test)
print(X_test.toarray())
# Returns
[[0 0 1 0 1 1]
[1 1 1 0 1 1]]

print(X_test.shape)
# Returns
(2, 6)

So you get your dimension of 6 (the number of features) for both training & testing set :)


Interpreting confidence interval results for datasets

No, that isn't what it means.
For one thing it is not clear what parameter the confidence interval that you calculated is for.
In any case, some care is needed in the interpretation of (frequentist) confidence intervals.
In frequentist statistics, a confidence interval is random, and the parameter that the interval is for is fixed. In the case of a 99% interval this means that if the data were collected again, many times, and the confidence interval re-calculated each time, then 99 times out of 100 it would contain the true value of the parameter. This is the only technically correct interpretation of the frequentist confidence interval. It is often interpreted, incorrectly, as being an interval that contains the parameter with 99% probability, and that appears to be the interpretation that you are using.


Implementation of a distributed data mining paper

Splitting your dataset

First you can load your data set to memory:

# Load the data to memory
with open('file.txt', 'r', encoding='utf-8') as file:
    header = file.readline()
    data = []
    for line in file.readlines():
        data.append(line)

Next, you use KFold to split your data into $10$ splits and save to $10$ different files.

# Split the data and save the data
from sklearn.model_selection import KFold
kf = KFold(n_splits=10)

split = 1
for _, split_index in kf.split(data): # Iterate over the splits

    with open('file_{}.txt'.format(split), 'w+', encoding='utf-8') as file: # Open a file with name file_{split number}
        file.write(header) # write the header of the file
        for index in split_index: # Write each point
            file.write(data[index])
    split += 1 # Increase split number

Weka4WS

The Weka4WS is an extension to Weka itself. Yes, it is like an application like Weka, with a Graphical User Interface. From the docs:

  Thus, Weka4WS also extends the Weka GUI to enable the invocation of the data mining algorithms that are exposed as Web Services on remote Grid nodes.

However, it is now as easy to run as the normal Weka. You need some additional steps to use it.

The official installation is here: http://scalab.dimes.unical.it/weka4ws/howtos/installation/

Here is a tutorial to run the application on Windows: http://scalab.dimes.unical.it/weka4ws/howtos/run-the-client-on-windows/

This is how the app looks like, this is the KnowledgeFlow screen:


What is the meaning for the coef and intercept

A (binary) logistic regression algorithm tries to determine whether the data $x$ belongs to class 0 or class 1 by the value
$f(x)=\omega x+b$. If the value $f(x)>0$, the algorithm believe $x$ is more likely to be in class 1; while if $f(x)<0$, then $x$ is more likely to be in class 0.

In your code alg.coef_ is the $\omega$ above, and alg.intercept_ is the $b$ above.

Why everything is predicted to be in class 0

If you run your code, you should see the result alg.coef_=-0.354, alg.intercept_=0.307. Therefore your alg is calculating $f(x)=-0.354x+0.307$. Now if you plug in your data, i.e. $x=$1, 2, 1.5 and 2.5 respectively, you should get the value $f(x)=$-0.047,-0.401,-0.224 and -0.579 respectively.

As you see, all the four $f(x)$ values are smaller than 0, therefore your alg determines that they all belong be class 0.

But still why? Why the result is not what I expected?

Here is were regularization comes in. In the context of logistic regression, the learning algorithm regularizes that the learned parameter, $w$, should not be too large. To be specific, by default it regularizes the $l_2$ norm norm of $w$. In your case $w$ is a scalar, then $l_2$ norm is its absolute value.

In general, regularization is used to prevent overfitting. But that's another broad topic.

If you want your algorithm to behave as you expected (classify 1 and 1.5 to class 1, and classify 2 and 2.5 to class 0) , there are 2 ways.

You tell you program to use extremely weak regularization.
That can be achieved by passing a large value to the C parameter in alg = LogisticRegression() (default value is 1.0, see detail), e.g.

alg = LogisticRegression(C=1000)

In this case your algorithm should return alg.coef_=-9.894, alg.intercept_=17.156, and it can classify your four data points correctly. Notice that the absotely value of alg.coef_ is now much larger than before.

Provide your algorithm with more data.

For example, simply repeat your four data points 100 times:

data   = [x[:-1] for x in points]*100
target = [int(x[-1]) for x in points]*100

Then you can get alg.coef_=-4.438, alg.intercept_=7.594 without setting C. In this case the same default regularization strength is still applied, but the additional data provide stronger evidence for your algorithm to believe that the alg.coef_ should be allowed larger (in absolutely value).


Imagine you have 1500 labeled data points, and you want to estimate how well some classifier will work on new data. One (naive) method would be to train a model with all 1500 of your data points, and then check how many of the 1500 data points were classified correctly. This is not likely to give a good estimate of the performance on new data, because new data was not used to test the model. Some models like decision trees and neural networks will often be able to get 100% accuracy on the training data, but perform much worse on new data.

So you think to yourself that you will split the data into two sets - a training set which you will build a model with, and a testing set that you will use to evaluate the model. Lets say you decided to train the model with 1000 of your examples, and evaluate with 500. This should give a reasonable estimate of how well your model will perform on new data, but it seems a bit limited; after all, one third of your data has not been used for training at all! We only have predictions for the 500 test samples - if these ones randomly happened to be easier to classify correctly on average, then our performance estimate is overly optimistic.

Cross validation is a way to address this. Lets set $k=3$, so the data is split into three sets of 500 points (A, B and C). Use A & B to train a model, and get predictions for C with this model. Use B & C to train a model, to get predictions for A. Finally, use A & C to train a model, and get predictions for B. Now we have a prediction for every point in our labeled data that came from a model trained on different data. By averaging the performance of each of these models, we can end up with a better estimate of how well the model will perform on new data.

Note that you should then re-train your model using all 1500 labeled points if you want to apply it to new data. Cross validation is only for estimating the performance of this new model. Also if your data is large enough, cross validation is probably unnecessary and you could just make a single train/test or train/valid/test split.


Using sigmoid in binary DNN output layer instead of softmax?

In a binary classification problem you have only 2 classes, let's call them the negative and the positive class.
You only need to ouptut 1 number which corresponds to the probability of your input point to belong to the positive class.
The sigmoid activation function is good for that because it maps any input value to the range ]0,1[ which is what we want for a probability (it is not a real issue that 0 and 1 are excluded).

Since you have only one output number, it makes no sense to use a softmax activation.
Softmax activation is used in the multiclass problem where you must predict 1 of N classes where N is greater or equal than 3 and in this case the number of outputs is N (1 probability by class).
The softmax function makes that all your outputs sum to 1 and it amplifies the gap between high and low probabilities.


When should I use StandardScaler and when MinMaxScaler?

StandardScaler and MinMaxScaler are more common when dealing with
continuous numerical data.

One possible preprocessing approach for OneHotEncoding scaling is "soft-binarizing" the dummy variables by converting softb(0) = 0.1, softb(1) = 0.9. From my experience with feedforward Neural Networks this was found to be quite useful, so I expect it to be also benefitial for your MLPClassifier.
StandardScaler is useful for the features that follow a Normal distribution. This is clearly illustrated in the image below (source).

MinMaxScaler may be used when the upper and lower boundaries are well known from domain knowledge (e.g. pixel intensities that go from 0 to 255 in the RGB color range).


Data science and MapReduce programming model of Hadoop

Let's first split it into parts.

Data Science is about making knowledge from raw data. It uses machine learning, statistics and other fields to simplify (or even automate) decision making. Data science techniques may work with any data size, but more data means better predictions and thus more precise decisions.

Hadoop is a common name for a set of tools intended to work with large amounts of data. Two most important components in Hadoop are HDFS and MapReduce.

HDFS, or Hadoop Distributed File System, is a special distributed storage capable of holding really large data amounts. Large files on HDFS are split into blocks, and for each block HDFS API exposes its location.

MapReduce is framework for running computations on nodes with data. MapReduce heavily uses data locality exposed by HDFS: when possible, data is not transferred between nodes, but instead code is copied to the nodes with data.

So basically any problem (including data science tasks) that doesn't break data locality principle may be efficiently implemented using MapReduce (and a number of other problems may be solved not that efficiently, but still simply enough).

Let's take some examples. Very often analyst only needs some simple statistics over his tabular data. In this case Hive, which is basically SQL engine over MapReduce, works pretty well (there are also Impala, Shark and others, but they don't use Hadoop's MapReduce, so more on them later).

In other cases analyst (or developer) may want to work with previously unstructured data. Pure MapReduce is pretty good for transforming and standardizing data.

Some people are used to exploratory statistics and visualization using tools like R. It's possible to apply this approach to big data amounts using RHadoop package.

And when it comes to MapReduce-based machine learning Apache Mahout is the first to mention.

There's, however, one type of algorithms that work pretty slowly on Hadoop even in presence of data locality, namely, iterative algorithms. Iterative algorithms tend to have multiple Map and Reduce stages. Hadoop's MR framework reads and writes data to disk on each stage (and sometimes in between), which makes iterative (as well as any multi-stage) tasks terribly slow.

Fortunately, there are alternative frameworks that can both - use data locality and keep data in memory between stages. Probably, the most notable of them is Apache Spark. Spark is complete replacement for Hadoop's MapReduce that uses its own runtime and exposes pretty rich API for manipulating your distributed dataset. Spark has several sub-projects, closely related to data science:

Shark and Spark SQL  provide alternative SQL-like interfaces to data stored on HDFS
Spark Streaming makes it easy to work with continuous data streams (e.g. Twitter feed)
MLlib implements a number of machine learning algorithms with a pretty simple and flexible API
GraphX enables large-scale graph processing

So there's pretty large set of data science problems that you can solve with Hadoop and related projects.


How to evaluate the quality of speech-to-text data without access to the true labels?

There is at least one way:

Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
Test the transcripts for beign grammaticaly/syntacticaly correct.
This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken


When is it necessary to use StandardScaler/MinMaxScaler on y_train and y_test?

As far as I know, scaling target values is not the common practice, though I do not expect it to change results much. Here is a more detailed discussion of this issue:

https://stats.stackexchange.com/questions/111467/is-it-necessary-to-scale-the-target-value-in-addition-to-scaling-features-for-re


Which graph will be appropriate for the visualization task?

Maybe you can try something like this?

df.hist(by='country', bins=50)

If you want to plot these histograms in the same figure, checkout this.

References

http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.hist.html


How to understand equations in research papers?

Study multiclass logistic regression; it's similar and has many tutorials. Good books include Larry Wasserman's All of Statistics, and An Introduction to Statistical Learning.

The way to understand research papers is simply to read more of them. Whenever you encounter something you don't understand follow the references or look it in one of the aforementioned books.


How do I find the minimum distance between zip codes in R?

I think I have read your question correctly it looks like you need a nearest neighbor implementation. If you are unfamiliar with the concept you can find the wiki article here https://en.wikipedia.org/wiki/Nearest_neighbor_search.

I went ahead a wrote an example implementation you can use as a guide. Please note that this is a brute force method and not useful for big data sets. Once you have a grasp of the material I suggest checking out some libraries like RANN that have "real" implementations.

Read in some random test data and clean
   For this test let us assume that we want to find the
   closest AMERICAN city for each location

coord_data = read.csv("~/Downloads/SalesJan2009.csv", stringsAsFactors = F)
coord_data$id = c(1:nrow(coord_data))
coord_data$is_usa = ifelse(coord_data$Country == "United States", 1, 0)
coord_data = coord_data[ , c("id", "Latitude", "Longitude", "is_usa")]
names(coord_data) = tolower(names(coord_data))

Define your distance function.
         Here we have geo-coordinates over long distance so Euclidean will not do.
         I am using the Law of Cosines to calculate great circle distance
        but Haversine and Vincenty should be considered given your needs.
        To learn more start here: https://en.wikipedia.org/wiki/Great-circle_distance.

greatCircleDistance = function(latAlpha, longAlpha, latBeta, longBeta, radius = 6371) {
  ## Function taken directly from Wikipedia
  ## Earth radius in km is default (6371)
  ## Long/Lats are in degrees so need helper function to convert to radians
  degreeToRadian = function(degree) (degree * pi / 180)
  deltaLong = degreeToRadian(longBeta) - degreeToRadian(longAlpha)
  sinLat = sin(degreeToRadian(latAlpha)) * sin(degreeToRadian(latBeta))
  cosLat = cos(degreeToRadian(latAlpha)) * cos(degreeToRadian(latBeta))
  ## acos is finicky with precision so we will assume if NA is thrown
  ## the argument was very close to 1 and therefore will return 0
  ## acos(1) == 0
  acosRaw = suppressWarnings(acos(sinLat + cosLat * cos(deltaLong)))
  acosSafe = ifelse(is.na(acosRaw), 0, acosRaw)
  acosSafe * radius
}

Distance between Basildon, UK and Parkville, US

greatCircleDistance(coord_data$latitude[1],
coord_data$longitude[1],
coord_data$latitude[2],
coord_data$longitude[2])

 Returns [1] [1] 6929.351 km.

It matches Google's calc so we are good to go!

Brute Force Example:
     As you noticed in your Excel sheet this will blow up quickly as data set gets larger. There are much more efficient ways of implementing the search. One idea is to start with the the geo-data structure itself and write an R-Tree, but I'll leave that for you.

 bruteForceNearestNeighbor = function(geoData) {
      makeCoordinate = function(idx) {
        c("id" = idx, "latitude" = geoData$latitude[idx], "longitude" = geoData$longitude[idx])
      }
      singleCoordMinDistance = function(coordinate, locations) {
        locationsUS = locations[locations$is_us == 1 & locations$id != coordinate["id"], ]
        distances = mapply(greatCircleDistance,
              latAlpha = coordinate["latitude"],
              longAlpha = coordinate["longitude"],
              latBeta = locationsUS$latitude,
              longBeta = locationsUS$longitude)
        closestIndex = which(distances == min(distances))
        locations[closestIndex, "id"]
      }
      nearestNeighbors = vector("numeric", nrow(geoData))
      for ( i in 1:nrow(geoData) ) {
        coord = makeCoordinate(i)
        nearestNeighbors[i] = singleCoordMinDistance(coord, geoData)
      }
      nearestNeighbors
    }

    coord_data$nearest_neighbor = bruteForceNearestNeighbor(coord_data)


I would try the following R-Code for subsetting:

World[World$isOfficial != 't' & World$percentage > 50, "country.y"]

And in case isOfficial is type logical, I would try this R-code:

World[!(World$isOfficial) & World$percentage > 50, "country.y"]


Should bag of words in training set include test set data when doing text classification?

This is the problem of out of vocabulary (OOV) words.
As a rule the training should not use anything from the test set for several reasons:

The risk of data leakage, which would cause an overestimated performance on the test set.
During training the model cannot use these words to distinguish between classes anyway, since they are not present. So it would be pointless to include them.
In principle the model is meant to be used with any new input text, not only the documents in the current test set. Using the test set words as features in the model actually restricts the model, since it cannot be applied to any other text than the test set.

The correct way to deal with OOV words is either:

To simply ignore them completely, i.e. filter them out before applying the model.
To account for this possibility in the model from the start. Typically this can be done by having a special token UNKNOWN. This option is often combined with filtering out rare words from the training set: every occurrence of a rare words can be replaced by the UNKNOWN token.

The cause of the higher performance when including these features is unclear. I suspect that the inclusion of these features causes the model to be different in a way which happens by chance to have a positive effect on the test set. It's quite unlikely in general, I'm not sure.


You should provide more information about your problem. How large is your data set? Do you have hyper-parameters? What is your classifier?

Normally, you should split your entire data set into three parts: training, validation and testing sets. The testing set you should never use before the model is finally tuned. If you data set is not large enough, you should you use the cross-validation method to measure the error of your classifier.

There are several metrics to assess a performance of your classifier: accuracy, precision and recall, Cohen's kappa, F1-score, etc.

If you are working with python and scikit-learn, I would recommend to check this book. It is nicely written and explains basics of model assessments.


How to select features which have non zero importance using SKLearn's Decision Trees Classifier?

You should use a sklearn.feature_selection.SelectFromModel instance: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html

You can specify the threshold that you want when you want to select features. You can add this transformer in a scikit-learn pipeline as well.


Why is ReLU used as an activation function?

In mathematics (linear algebra) a function is considered linear whenever a function$f: A  \rightarrow B$ if for every $x$ and $y$ in the domain $A$ has the following property:  $f(x) + f(y) = f(x+y)$.  By definition the ReLU is $max(0,x)$.  Therefore, if we split the domain from $(-\infty, 0]$ or $[0, \infty)$ then the function is linear.  However, it's easy to see that $f(-1) + f(1) \neq f(0)$.  Hence by definition ReLU is not linear.
Nevertheless, ReLU is so close to linear that this often confuses people and wonder how can it be used as a universal approximator.  In my experience, the best way to think about them is like Riemann sums.  You can approximate any continuous functions with lots of little rectangles.  ReLU activations can produced lots of little rectangles.  In fact, in practice, ReLU can make rather complicated shapes and approximate many complicated domains.
I also feel like clarifying another point.  As pointed out by a previous answer, neurons do not die in Sigmoid, but rather vanish.  The reason for this is because at maximum the derivative of the sigmoid function is .25.  Hence, after so many layers you end up multiplying these gradients and the product of very small numbers less than 1 tend to  go to zero very quickly.
Hence if you're building a deep learning network with a lot of layers, your sigmoid functions will essentially stagnant rather quickly and become more or less useless.
The key take away is the vanishing comes from multiplying the gradients not the gradients themselves.


Apply GroupByKey to identify the Frequently Products Purchase Together

Since you are using Spark 1.6, I'd rather do these kind of transformations with DataFrame as it's much easier to manipulate.

You'll need to use SQLContext implicits for this :

import sqlContext.implicits._ // not need in spark-shell

Now, let's create some dummy data just to follow the code snippet that you have provided :

scala> val data = sc.parallelize(Seq("1,2,3,4", "2,3,4,5", "1,3,4,5", "1,6,6,7"))
// data: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[13] at parallelize at <console>:27

Back to your code, we define a case class Transactions and the CSV converter :

scala> case class Transactions(Transaction_ID: String, Customer_ID: String, Department: String, Product_ID: String)
// defined class Transactions

scala> def csvToMyClass(line: String) = { val split = line.split(','); Transactions(split(0), split(1), split(2), split(3)) }
// csvToMyClass: (line: String)Transactions

We will use implicits now to convert our data into a DataFrame after converting it to Transactions :

scala> val df = data.map(csvToMyClass).toDF("Transaction_ID", "Customer_ID", "Department", "Product_ID")
// df: org.apache.spark.sql.DataFrame = [Transaction_ID: string, Customer_ID: string, Department: string, Product_ID: string]

Let's take a look at the DataFrame :

scala> df.show
// +--------------+-----------+----------+----------+
// |Transaction_ID|Customer_ID|Department|Product_ID|
// +--------------+-----------+----------+----------+
// |             1|          2|         3|         4|
// |             2|          3|         4|         5|
// |             1|          3|         4|         5|
// |             1|          6|         6|         7|
// +--------------+-----------+----------+----------+

Now all we have to do is a simple group by and perform a collect_list aggregation on the first dataframe :

scala> val df2 = df.groupBy("Transaction_ID").agg(collect_list($"Product_ID"))
// df2: org.apache.spark.sql.DataFrame = [Transaction_ID: string, collect_list(Product_ID): array<string>]

We can check the content of our new DataFrame df2 :

scala> df.groupBy("Transaction_ID").agg(collect_list($"Product_ID")).show
// +--------------+------------------------+
// |Transaction_ID|collect_list(Product_ID)|
// +--------------+------------------------+
// |             1|               [4, 5, 7]|
// |             2|                     [5]|
// +--------------+------------------------+

I hope that this answers your question.

Note: If you wish to know what's the difference between RDD and DataFrames, I advice you to read Databrick's blog entry about it here.


Predict buying behavior under the condition that a customer is advertised or not

It depends what you want to do. If you want a model that will predict who to target/not-target for a marketing campaign, then you want to train a model using only the people who were marketed to, which defines your two classes "responders" and "nonresponders". Given an unseen customer, your classifier will then determine whether they are likely to respond if you send them a directed marketing advertisement. Take a look at uplift marketing.


For every Bayesian Network, is there a Neural Network that gives the same output?

We don't have to do anything crazy to prove this, we can simply leverage the UAT.

By the universal approximation theorem, yes, you could concieve of a neural network that precisely matched your Bayesian network.


How to make predictions based on correlations?

The correlation coefficient will tell you, how much predictive power the individual attributes posess with respect to profit in a linear sense.
Easier: The correlation coefficient tells you how line-like your data is.
( There might still be a very predictive arc like structure in your data which the correlation coefficient can't capture )

So you can either calculate a regression line for profit with the attribute having the highest correlation coefficient, or you can calculate a hyperplane for all three attributes to get the desired prediction. Regression lines and hyperplanes are usually calculated by least-squares fitting.

But beware to overestimate predictive power, because it is extrapolation what you do. New data might behave different than old data which makes extrapolation potentially less exact the further you are away from known data.


Google Colab conda packages permanently installation

There is no way to permanently install packages because the code is run on virtual machines that are sometimes recycled. You'll need to install it each time.
From the FAQ:

Where is my code executed? What happens to my execution state if I close the browser window?
Code is executed in a virtual machine dedicated to your account. Virtual machines are recycled when idle for a while, and have a maximum lifetime enforced by the system.


Which dataset I should use when I am retraining my model?

So, I understand the question is asking what data should you use, when given new data.
You might want to provide more context to your question as this might seem like a loose answer. The way you go forward with this depends on a couple of factors:

Is the initial dataset ($D_1$) have the same features/columns as the new dataset ($D_2$), such that you can integrate them (as you said). This obviously can help to answer whether to use either $D_1$/$D_2$ or both.
It also depends on the problem specification: if this new dataset provides new features (maybe more accurately measured), which are more appropriate to the problem spec, then obviously you should then work with $D_2$ and train the model with this dataset. If the problem spec can be solved using both datasets, then use both as this can help to improve generalisation performance more so than using $D_2$ alone.


How to choose a kernel function and a feature mapping function?

To answer your questions:

Yes, $\phi((x_1,x_2))=(x_1,x_2,(x_1-2.5)^2)$ is a good choice. You can also remove the second dimension $x_2$, as it's zero for all of your training examples. By doing so, you will have a mapping function $\phi:\mathbb{R}^2 \to \mathbb{R}^2$ which is: $\phi((x_1,x_2))=(x_1,(x_1-2.5)^2)$. (Also note that mapping into higher dimension in not necessary; mapping to a space where data becomes linearly separable is sufficient.)
Kernel function $K$ calculates the dot product of points in the new space, so we have:

$$K(x,y)=\left\langle \phi(x),\phi(y)\right\rangle=\left\langle (x_1, (x_1-2.5)^2),(y_1,(y_1-2.5)^2)\right\rangle=x_1y_1+(x_1-2.5)^2(y_1-2.5)^2 \to K(x,y)=x_1y_1+((x_1-2.5)(y_1-2.5))^2$$


What's The Difference Between The Terms Predictor And Feature

Feature and predictor are used interchangeably in machine learning today though I must admit that it seems that feature is being used more than predictor. The definition is the one on Wikipedia which you have already mentioned. The term predictor comes from statistics, here one definition:

An independent variable, sometimes called an experimental or predictor variable, is a variable that is being manipulated in an experiment in order to observe the effect on a dependent variable, sometimes called an outcome variable.

and my favorite definition:
A predictor variable explains changes in the response.

In a nutshell:
X columns: features, predictors, independent variables, experimental variables.
y column(s): target, dependent variable, outcome, outcome variable.


Can the attention mask hold values between 0 and 1?

In theory maybe yes, but you would probably need to reimplement the model yourself.
In practice, with the current implementations, probably no. (Judging from the documentation snippet, you use Huggingface Transformers.) The documentation says it expects a LongTensor, i.e., a tensor with integer values. Internally, the attention mask is used to compute sequence lengths, but summing the mask along dimension 1. This would need to be fixed and there might many other places in the code just assume the mask values are zeros and ones.


As you can see, your predictions are able to catch the trends. In other words, the model is able to predict the direction of movement almost every day.
The only point that it is not able to catch is those high peaks, which can be treated as outliers. It is because due to seasonality or some other cause your daily data drastically changes on some of the days. This change is not normal for models to capture because those points deviate from the general characteristics of your time series.
It is quite normal having low energy consumption for 6 consecutive days but having a large energy consumption on the 7th day if you would consider that 6 days were sunny and suddenly the weather gets cold. This is just one single case where there might be lots of those.
To capture these anomalies you should have a variable to explain to them (e.g. Image that those anomalies are only due to weather conditions, then including weather variable would help you. However, it is too hard to find all those variables that explain all anomalies).
To solve the issue, you can decrease the frequency of your data. That is, instead of modeling daily data you can model weekly average or weekly end (means last day of every week), or even monthly average or monthly end.
If these outliers are consistent every week, averaging them will help you. In any case, I think decreasing the frequency of your data will help you.
To summarize, there is not a wrong thing with your model, it is just outliers make it unhandy in some predictions.


KNN is a classification algorithm where, typically, continuous variables are used to apply a classification. Your problem is a problem of predicting a continuous value using variables that are not continuous.

Your problem presents several challenges which, to me, would suggest KNN isn’t a good choice.

The biggest problem is probably the road_category where you have an ordinal value. Is category 4 four times greater than the difference between category 0 and category 1? Probably not.
No matter how deep your data, due to the fact that your predictors are integers or categorical, your neighbors won’t really be neighbors, more like roommates in a warehouse; you will probably have hundreds of observations on the same vector. So, now you have to choose k values from m observations. So, you will need to come up with an algorithm to do that. Then you’ll have to come up with a methodology to get a prediction. An average might work, but depending on your application, Min or max may be more beneficial.

Bottom line, KNN expects the variables to be close to the same order of magnitude otherwise, larger variables tend to determine the neighbors even if they are not necessarily more important. Some limited categorical variables might be accommodated using dummy variables, but I’d be really careful.

Based on your description of the problem, I think you should continue with regression, but use dummy variable encoding for your categorical values.


Containing multicomponent data in rows or columns

Note: This more of a comment to help your question being answered I will delete it if needed.
It is not clear to me which analysis you want to do with this dataset thus it is hard to advice on the structure and format of the dataset.
In my experience when dealing with DNA strings and their associated features, R and in particular the Bioconductor package are the best. These packages contain already data structure, functions and operations purely design to deal with bioinformatics tasks.The GenomicAlignements could be a good start.
Hope this helps!


Training my neural network to overfit my training dataset

If you want to overfit, then yes you just need to keep fitting the training data through your network until you reach as close to zero training loss as possible (note that zero loss is stronger than 100% prediction, and will result in a greater amount of overfitting). There is no need to use cross validation. If your network is sufficiently complex, then it won't take very long to reach very high accuracy on the training data.


LSTM with auxiliary data achieves abismal results

Is there a reason why this new information has to be a part of your model? There's no rule that says that all data must be fed into your model. Maybe you have just found a feature that shouldn't be included in your predictive phases? And if so, that's perfectly fine, there is nothing wrong with that.


Autoencoder or layer-based dimensionality reduction?

Autoencoder is a good option for doing dimensionality reduction, however, you will need another training for autoencoder.

If you want dimensionality reduction before feeding into a classification model, how about PCA and t-SNE? They don't require training process for doing dimensionality reduction.


Create an RNN on text sources with different lengths

I think that if you append a token <EOS> (end of sentence) at the end of each sentence when you merge, this would not be a problem, because the RNN would learn to cut sentences and to generate independently if you shuffle your data and train with several shuffles.
However, as you say your data is heterogeneous, you might consider to first run some clustering algorithm and then train your RNN to generate conditionally to a cluster. To build the training dataset, you can merge all data for each cluster. This is a bit more complex than simply "automatic generation" but it depends on what your goal is.


Using  Logistic regression or Linear regression depend on the dependent variable(DV). Based on your question, I believe that your DV will be infertitlity(Yes/No) so you should use logistic regression because linear regression is for the continuous variables(e.g:exam score) and logistic regression is for categorical variables(e.g.L Yes/NO)


How to use ML to forecast sales of a brand new product

You could use Bass or Gamma/Shifted Gompertz method to do that. You could use  diffusion package in R.

Here an example (it is form documentation of the package) :

library(diffusion)

fitbass <- diffusion(tsChicken[, 2], type = "bass")
fitgomp <- diffusion(tsChicken[, 2], type = "gompertz")
fitgsg <- diffusion(tsChicken[, 2], type = "gsgompertz")
# Produce some plots
plot(fitbass)

plot(fitgomp)

plot(fitgsg)

fc_bass <- predict(fitbass,h=20)

fc_gomp <- predict(fitgomp,h=20)

fc_gsg <- predict(fitgsg,h=20)

plot(fc_bass)

plot(fc_gomp)

plot(fc_gsg)

The only disadvantage of the methods is that they do not take care of seasonality of the sales of the products.


Creating features from raw accelerometer data

This is an active area of research called Human Activity Recognition. There are several public datasets available to cross-validate your methods, and you might want to start here: UCI HAR Dataset. There's a paper that accompanies the dataset that describes their preprocessing methods, so you'll want to have a look at that and see if anything helps in terms of feature engineering.
There are plenty of regression-based models that exist in the exercise science space, but I've had good results using 1D convolutional neural networks on very simply transformed data. We just downsampled the signal (by averaging) to 1 or 2 Hz and reduced the 3 axes to a single time series via euclidean vector magnitude (our subjects were performing higher-intensity activities, which often cause the sensor to rotate on their limb, changing the direction of each axis' input - reducing to EVM removes that potential confounding).
I'm not totally sure what your data looks like - do you have transitions (from sitting to standing and back)? I wouldn't use PCA (there is some argument that LDA can work if you're careful). There's something called tICA (time-series independent components analysis) but I don't think that's what you want because it's reducing the dimensionality at each time step (and each of your time steps is only 3 features). You're probably going to be going with fourier features (see the link above) or simple downsample / EVM.
Another concern here is that it's possible that the data isn't reliably separable because the intensity of each activity is low. How long is each "event" (I'm understanding each event to be a "bout" of activity with some duration)? My models have, in the past, had difficulty separating certain low-intensity activity bouts because if one person isn't moving standing up and another person isn't moving sitting down, the model can't tell 0 from 0 and is basically guessing.  Hopefully there's enough signal in there to get a good result, it just depends on your experimental setup.


How can I predict the last element of the fixed length=8 sequence after I get each element?

This can be modelled as a Markov Chain.
Let us say each element in your list can take N states. You can then represent a first order markov chain by build an N x N matrix - where each cell represents a state transition probability.
You can use this state transition probability to predict the most likely value X9 takes based on the known value from the previous timestep X8.
You also have the option of using a higher order markov chain - for e.g. a second order chain could allow you to predict X9 based on values from the previous 2 timesteps - X7 and X8.


Can the output of convolution on image be higher than 255?

It is possible.  You may have values less than 0 or greater than 255.  It will depend on the values in the kernel.  If you want to display the convolution output properly you will need to scale it first.  This process is referred to as 'normalization".


Using packages such as sklearn vs building ML algorithm from scratch

It depends entirely on your goal.

Student Phase  When you're learning about machine learning algorithms, I think it is a really good idea to implement toy examples.  I find this process helps with finding what you understood well and what you did not understand as well as you thought.  It's doing this work where you'll find deeper understanding of how the algorithm really works and different internal choices you have to make.

Professional Phase  When you have a project to deliver, you don't necessary want to rewrite a random forest implementation from scratch.  Even if you could build one in a reasonable amount of time, there's value in having something like sklearn that is well vetted and robust enough to handle edge cases that you wouldn't even consider.  That's the advantage of using pre-built libraries.

I need more Phase  Eventually, you get the point where you understand the math and know how to use the packages well and you realize that there's a feature lacking.  That's when you rip open a framework like xgboost or sklearn and modify existing code or even create you own implementation.  The reason you would do this is because methods are cutting edge so there just isn't anything out there or the implementation of the framework is actually a handicap in production (as I tend to find with sklearn).

The issue you seem to be facing is lack of accountability for your output.  If all you do is clean data push it through a model and get good results, make a chart showing results, I would say you are forgetting the "science" part in data science.

The challenging part isn't using the model it is knowing what moves your model and the potential hurdles your model may face in the real world.  I've seen this play out often in my career, where a junior member will make an awesome model on training and test dataset, and suddenly production comes along and performance tanks.  Why would that happen?  Well, because test and training were similar (and often from the same source), but the junior member failed to question the lack of variety in the data source and question if the real world behaved the same way.

What I am trying to say is, being a data scientist, a small part of the job is cleaning data, running model and making pretty pictures.  The real challenge is asking the why question.  Why does this work?  Why does the model perform poorly?  Why does the model perform well?  Why is this feature important?


One idea would be to plot the daily average power consumption in a bar plot:

For a finer visualization of day-hour peaks, you can try to plot it in a matrix format:


How to Bootstrap dataset for 10000 AUC scores?

In this question of stats exchange you can see an answer to your question of when to use bootstrap over CV.

You can see a simple tutorial of how to do Bootstrap in Python in this link

How to generate 10k AUC Scores?
AUC is a performance metric and what you are going to measure is the performance of your model 10k times. For that, you have to select 10k times the number of samples that you consider and measure AUC

for i in range(0,10_000):
    sample = df.sample(df.shape[0]/10,random_state=i)
    X = df.drop(columns='target')
    y = df.target
    preds = logreg.predict(X)
    print(roc_auc_score(preds,y))


--variability of the response data around its mean---
It means the model You trained tells you how much Accuracy it can provide when predicting. Higher the R_squared ,More accurate the model.

it is the percentage of the response variable variation that is explained by linear
This is literally the same question as above.


When would C become nescessary to do an analysis or manage data?

Most of the common libraries you would use for data manipulation do actually use C (or C++ or Fortran, etc.) under the hood.
There are even libraries such as CuPy, which offers the entire NumPy API, but can run your code on a GPU. Using GPUs for speed is a much more common use case these days (in my experience), compared to writing the C/C++ version.
EDIT: here is a related answer, about which programming languages are most competitive for AI, ML, DataScience, etc.

In my opinion, you might need to "do it yourself" in one of 3 cases:
1. Speed
You need it to run faster than current libraries offer - e.g. if the clustering algorithms in Scikit-Learn are too slow
2. Memory
You need to use less memory that existing algorithms - perhaps a specific method on your Pandas DataFrame uses more memory that you have available
3. New Algorithms
You need something that is fairly fast or very low level, and no existing library offers it. I would normally suggest trying your idea first using NumPy though, before trying to roll your own binaries.


Is positional encoding (in transformers) an estimation of the relative positions of words in the training corpus texts?

First, we should clarify that positional encodings are not necessarily applied to words but tokens. Nowadays, using subword tokens is the norm, and infrequent words can be divided into many subword tokens.
Regarding the role of positional encoding: according to research, positional encodings have different roles depending on where they are used:

Transformer decoders for autoregressive language modeling: positional embeddings learn about absolute positions (see What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding).

Masked language models (i.e. BERT): the positional encodings only help differentiating between tokens, and the order does not actually matter much (see Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little).

So, to answer the question:
No, positional encodings do not estimate the relative positions of words in the training corpus texts.


Technical term for using regular expressions to classify text?

If the answers are stored correctly with no errors, confusion, or ambiguity then the process is a straight up search and match. If there are typos, misspellings, or what-have-you, the process is a Fuzzy string search.

  Here's an excellent Medium article delving into the details of a fuzzy string search.

An example where these cases might exceeded is when approved and denied are included in the same transcript. Say 5% of all your cases are such. You'd likely want to apply some machine learning classifier to correctly categorize your input, in which case, your Q might be about the depths of possible solutions.


Subtract Rows of Matrix from rows of another matrix numpy

This is what you're looking for :  np.linalg.norm(vecs-V_r[:,np.newaxis], axis = 2)


Evaluating information extraction from structured documents

I have worked with Structured text using OCR. OCR is prone to errors even while reading the content and a slight change in string arrangement would lead to false positives. I used Cosine Similarity and LEvenstein distance. I curated the data with pip install ftfy, Flashtext fuzzywuzzy ,chardet and set a threshold of 97% giving with 2% buffer for errors which might be unintentional/Typos or Inability of OCR to read the character etc. With this approach you are just relaxing on the 2% error and you can still use the same Precision, Recall or Specificity and Accuracy metrics. And we based on new data, you can gather observations and adjust your threshold on run time. And yes, Make your threshold configurable than hard coded which will be easier to run multiple experiments.

I hope this helps.


Although what you describe is correct, such online/real-time usage is far from being the only (or even the most frequent) use case for DL inference. The keyword here is "batch"; there are several applications where the inference can be also run in batches of incoming data instead of on single samples.
Take the example mentioned by NVIDIA in their AI Inference Platform technical overview (p.3):

Inference also batch hundreds of samples to achieve optimal throughput on jobs run overnight in data centers to process substantial amounts of stored data. These jobs tend to emphasize throughput over latency. However, for real-time usages, high batch sizes also carry a latency penalty. For these usages, lower batch sizes (as low as a single sample) are used, trading off throughput for lowest latency. A hybrid approach, sometimes referred to as “auto-batching,” sets a time threshold—say, 10 milliseconds (ms)—and batches as many samples as possible within those 10ms before sending them on for inference. This approach achieves better throughput while maintaining a set latency amount.

Although, as correctly pointed out in another thread comment, it is on NVIDIA's best interest to convince you that you need GPUs for inference (which is indeed not always true), hopefully you can see the pattern: whenever we want to emphasize throughput over latency, GPUs will be useful for speeding up inference.
Practically, any application that runs on existing archives of data (videos, audio, music, text, documents) instead of waiting for incoming streams in real-time can meaningfully rely on GPUs for inference. And here "archives" does not imply necessarily time spans of years or months (although it can be so, e.g. in astronomy applications); the archive consisting of the photos uploaded to Facebook in the last 3 minutes (or since I have started writing this...) is huge, and it, too, can benefit from GPU-accelerated inference.
Videos, in specific, since they are usually broke up in frames to be processed, may benefit from sped up GPU inference even in near-real time applications.
Bu if you want to just set up your web app that will process low-traffic incoming photos or tweets to respond in real-time, then indeed GPU may not offer anything substantial in performance.


How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?

Lets assume you are building a text classifier with a training set of 5 sentences. For this example, lets say you are trying to classify tweets (which are usually a sentence long) to whether it was a  tweet made by Trump or not. You are given a tweet, "I have huge respect for women" and your goal is to classify it to 'Trump' or 'Not Trump'. Moreover, you are given 5 other random tweets and you also know whether those tweets were by Trump or not (Basically, they are already classified) this is your training set.

in other words you are calculating,

$$P(I\,have\,huge\,respect\,for\,women) = P(I)\times P(have)\times P(huge)\times P(respect)\times P(for)\times P(women)\\$$
$$\\$$
$$P(I\,have\,huge\,respect\,for\,women|Trump) = P(I|Trump)\times P(have|Trump)\times P(huge|Trump)\times P(respect|Trump)\times P(for|Trump)\times P(women|Trump)\\$$
$$\\$$
$$P(I\,have\,huge\,respect\,for\,women|Not\,Trump) = P(I|Not\,Trump)\times P(have|Not\,Trump)\times P(huge|Not\,Trump)\times P(respect|Not\,Trump)\times P(for|Not\,Trump)\times P(women|Not\,Trump)$$
$$\\$$
Depending on the probablity of equations (2) and (3) the user would make a decision whether the statement was made by Trump or not. Now lets say that none of the 5 training set tweets have the word huge in it, in which case,
$$\\$$
$$P(huge|Trump)=P(huge|Not\,Trump)=0$$
$$\\$$
And hence, equations (2) and (3) are now zero and this is bad. A solution to this is to do smoothing or rather Laplace smoothing. The basic idea is to increase the probablities of all bigrams in your non-maximum likelihood equation (Since we are changing the counts from what they occurred in hopes to make it better) by 1 to make everything non-zero. That increase by 1 is called adding the pseudocount or as you know it as $\alpha$. Now, $\alpha=1$ may not always give the most accurate probablities and hence they could attain any finite non-negative integer. The way to know what $\alpha$ gives the most accurate responses is through iterating over all values of $\alpha$ on the training set unfortunately. The way to do that is whole different topic.

PS : Trump did say he has huge respect for women. Such irony.


What is the best way for synthetic data generation while maintaining privacy?

Whichever method you choose is fine but assuming that you wish to mitigate inference attacks something like differential privacy is required for either approach.
Formally speaking, differential privacy provides some of the strongest guarantees against reverse engineering. Specifically, it promises that any attacker, regardless of attack methodology or available computing power, will be unable to conclude with certainty whether or not any individual has contributed data to a dataset. This is because the results of differentially-private methods are ambiguous up to the addition or removal of the input contributions of any individual. In essence, every individual gets deniability about their participation (or non-participation) in the input.
The problem with synthetic data is that it is generated from a model that is fit to real data. This means that model parameters are aggregate functions of the real data. This is problematic as it is often possible to make inferences from aggregates of data or estimates thereof (this is the motivation for differential privacy in the first place), and parameters of the generative model can be often be estimated from the synthetic data. I am happy to give an example of such an attack if there is interest. Further, it should be noted that this reasoning also implies that white-box exchange of the model is at least as risky and comes with additional concerns like has this network memorized training data. A straight-forward mitigation for this is to apply differential privacy in building the generative model for synthetic data.
In regards to privacy budgets, one can interpret the budget (often referred to as $\varepsilon$ and also the privacy-loss) as the amount of information (say in bits) which can be inferred about any individual by an adversary who has access to differentially-private results. Perhaps surprisingly, it can and ideally should be much less than 1. If there are future updates which reference the same individuals then one has to worry about how much individual information can be inferred from the aggregate collection of releases. There is a straight-forward composition theorem (See e.g. Sect 3.5) that follows directly from the definition of differential privacy. It states that aggregate privacy-loss is at most the sum of the individual privacy-losses of the constituent releases. In other words it's additive in the worst case. It may also be helpful to know that when the inputs are disjoint it goes like the max.


How to use data generator for regression keras?

When you are using a pre-trained model, you should use it's specific pre-processing function,
Below is an example for resnet50.
from keras.applications.resnet50 import preprocess_input
train_datagen = ImageDataGenerator(#rescale = 1./255
                                    preprocessing_function=preprocess_input)

Always display/print your image/label for a sanity check.
You may use the below snippet.
#Quick check the DataGenerator

import matplotlib.pyplot as plt
for image, label in train_generator:
    plt.imshow(image[0])
    print(label[0])
    break;


Recommendations based on other products seen

For very large datasets there are more powerful methods than k-nn, try to look at what the well known companies like Google, Amazon, Netflix, etc. have published.

Since you are learning, I assume your dataset is small. You can define a graph and recommend based on all views of a user. One simple method is "user-based" recommendations. A more general perspective is that of path counting / random walk ("personalized pagerank", spectral methods, etc).

If you can read math papers, you could start with C. Cooper et al Random Walks in Recommender Systems:Exact Computation and Simulations.

For book content similarities, with a small dataset, apart from the item-based similarity itself, probably your only choice is to try to find some external metadata.


ML algorithm for determining CSV file header names based on content

Yes, it might not be exactly natural Language Understanding but CRF is an excellent algorithm to train Named Entity Recognition tasks and is the stamdard model used by Stanford NLP group. You can try out their NER tagger here. If you want something that accounts for language understanding then there are certain papers who have trained recurrent neural network architectures such as LSTM and Bi-directional RNNs. Look at this paper. I must tell you that Named Entity Recognition is an incredibly hard problem and if you want to use Deep Learning architectures, it will require tremendous amount of data. I will suggest try out the Stanford NER tagger, since your data doesn't have much of a sequential nature, I am certain it will perform well. In the end here is part 1 of an excellent blog post that goes in detail about training your own NER model using the Stanford NER tool.


Rescaling each image Individually with keras

Normalize to $[0, 1]$ is what I'd do to ensure that the model gets expected (sanitized) inputs.

Using the samplewise_std_normalization is something that I'd do inside the model to highlight features.
E.g. a white pixel is more important in a mostly black image than in a noisy image.


How do I represent SURF Features into Bag of Words to determine Nearest Neighbors?

The Bags of Visual Words (BoWs) approach to image retrieval, described in works such as Sivic et al. "Video Google: a text retrieval approach to object matching in videos" (2003) and Csurka et al. "Visual categorization with bags of keypoints" (2004), is composed of multiple phases:

First, a visual vocabulary, often called a codebook, is generated. This is usually done by applying k-means clustering over the keypoint descriptors of a data set, or a sufficiently descriptive fraction of it. The vector $\mathcal{V}$ of size $k$ containing the centroids $\mathcal{V_i}$ of each cluster is your visual vocabulary.

In this case, each image $x$ should yield a variable number $x_n$ of SURF keypoint descriptors, usually of size 64 each. One can aggregate all keypoint descriptors from multiple images and perform k-means clustering over all of them.
The choice of the $k$ hyperparameter in clustering depends on the image domain. One may try multiple values of $k$ (10, 100, 1000) to understand which is more suitable for the intended task.

Afterwards, each image is "tested" against the codebook, by determining the closest visual vocabulary points and incrementing the corresponding positions in the BoW for each keypoint descriptor in the image. In other words, considering an image's BoW $B = \{ o_i \}$: for each image keypoint descriptor $d_j$, $o_i$ is incremented when the smallest distance (often the Euclidean distance) from $d_j$ to all other visual vocabulary points in $\mathcal{V}$ is the distance to $\mathcal{V}_i$.

The result is a histogram of visual descriptor occurrences of size $k$, which can be used for representing the visual content. As similar images will yield similar bags of words, one can compare images through their BoWs. The Euclidean distance between them is a commonly used metric here.

Therefore, the bag of words of each image makes a global representation of  that image. When performing content-based image retrieval, the $n$ most similar images are retrieved by fetching the $n$ closest (or approximately closest) bags of words to the query image's. No training process is required at this point (we can picture visual vocabulary generation as an offline training phase).


How can I compare classes from clusterings performed on two different data sets?

I suggest using Kullback–Leibler divergence (KLD) to compare the classes. The procedure is as follows:

1- Find the mean vector and covariance matrix of each class of each dataset.

2- Calculate the KLD between each class of the first dataset and all classes of the second dataset. So, for example if you have two dataset and each of them has 5 classes, you should calculate KLD, 25 times.

3- For each class of the first dataset, find the class from the second dataset with the lowest KLD between them. Doing this, you will find the most similar classes of the two datasets.

Note that KLD measures the similarity between two probability distributions and by calculating the mean vector and covariance matrix of each class, you are fitting a normal distribution to it. To find the formula for calculation of KLD between two multivariate normal distribution, refer to here:

https://en.m.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence


Specifying number of threads using XGBoost.train

You can set the number of threads by nthread parameter in XGBClassifier or XGBRegressor

import time
import numpy as np
from sklearn.datasets import load_boston
import xgboost as xgb
num_threads = [1,2,3,4,5,6,8,16,32,64]
for n in num_threads:
    start = time.time()
    model = xgb.XGBRegressor(objective='reg:squarederror',nthread=n)
    model.fit(X, y)
    elapsed = time.time() - start
    print(n, round(elapsed,3))
    results.append(elapsed)

The output after execution of this code is

    1 0.059
    2 0.071
    3 0.063
    4 0.094
    5 0.075
    6 0.078
    8 0.09
    16 0.099
    32 0.157
    64 0.235


label encoding or one-hot encoding or none when using decision tree?

If we are using label encoder we would only need to convert gender however if that maps male = 0, female = 1 wouldn't the machine treat female > male?

You are correct, using label encoder to encode categorical features is wrong in general, for the reason you mention. Note that scikit documentation advises against using it with features, it's supposed to be used only with a response variable.
In the particular case of a binary variable like "gender" to be used in decision trees, it actually does not matter to use label encoder because the only thing the decision tree algorithm can do is to split the variable into two values: whether the condition is gender > 0.5 or gender == female would give the exact same results.
Also note that whether the variable is interpreted as ordinal or not is a matter of implementation. For example in Weka it's possible to specify that a feature is categorical ("nominal").

and if it ignores ordinality it will ignore level1 < level2.

Not necessarily, because in theory it's possible to have features with different types (e.g. some categorical and some numerical). However this may depend on the implementation as well.


How to obtain the final values of a DecisionTreeRegressor in Scikit-Learn?

For documentation, check the Scikit-Learn code at Github i.e. line#535 [Link]

value : array of double, shape [node_count, n_outputs, max_n_classes]
Contains the constant prediction value of each node.

You can check the same using this sample code.

Identify the leaf nodes
Slice the value attributes for leaf nodes

from sklearn.datasets import load_diabetes
import numpy as np, matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeRegressor

X, y = load_diabetes(return_X_y=True)
regressor = DecisionTreeRegressor(max_depth=5)
regressor.fit(X, y)

children_left = regressor.tree_.children_left
children_right = regressor.tree_.children_right
leaf_nodes = []
for i in range(n_nodes):
    if children_left[i] == children_right[i]:
        leaf_nodes.append(i)

y_pred =  regressor.predict(X)
all_val = regressor.tree_.value[leaf_nodes,0,0] # Sliced on Leaf nodes

# Check if all the y_pred is from these values
set(y_pred) - set(all_val) # Or, [elem for elem in y_pred if elem not in all_val]


Choice of replacing missing values based on the data distribution

I would not definitely recommend substituting missing values by mean or by median or mode. If you want to go through some techniques and get a glance at them, I would recommend going through this link and for imputation techniques this wiki page gives you a brief .
Do you think that there is a way to predict missing values from the other cells. If yes, apply a regression model on those variables and estimate the missing value. But remember this lacks variability as values fall on the regression line itself. There are methods like regression imputation which can add this variability component to the estimated value.
If you are unable to go anywhere from the previous step, then see how the values are distributed for the missing variable, substitute them according to that distribution by using a random function.
And if your unable to perform any of the above mentioned ones and want to go by mean, median. I cant really give my opinion as in this case they are nearer to each other. See what gives you the best predictable and decide between them.
Coming to your final question, Std. deviation just shows how far your values are falling away from mean. If your data has large range with good enough number of  points distributed at extremes, you would be expected to have high std. deviation.


Newbie: What is the difference between hypothesis class and models?

Your hypothesis class consists of all possible hypotheses that you are searching over, regardless of their form. For convenience's sake, the hypothesis class is usually constrained to be only one type of function or model at a time, since learning methods typically only work on one type at a time. This doesn't have to be the case, though:

Hypothesis classes don't have to consist of only one type of function. If you're searching over all linear, quadratic, and exponential functions, then those are what your combined hypothesis class contains.
Hypothesis classes also don't have to consist of only simple functions. If you manage to search over all piecewise-$\tanh^2$ functions, then those functions are what your hypothesis class includes.

The big tradeoff is that the larger your hypothesis class, the better the best hypothesis models the underlying true function, but the harder it is to find that best hypothesis. This is related to the bias–variance tradeoff.


Retrain Tensorflow CNN Model on additional Images

There are two basic approaches to this problem:

The performance of the old model was not good enough, you are able to gather much more data and train a new better model

You setup an "online model" which continuously learns and improves but is setup very differently.

Do you have a specific performance level in mind that you need / want to hit? In that case simply gathering more data and retraining may be fine. However if you simply want to continuously improve you need to approach it with an "online model".


Should I concat multiple stock timeseries datasets into one?

I would suggest option two. Because this way your model would have the chance to learn something for one stock, which it can apply for other stocks as well. If you provide the type of stock as an input feature, it should be able to distinguish between the specialities which only occure within one stock and the common things. So it is kind of able to transfer knowledge from one stock to the other.
But at the same time, I would suggest to try out both and choose whatever performs the best. This way you can also justify your choice in the end.


Does stronger regularization always improve performance on testing set?

Title question
The answer to the title question is a pretty clear-cut "no." For any fixed dataset, taking sufficiently strong regularization will make the linear model essentially constant, presumably not the best model.  Like most other things, there's a bias-variance tradeoff: increasing regularization weakens the model's ability to learn the training data, which to a point will tend to prevent overfitting and improve test performance.
Your more specific context
Word embeddings are substantially different from the other two, so the optimal regularization strength is probably mostly unrelated.
Between bow and tfidf, if nothing else the scale of the features will be larger in bow, so if you're not scaling, the penalties will be larger and the optimal regularization strength is likely to be smaller.


Pandas Data Frame: Calculating custom moving average

Worked with the below to add a column that calculates the above function

quote['MFITEST']=quote['Date'].apply(lambda x: calc_mfi(quote,14,x))


Extracting MM-YYYY from python date and creating a new column with the same

import datetime
df1['Month'] = df1['booking_date'].apply(lambda x: datetime.datetime.strftime(x, "%B-%Y"))


Have you seen this blog post? It is doing exactly what you are trying to do.

Convolutional neural network
Use a fixed sliding window to crop plate number
Train on a fixed-length number plate

If you want to train on the digit-level, you should define a fixed rectangle size, large enough to fit a single digit in your data set. If your convolutional neural network is good, it should be able to detect your digit.

If you haven't you should take a look at the Google paper: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42241.pdf


Hyperparameter tuning with Bayesian-Optimization

The pbounds must all be pairs; you cannot specify a list of options for max_depth.
The package cannot deal with discrete hyperparameters very directly; see section 2, "Dealing with discrete parameters", of their "advanced tour" notebook about this.


What ML architecture fits fixed length signal regression?

One common approach for this type of data is take the integral and learn either a translation function to fish weight. Taking the integral simplifies the problem to a single number.
You probably do not need a state of the art model. A general linear model would probably pick out a signal.


It is helpful to normalize target variables for a regression neural network?

One reason for normalising the inputs is to make gradient descent more stable, as gradients spend more time in a comfortable region with meaningful updates and less neurons 'die' during trainings - getting stuck at one of the tails of e.g. the sigmoid non-linearity.

Normalising the output distribution is perhaps not the best idea, as you are by definition altering the defition of the target. This means you are essentially predicting a distribution that doesn't mirror your real-world target (at least without some reverse non-linear transforms later on).

On this you could do would be to scale the target, instead of normalising. The shape of the distribution should remain almost identical (thinking about the shape of the distribution), but the values themselves might be more easily attainable and therefore faster to optimise for; they are all closer in magnitude to the gradients that are being computed.


You can specify the weights directly within the weighted.mean() function, within the call to funs() like so:

data.frame(x=rnorm(100), y=rnorm(100), weight=runif(100)) %>%
      summarise_at(vars(x,y), funs(weighted.mean(., w=weight)))


You have to add higher order polynomial terms to your dataset yourself. Accordingly, you should add new columns to your dataset which consists of higher order polynomials with the desired combination of the existing features. It means that if you have already a CSV file which contains two columns for different samples, you have to add your desired extra feature columns which are the polynomial combination of the existing features. Then use simple regression.


Is there an open source implementation for bag-of-visual words?

There is one implementation of BoVW in openCV. You can find the documentation here :

http://docs.opencv.org/2.4/modules/features2d/doc/object_categorization.html


Error loading tidyverse in R Studio - No broom package

It worth trying again because sometimes there is an issue with CRAN.

Try installing first

> install.packages("tidyverse")

Then loading

> library(tidyverse)


How to create a score for a SWOT analysis (strengths, weaknesses, opportunities, and threats)?

Let's suppose you have a variable $x_{old} \in [-1,1]$. If you would like to transform that to $ x_{new} \in [-2,2]$ then you could just apply the transformation formula you posted:

With $min_{old}=-1, max_{old}=1, min_{new}=-2, max_{new}=2$ you would get $$\frac{max_{new}-min_{new}}{max_{old}-min_{old}} (v-min_{old}) + min_{new}
= \frac{2-(-2)}{1+1} (v+1) + (-2) = \frac{4}{2} (v+1) -2\\= 2v+2-2=2v$$

Now if you compare that to your formula

= (((strengths + opportunities) - (weaknesses + threats)) /
 ((strengths + opportunities) + (weaknesses + threats)) * 2

then you can see that it is exactly what you did here by multiplying with 2 at the very end. You transformed your first scale from $[-1,1]$ to $[-2,2]$. Therefore, if you want it to be in the interval $[0,10]$ just apply a different transformation at the very beginning when deriving the first scale:

$$\frac{max_{new}-min_{new}}{max_{old}-min_{old}} (v-min_{old}) + min_{new}
= \frac{10-0}{1+1} (v+1) + 0 = 5v +5$$

Which means your calculation for the first scale needs to be

= (((strengths + opportunities) - (weaknesses + threats)) /
 ((strengths + opportunities) + (weaknesses + threats)) * 5 + 5

(instead of scaling it to $[-2,2]$).

Intuitively I like to think of it like this: you "stretch" the interval from $[-1,1]$ to $[-5,5]$ by multiplying with 5 ("making it 5 times larger"). And then, since you want it to be positive and start at $0$, you "shift" it "upwards" by adding 5.


Is there HMM that looks two elements back?

I think the keyword you are looking for is Higher Order Hidden Markov Models. Your "ordinary" HMM would be considered a First Order HMM. So yes, there is literature these types of HMMs.


Is there any way to estimate the accuracy of my model on the new dataset?

No.

If you want to evaluate the classification accuracy of your model, you need to know the ground truth to compare, simple as that. Estimating the similarity of the datasets is totally irrelevant, unless there is prior knowledge that feature similarity  leads to label similarity.


Situations where advanced theoretical knowledge of ML helped solve a real world problem?

At the end of the day, as an applied data scientist you have a bag of tools you can use to solve business problems.  It's helpful to know what your tools are capable of, where each is useful, and also what their limits are.  If you don't have at least some theory, knowing which method to use in which circumstance will seem like a meaningless list of details, and when things break, you may not know why - is there an error in my code, or am I making a fundamental error in applying an algorithm in a scenario where it has fundamental limitations?

You can be pragmatic in applying tools as you've seen them applied before.  Theory will help you have an intuition about whether it's possible or reasonable to apply given tools to new situations that you haven't seen.  It can help you answer questions like: how will this algorithm scale with the size or dimensionality of data?  SVMs in particular suffer from poor scaling with data size, precisely because they rely on kernels.  That kind of basic knowledge is probably more useful than being able to write down the kernel trick.


Has anyone tried to use the hierarchy of ImageNet?

Yes, multiple papers have used this. I've heard of multiple ways to exploit this hierarchial structure. This paper Hierarchical Deep Convolutional Neural Network for Large Scale
Visual Recognition uses multiple levels by predicting the more coarse distribution and I think it then passes this as features to the more low level classification. YOLO9000 actually uses the ImageNet hierarchy for the problem of object detection. Hierarchial classification is the name of the problem you are describing.


No, that is the steps per epoch. One step is taking 32 images(batch size) So, 709*32 images used in that epoch and validated with 79*32 images.


This is how the problem is formulated as a Causal Inference problem:
Take group 1 as the Control Group and group 2  the Treatment Group. Based on the observations ( "S-4","S-3","S-2","S-1","S+1","S+2","S+3"), a model is fit on the observations. How many models depend on the type of learning, S,T or X-learner.
Basically, what these learners do, is separately fit the features to the target (S), conditionally on the features originating from control or treatment.
Once you have fit the learners, a treatment effect is estimated.
You can see the treatment effect as the difference between the predictions on either group, given the knowledge restricted to the opposite group.
Here is a sample code from Uber's causalML
from causalml.inference.meta import XGBTRegressor

y, X, treatment = load_data()

xg = XGBTRegressor(random_state=42)
te, lb, ub = xg.estimate_ate(X, treatment, y)
print('Average Treatment Effect (XGBoost): {:.2f} ({:.2f}, {:.2f})'.format(te[0], lb[0], ub[0]))


For windows, use the task scheduler to set the task to run for example daily at 4:00 AM

It gives you many other options regarding frequency etc.
http://en.wikipedia.org/wiki/Windows_Task_Scheduler


When to use Random Forest over SVM and vice versa?

I would say, the choice depends very much on what data you have and what is your purpose. A few "rules of thumb".

Random Forest is intrinsically suited for multiclass problems, while SVM is intrinsically two-class. For multiclass problem you will need to reduce it into multiple binary classification problems.

Random Forest works well with a mixture of numerical and categorical features. When features are on the various scales, it is also fine. Roughly speaking, with Random Forest you can use data as they are. SVM maximizes the "margin" and thus relies on the concept of "distance" between different points. It is up to you to decide if "distance" is meaningful. As a consequence, one-hot encoding for categorical features is a must-do. Further, min-max or other scaling is highly recommended at preprocessing step.

If you have data with $n$ points and $m$ features, an intermediate step in SVM is constructing an $n\times n$ matrix (think about memory requirements for storage) by calculating $n^2$ dot products (computational complexity).
Therefore, as a rule of thumb, SVM is hardly scalable beyond 10^5 points.
Large number of features (homogeneous features with meaningful distance, pixel of image would be a perfect example) is generally not a problem.

For a classification problem Random Forest gives you probability of belonging to class. SVM gives you distance to the boundary, you still need to convert it to probability somehow if you need probability.

For those problems, where SVM applies, it generally performs better than Random Forest.

SVM gives you "support vectors", that is points in each class closest to the boundary between classes. They may be of interest by themselves for interpretation.