How much more data does my classifier need?

I think you already have enough data. Your problem seems to be one of generalization, which is in general a tricky one to solve.
For SVM, depending on what kind you have, you could try tinkering with the parameters and kernels.
Have a look here. -- https://stats.stackexchange.com/questions/35276/svm-overfitting-curse-of-dimensionality?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa


Vanishing Gradient vs Exploding Gradient as Activation function?

ReLU is considered as an activation function, on similar fashion can we use Gradient Clipping also as an activation function?

ReLU is an activation function. Gradient clipping is a technique to keep the problem of exploding gradient at bay.

I wish also to stress that the best technique to control for vanishing/exploding gradients is, at the moment, batch normalization. Dropout (a technique born to fight overfitting) also has a similar regularization effect - by forcing the model to distribute weights more evenly through the layer. That's why you don't see gradient clipping that often as it used to.

EDIT:

I forgot to mention that a proper scaling of your variables and appropriate weight initializations make the problem of vanishing/exploding gradient not very frequent. This of course is purely based on personale experience. It's still very important to take it into account


Data simulation using make_classification in Python

There is no obstacle to doing this. For example you can create data by make_classification, and compare different algorithms by building model on it. You can also pass a random_state value to obtain same data each time you call the function. Both SVM, and Decision Trees can work with continuous data.


You should post your code, or no one here can see what you have tried so far.  Anyway, I'll throw this out there for you, and hopefully it will clarify things.

from pandas_datareader import data as wb
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.pylab import rcParams
from sklearn.preprocessing import MinMaxScaler

start = '2019-02-20'
end = '2020-02-20'

tickers = ['AAPL']

thelen = len(tickers)

price_data = []
for ticker in tickers:
    prices = wb.DataReader(ticker, start = start, end = end, data_source='yahoo')[['Open','Adj Close']]
    price_data.append(prices.assign(ticker=ticker)[['ticker', 'Open', 'Adj Close']])

#names = np.reshape(price_data, (len(price_data), 1))

df = pd.concat(price_data)
df.reset_index(inplace=True)

for col in df.columns:
    print(col)

#used for setting the output figure size
rcParams['figure.figsize'] = 20,10
#to normalize the given input data
scaler = MinMaxScaler(feature_range=(0, 1))
#to read input data set (place the file name inside  ' ') as shown below
df.head()

df['Date'] = pd.to_datetime(df.Date,format='%Y-%m-%d')
#df.index = names['Date']
plt.figure(figsize=(16,8))
plt.plot(df['Adj Close'], label='Closing Price')

ntrain = 80
df_train = df.head(int(len(df)*(ntrain/100)))
ntest = -80
df_test = df.tail(int(len(df)*(ntest/100)))

#importing the packages
from sklearn.preprocessing import MinMaxScaler
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM

#dataframe creation
seriesdata = df.sort_index(ascending=True, axis=0)
new_seriesdata = pd.DataFrame(index=range(0,len(df)),columns=['Date','Adj Close'])
length_of_data=len(seriesdata)
for i in range(0,length_of_data):
    new_seriesdata['Date'][i] = seriesdata['Date'][i]
    new_seriesdata['Adj Close'][i] = seriesdata['Adj Close'][i]
#setting the index again
new_seriesdata.index = new_seriesdata.Date
new_seriesdata.drop('Date', axis=1, inplace=True)
#creating train and test sets this comprises the entire data’s present in the dataset
myseriesdataset = new_seriesdata.values
totrain = myseriesdataset[0:255,:]
tovalid = myseriesdataset[255:,:]
#converting dataset into x_train and y_train
scalerdata = MinMaxScaler(feature_range=(0, 1))
scale_data = scalerdata.fit_transform(myseriesdataset)
x_totrain, y_totrain = [], []
length_of_totrain=len(totrain)
for i in range(60,length_of_totrain):
    x_totrain.append(scale_data[i-60:i,0])
    y_totrain.append(scale_data[i,0])
x_totrain, y_totrain = np.array(x_totrain), np.array(y_totrain)
x_totrain = np.reshape(x_totrain, (x_totrain.shape[0],x_totrain.shape[1],1))

#LSTM neural network
lstm_model = Sequential()
lstm_model.add(LSTM(units=50, return_sequences=True, input_shape=(x_totrain.shape[1],1)))
lstm_model.add(LSTM(units=50))
lstm_model.add(Dense(1))
lstm_model.compile(loss='mean_squared_error', optimizer='adadelta')
lstm_model.fit(x_totrain, y_totrain, epochs=3, batch_size=1, verbose=2)
#predicting next data stock price
myinputs = new_seriesdata[len(new_seriesdata) - (len(tovalid)+1) - 60:].values
myinputs = myinputs.reshape(-1,1)
myinputs  = scalerdata.transform(myinputs)
tostore_test_result = []
for i in range(60,myinputs.shape[0]):
    tostore_test_result.append(myinputs[i-60:i,0])
tostore_test_result = np.array(tostore_test_result)
tostore_test_result = np.reshape(tostore_test_result,(tostore_test_result.shape[0],tostore_test_result.shape[1],1))
myclosing_priceresult = lstm_model.predict(tostore_test_result)
myclosing_priceresult = scalerdata.inverse_transform(myclosing_priceresult)

Epoch 1/3
 - 7s - loss: 0.0163
Epoch 2/3
 - 6s - loss: 0.0058
Epoch 3/3
 - 6s - loss: 0.0047

totrain = df_train
tovalid = df_test

#predicting next data stock price
myinputs = new_seriesdata[len(new_seriesdata) - (len(tovalid)+1) - 60:].values

#  Printing the next day’s predicted stock price.
print(len(tostore_test_result));
print(myclosing_priceresult);

# next day's predicted closing price
[[329.42258]]

So, on 2020-02-20, we are predicting what AAPL will close at, on 2020-02-21.  The model said it would be 329.42 and the actual close was 313.05.  Less than 5% difference.  Not bad, but I would have expected a little better accuracy.  Oh well, we illustrated the point, and that was the goal of this exercise.

See the link below for more info.

https://www.codespeedy.com/predicting-stock-price-using-lstm-python-ml/


Number of units for first layer in Keras Sequential Model

You can try "Border Pairs Method", "Bipropagation" or "One step method" to finding out how many neurons you need in each layer.
https://www.researchgate.net/publication/322617800_New_Deep_Learning_Algorithms_beyond_Backpropagation_IBM_Developers_UnConference_2018_Zurich


Using doccano for Aspect Based Sentiment Analysis annotation

You can use sequence labeling feature to annotate the text:


Scikit-learn pipeline with scaling, dimensionality reduction, average prediction of multiple regression models, and grid search cross validation

You might looking for sklearn.ensemble.VotingRegressor which takes the mean of two regression models.
Here is an example to get you started:
from sklearn.datasets        import make_regression
from sklearn.decomposition   import PCA
from sklearn.ensemble        import GradientBoostingRegressor, RandomForestRegressor, VotingRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline        import Pipeline
from sklearn.preprocessing   import StandardScaler

# Make fake data
X, y = make_regression(n_samples=1_000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=42)

pipe = Pipeline([('scl', StandardScaler()),
                 ('pca', PCA()),
                 ('vr', VotingRegressor([('gbr', GradientBoostingRegressor()), ('rfr', RandomForestRegressor())]))
                ])

search_space = [{'vr__gbr__learning_rate':    [.07, .1, .15]}]

gs_cv = GridSearchCV(estimator=pipe,
                     param_grid=search_space,
                     n_jobs=-1)

gs_cv.fit(X_train, y_train)
gs_cv.predict(X_test)


How effective would a data-driven vaccination program be over a simple rules based approach

The reason for the rules based approach is the simpson paradox.

In short, the way a test group is sampled can easily overturn the conclusion of a study. It is very important to include sufficient patients from a particular risk group in the study.
This is why rules are set.
This approach however is the basis of causality theory. A data driven-approach would be as followed:

experts flag expected causal patterns, i.e. draw causal graph of the study. [This is the mathematical formulation of risk factors]
Causal graph generates ideal distribution of study and tools to draw conclusion


As there is no answer yet I want to try to give an at least somehow useful answer.
Including the constants definitions would help a bit to understand the provided code.

Generally speaking a batch uses n times a record or item. How you define an item depends on your problem. In tensorflow the batch is encoded in the first dimension of a tensor. In your case with the csv file it might be line by line (reader=tf.TextLineReader). It could learn by column but I don't think that this is happening in your code. If you want train with your whole dataset (=one epoch) you can do so by using numBatches=numItems/batchSize.


Treemaps are not very intuitive when trying to understand hierarchy levels. Sunburn chats are better.

First hierarchy level is either shown by borders or different colors.
Subsequent hierarchy level is represented by smaller rectangle which is part of bigger rectangle.

Refer this article


How to quantitatively evaluate raw neural network activations?

1) It is difficult to interpret a given weight on a given node for a given instance, especially in hidden layer.

2) You can only deal with specifical cases :

Weights being very low for a given variable for all instance on the first layer means that you can remove the variable (If the variable is relatively uniform, standardized).
For some activation functions, you can remove 'dead' neurons that activate to few (or no) instances. More generally, regarding to your questions, answers highly depends on the kind of activation. I would reccomend you to read : http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf by Lecun to avoid common traps. It will also give you some idea about what problems to look for (vanishing / exploding gradient, dead neurons... etc.)

3) On the architecture : as a rule of thumb for the size of your networks I would advise to start big, get a performance benchmark and remove layers / units untill the performance drops. I am not sure if you plotted only hidden layer or not, but in most cases the numbers of layers shoud decrease.

4) You might want to look some explanatory tools, like DEEPLIFT or SHAP, that will give you variable importance, summarizing all variable impact for an instance. These methods have some limitations but they will be easier to read.


Reinforcement learning for continuous (rather than discrete) actions

The DDPG described in that paper is very good to start working in control problems. There are many implementations of this algorithm online. However, bare in mind though that the algorithm is very unstable and needs many hours of monitoring and tuning the parameters of the networks. I would suggest you to create a very simple scenario (simulator) and put the DDPG to interact with it. It is very important to understand what your networks are learning and which state features are important or no. Furthermore the reward function will play a very important role in the performance of your algorithm.

Another important point is the way you formalize your problem will determine if you are trying to solve a one-step decision making problem (e.g. given state features ---> credit limit value, reward) or a sequential decision making problem (e.g. given state features ---> credit limit value, reward, new state features and so on). This will specify if you need to modify the Q learning part of the Critic and remove the $maxQ(s',a')$ part (one step decision case).

Regarding pretraining, as Neil suggested you can use regression to pretrain Actor and Critic networks so the weights are initialized to better than random values. A very interesting approach from Deepmind is the one that is described in this paper Learning from Demonstrations for real-world RL, and most likely helped them optimize the energy consumption at their server centers.


Is there an overview over recommender system architectures?

A great source is the personal page of Xavier Amatriain (former Head of Engineering at Netflix and Quora). There are several links at publications detailing the system and architecture of recommender systems. In particular you could take a look at this Building industrial-scale real-world recommender systems, there is also this talk Lessons Learned from Building Real-Life Recommender Systems and this blog post System Architectures for Personalization and Recommendation


Document matching with more priority to certain features than others

If you don't want all features to have the same weight in the cosine similarity you could just use weighted cosine similarity.

Cosine similarity is calculated as

$similarity = \frac{\vec A\cdot \vec B}{\Vert A\Vert\cdot\Vert B\Vert}$

Now, if you multiply each component $a_i$ of $\vec A$ and $b_i$ of $\vec B$ with a weight $w_i$ you will get a weighted cosine similarity.

Which weights you should use depend on the application. If you have a ranking of your features $\vec R$ you could give $\frac{1}{r_i}$ or $\frac{1}{r^2_i}$ a try. The feature with rank 1 will have a weight of 1 in both cases. In principle any measure of feature importance will do as weights.


Plotting in Multiple Linear Regression in Python 3

You cannot plot graph for multiple regression like that. Multiple regression yields graph with many dimensions. The dimension of the graph increases as your features increases. In your case, X has two features. Scatter plot takes argument with only one feature in X and only one class in y.Try taking only one feature for X and plot a scatter plot. By doing so you will be able to study the effect of each feature on the dependent variable (which i think is more easy to comprehend than multidimensional plots).I think your issue should resolve.


Difference between validation and prediction

In the GridsearchCV function, there is a parameter, cv that tells you how many times cross-validation is performed. If you're performing cross-validation on the train set, all you need is a train/test set since the hyperparameters are being tuned without seeing the test set.

If you want to use your own validation set, see the link below. The link below that also has a good description of how K-Fold CV works in GridSearchCV.

Finally, gridsearch optimization is pretty slow and there are more advanced methods for hyperparameter tuning now, in case performance is an issue.

See this previous answer:

Difference between test and validation set

Preselected Validation Set

Gridsearch CV and KFold CV


How to compare two unsupervised anomaly detection algorithms on the same data-set?

For unlabeled data-sets, unsupervised anomaly detectors can be compared either subjectively or objectively.

Subjective comparison: based on our domain-knowledge and by using some visualizations and statistics, we can compare two detectors and select the one that outputs better anomalies subjectively.

Here is a well-cited survey on unsupervised anomaly detectors that compares the algorithms on labeled data-sets (with known, domain-specific outliers) using AUC, and concludes that local detectors (such as LOF,
COF, INFLO and LoOP) are not good candidates for global anomaly detection:
2016 A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

Objective comparison: possible in theory, impossible in practice.

Requirements for objective comparison:

Anomaly definition: $x$ is an anomaly if $P(x)< t$ for some threshold $t$,
Anomaly detector requirement: $D$ is an anomaly detector if for every detected $x$, $P(x)< t$,
Comparing anomalies: $x_1$ is more anomalous than $x_2$ if $P(x_1)<P(x_2)$ or equivalently $r(x_1, x_2) = P(x_1) / P(x_2) < 1$,
Comparing anomaly detectors: proposal $x_1$ from detector $D_1$ is better than $x_2$ from $D_2$ if $r(x_1, x_2) < 1$,

As you can see, for qualification and comparison of two detectors we need to know the underlying $P(x)$ or at least $r(x_1, x_2)$. But if we know these quantities (which act as a  judge $J$) or at least a close enough estimation of them, we have a better anomaly detector $J$ and can throw $D_1$ and $D_2$ away! We plug any observation $x$ or pair of observations $x_1$ and $x_2$ into $J$ and check which one is an anomaly or which one is more anomalous, done! So it is impossible to compare two anomaly detectors objectively unless we have a better anomaly detector (judge). So we should use a subjective comparison.


What is "position" in CNN (im2latex) for Positional Encoding?

In the original transformer model, one input 'pos' was required as input was 1 dimensional (words). With encoder being CNN, the input feature vector will be 2D, so x and y are just the indexes of an 2D feature vector. By using those indexes, PE functions can produce those positional encoder images that you can find in the paper.


Using BERT for the first time, what are the two columns of my test_results.tsv?

If you look at your output you will see that the two values in every row add up to 1. This means that the first value is the probability of the output having value 0 in your classification, and the second value the probability of the out having value 1. If you want to only have the 0/1 classification instead of the probabilities you can use a simply threshold for the probabilities (i.e. if value < threshold then 0, else 1). The simplest way to set the threshold could be 0.5, but you could try changing threshold and see how the performance on your test set changes.


What is an 'old name' of data scientist?

In reverse chronological order: data miner, statistician, (applied) mathematician.


To make it a fair test, you need to ensure that neither model has seen any of the data in test_02. The closer you can get each subset of the data to match the distribution of the overall combined data set, the better. If you steer by that, the approach you suggested is fine.

A more direct way of answering the question of whether you need a new model is to work out the average accuracy of a new model trained on the updated data set (with the new samples) by using k-fold cross-validation, and then compare that to the accuracy of the average model trained on the old data and tested on the combined data.


How can I access to loss value in Keras LSTM implementation?

You can access it by assigning a variable when calling fit

hist = model.fit(X, y)

Where hist is a dictionary containing history of various variables during training. To get your training loss you would do hist['loss']


optimal combination of hyper parameters and model selection

"What exactly is an optimal combination of hyper parameters?" The combination of hyperparameters that produces the lowest possible error on unseen data for that model architecture.

"What should the model selection decision be based upon?" The best way to estimate error on unseen data that I know of is k-fold cross-validation. I examine the mean and standard deviation of the error across the k-folds; in most cases, I select the model with the smallest standard deviation among the models with the best mean error.

"And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?" As far as I know, one can never know what combination of hyperparameters will produce the lowest possible error on unseen data. In my experience, a good search strategy will get you close enough to the optimal combination so that further search is not worth the effort.


Just to add - I've found a second high quality monitor to be invaluable in improving productivity when working with datasets.


Extract the "path" of a data point through a decision tree in sklearn

Looks like this is easier to do in R, using the rpart library in combination with the partykit library. I'd ideally like to find a way to do this in python, but here's the code, for anyone who is interested (taken from here):

pathpred <- function(object, ...){
    ## coerce to "party" object if necessary
    if(!inherits(object, "party")) object <- as.party(object)

    ## get standard predictions (response/prob) and collect in data frame
    rval <- data.frame(response = predict(object, type = "response", ...))
    rval$prob <- predict(object, type = "prob", ...)

    ## get rules for each node
    rls <- partykit:::.list.rules.party(object)

    ## get predicted node and select corresponding rule
    rval$rule <- rls[as.character(predict(object, type = "node", ...))]

    return(rval)
}

Illustration using the iris data and rpart():

library("rpart")
library("partykit")
rp <- rpart(Species ~ ., data = iris)
rp_pred <- pathpred(rp)
rp_pred[c(1, 51, 101), ]

Yielding,

       response prob.setosa prob.versicolor prob.virginica
 1       setosa  1.00000000      0.00000000     0.00000000
 51  versicolor  0.00000000      0.90740741     0.09259259
 101  virginica  0.00000000      0.02173913     0.97826087
                                           rule
 1                          Petal.Length < 2.45
 51   Petal.Length >= 2.45 & Petal.Width < 1.75
 101 Petal.Length >= 2.45 & Petal.Width >= 1.75

Which looks to be something I could at least use to derive shared parent node information.


ResourceExhaustedError when building Sequential model

As described in the error message, the problem is that your model needs more memory than your GPU has. Note that OOM stands for "out of memory".
The specific layer that is demanding too much memory is Dense(154457, activation='relu'). Nevertheless, the last layer is even bigger.
You should think if you really need an output of dimensionality 154457. If you really need it, you maybe should go for approaches that are specifically designed for that scenario, like adaptive softmax.
Another option is to use a GPU with more memory that can fit this model.


Tensorflow uses more memory, the more epochs it completes

It seems like you are adding nodes to your computational graph in this line: layer = layer.assign(evolved_layer). The assign-operation is just as multiplication or addition a node in the graph and you construct a new one in every step. Call tf.Graph.finalize() after defining your model. This will raise an error whenever a node is added to the graph and help you debugging your code. Ideally you want the graph to be fixed after construction.  I suspect you will get an error in the optimizer.

You want all the operations to be defined before you run the training loop and then just run the respective nodes. In your case you'd define layer_assign = layer.assign(evolved_layer) before the loop and in the loop only use sess.run(layer_assign). Of course you would have to treat all operations in evolve_layers in the same way.


Accessing and Multiplying Individual Elements of a Layer's Output in Keras

This line of code will select the highest value in your softmax function. If you'd like more than one value just change the 1 to whichever value you'd like. I hope this helps.

top_value = tf.nn.in_top_k(logits, y, 1)


Point of dropping weights in mini batch for purpose of regularization

Yes dropout is useful this case too. The purpose of dropout is to make the model generalize better by causing it to use all available weights from a given layer. It is supposed to make the subsequent layer not rely on a small number of activations from the previous layer, but to use them all.


Can machine learning learn a function like finding maximum from a list?

Maybe, but note that this is one of those cases where machine learning is not the answer. There is a tendency to try and shoehorn machine learning into cases where really, bog standard rules-based solutions are faster, simpler and just generally the right choice :P

  Just because you can, doesn't mean you should

Edit: I originally wrote this as "Yes, but note that..." but then started to doubt myself, having never seen it done. I tried it out this afternoon and it's certainly doable:

import numpy as np
from keras.models import Model
from keras.layers import Input, Dense, Dropout
from keras.utils import to_categorical
from sklearn.model_selection import train_test_split
from keras.callbacks import EarlyStopping

# Create an input array of 50,000 samples of 20 random numbers each
x = np.random.randint(0, 100, size=(50000, 20))

# And a one-hot encoded target denoting the index of the maximum of the inputs
y = to_categorical(np.argmax(x, axis=1), num_classes=20)

# Split into training and testing datasets
x_train, x_test, y_train, y_test = train_test_split(x, y)

# Build a network, probaly needlessly complicated since it needs a lot of dropout to
# perform even reasonably well.

i = Input(shape=(20, ))
a = Dense(1024, activation='relu')(i)
b = Dense(512, activation='relu')(a)
ba = Dropout(0.3)(b)
c = Dense(256, activation='relu')(ba)
d = Dense(128, activation='relu')(c)
o = Dense(20, activation='softmax')(d)

model = Model(inputs=i, outputs=o)

es = EarlyStopping(monitor='val_loss', patience=3)

model.compile(optimizer='adam', loss='categorical_crossentropy')

model.fit(x_train, y_train, epochs=15, batch_size=8, validation_data=[x_test, y_test], callbacks=[es])

print(np.where(np.argmax(model.predict(x_test), axis=1) == np.argmax(y_test, axis=1), 1, 0).mean())

Output is 0.74576, so it's correctly finding the max 74.5% of the time. I have no doubt that that could be improved, but as I say this is not a usecase I would recommend for ML.

EDIT 2: Actually I re-ran this this morning using sklearn's RandomForestClassifier and it performed significantly better:

# instantiation of the arrays is identical

rfc = RandomForestClassifier(n_estimators=1000, verbose=1)
rfc.fit(x_train, y_train)

yhat_proba = rfc.predict_proba(x_test)

# We have some annoying transformations to do because this .predict_proba() call returns the data in a weird format of shape (20, 12500, 2).

for i in range(len(yhat_proba)):
    yhat_proba[i] = yhat_proba[i][:, 1]

pyhat = np.reshape(np.ravel(yhat_proba), (12500,20), order='F')

print(np.where(np.argmax(pyhat, axis=1) == np.argmax(y_test, axis=1), 1, 0).mean())

And the score here is 94.4% of samples with the max correctly identified, which is pretty good indeed.


Similarity of Dictionaries Using Cosine Similarity

This is wrong: vector2.append(float(input1[k]))
Replace by vector2.append(float(input2[k]))


Looking at another question on Stack Overflow about serialization exceptions in Spark, it says that anonymous functions serialize their containing class, and if that class contains the SparkContext -- which is not serializable -- then an error is thrown.

Perhaps this is happening to you?


Public dataset for news articles with their associated categories

Here is a massive dataset of news with categories which I created for exactly such a reason.
Includes all the headlines published by Times of India from 2001-2019 with categories.
Contains ~3 million entries.


The problem is with the y value.

x=dataset.iloc[:,:-1].values -> selecting all columns but last #this one is fine

y=dataset.iloc[:,:2].values -> selecting all columns till 2nd #this one is wrong

Change this to :

y=dataset.iloc[:,2].values #selecting only 2nd column.


Depending on your target task. If you are to classify documents, then e.g. fastText has it's own approach and there are other classification techniques, not strictly generating embeddings, like LSA / LDA (using topic modelling) or word mover distance.


What is dimensionality reduction? What is the difference between feature selection and extraction?

Simply put:

feature selection: you select a subset of the original feature set; while
feature extraction: you build a new set of features from the original feature set.

Examples of feature extraction: extraction of contours in images, extraction of digrams from a text, extraction of phonemes from recording of spoken text, etc.

Feature extraction involves a transformation of the features, which often is not reversible because some information is lost in the process of dimensionality reduction.


General question on EDA, correlations, classification, ML

If the relation between predictors is nearly 0, it's always better to drop that feature, the caveat here depends on the domain knowledge you have.

Did you check the correlation between B and the target variable and also A and target variable? if it's negative drop it, If it's significantly high .i.e greater 0.7, use that as your feature.

Yes, pair plots and scatter plots are really important, but it would be tedious to plot features with 150 variables.


If i understand correctly you want to make sure your results will remain constant and won't change?
Or in other words, are you trying to make sure your results are reproducible?

If so, this FAQ in the scikit-learn site has exactly what you are looking for.

I'll also repeat what it says here:
An algorithm might have multiple points that introduce randomness to the process and thus introduce randomness to the result.

One method to make sure your result are constant is to set every possible random_state available in the functions that you use.
The risk in this method is that you might miss some of the places that uses a seed.

The second method, which they also describe in the FAQ, is to set a global seed once.
scikit-learn will use it for all it's random processes.
So setting a global seed like this

import numpy as np
np.random.seed(42)

will make your results constant and reproducible.


Cross between an edit distance algorithm and a phonetic algorithm

The operations in the edit distance can have different weights that you can try to set manually. In Python, you can use e.g., strsympy or weighted-levenshtein package.
There is also Learnable Edit Distance algorithm that allows learning the operation weights from data. I think it might be a good idea first to estimate the operation cost first manually and then fine-tune it with the learning algorithm.  Unfortunately, I am not aware of any reasonable implementation.
There is also a CRF-based classifier that is based on the learnable edit distance. Based on training data, it estimates the probability of one string being a transcription of another one. This approach requires positive and negative training examples.


You are simply defining your array so that it is made of python sets. That is a different data structure which is not able to be  multiplied, unlike an array.

Just change your code to this:

array1 = np.array([[1,2,3,4],[5,6,7,8]])

The only difference is using square brackets instead of curly ones. These are python list objects (or standard arrays).

array2=array1*array1
print(array2)

[[ 1  4  9 16]
 [25 36 49 64]]


How can I do tree_method ='exact' in XGBoost classifier?

Indeed tree_method is a parameter for Tree Booster. There are 4 choices, namely, auto, exact, approx and hist. The default is set to auto which  heuristically chooses a faster algorithm based on the size of your dataset. Approximate algorithm (approx) is chosen in your case since your dataset is huge.

To change the tree construction algorithm, you have to pass tree_method in the train function as follows:

import xgboost as xgb
...
parameters = {'tree_method': 'exact'} #You can add other parameters as well
model = xgb.train(parameters, x_train, num_round) #num_round is the number of rounds for boosting
prediction = model.predict(x_test)
...

I hope this helps.


Neural Network Loss Function - Mean Square Error: questions about what 'n' signifies

First, try to understand a few points -

Output Neuron value and the prediction both are the same things. In the case of Classification, we convert the output probability to Class based on a Threshold.

MSE is used in Regression and In a regression problem, you mostly have one output Neuron e.g. Price. You may have more if you want to club multiple targets e.g. Bounding Box problem

The "N" in the denominator is the number of different errors calculated which is basically the number of samples in context.
With "In context", I meant if it's at the time of backpropagation it is the batch_size. if it's at the end of epoch/training it's the whole training dataset.

what does $h_θ(x^i)$ represent?

It is the prediction(value of output Neuron) for $i_{th}$ sample.


Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

Perhaps you could use word embeddings to better represent the distance between certain skills. For instance, "Python" and "R" should be closer together than "Python" and "Time management" since they are both programming languages.

The whole idea is that words that appear in the same context should be closer.

Once you have these embeddings, you would have a set of skills for the candidate, and sets of skills of various size for the jobs. You could then use Earth Mover's Distance to calculate the distance between the sets. This distance measure is rather slow (quadratic time) so it might not scale well if you have many jobs to go through.

To deal with the scalability issue, you could perhaps rank the jobs based on how many skills the candidate has in common in the first place, and favor these jobs.


Do logistic regression and softmax regression do the same thing?

There is a key difference:

Softmax regression provides class probabilities for mutually exclusive classes.
Logistic regression treats class membership for each class separately. Classes do not need to be mutually exclusive.

The two are equivalent for a scenario with two mutually exclusive classes - e.g. a "positive" and "negative" class - where softmax would have two outputs summing to 1, and logistic regression would have one output giving probability of the "positive" class.

I am not certain of the results comparing logistic regression one-vs-all (taking max output) with softmax regression on the same multi-class problem. I would expect the performance to be quite similar. Neither model copes well with non-linear relationships between input and target classes.


I think there is 2 ways to do it.

Dimensionality reduction

Using technique such as PCA, you can represent your data in lower number of dimensions.

This is a good approach if you want to represent all your data in one graph.

Dimension removal

Another way to visualize is simply to visualize only a few dimensions at a time. For example you can visualize a/b, c/d, a/c in 3 different 2D graphs.

While much simpler, this approach does not allow you to have a single graph visualization.


Find parameters to maximise output score

In this kind of optimization problem a genetic algorithm is often a good approach, assuming computing the value of  f() is not too costly.
The idea is to represent the 6 parameters as "genes". In the first generation their values are assigned randomly, then each "individual" in the generation (combination of parameters) is evaluated (i.e. calculate f), and the top performing "individuals" are selected. The next generation is obtained by cross-over and random mutation, and the process is repeated until f converges to a maximum.


Can we consider high correlation to be a good predictor?

In this case, I would fit a linear model (lm(tweets ~ cases)) and then use bootstrap or cross-validation to test the model.


Classes of neural nets and their applications

Convolutional Neural Networks have consistently outperformed other methods for image recognition and related tasks. In fact, beyond a certain image size it is not practical to train a fully-connected network on raw pixels due to the enormous number of parameters that would be required. The number of parameters in a CNN however grows much more slowly as you increase the input image size, since they usually do not require larger filters, and the number of parameters per filter is independent of input size.
With that said, there is a much more remarkable demonstration of the power of the architecture of a CNN for image processing in the recent paper Deep Image Prior. The abstract reads:

Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, superresolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs.
Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity.

So they were able to demonstrate that the raw network prior to any training was powerful enough for sophisticated tasks, which is the strongest and most direct illustration of the power of the specific architecture itself that I have seen.
The answer is not so clear for time series prediction, since CNNs with various tricks have performed on par with RNNs for some datasets, but in general RNNs seem much more suited to the problem, since they can handle inputs of arbitrary length, and they can store state.
However, either method will significantly outperform previous methods such as passing in a sliding window of fixed size into a fully-connected network. Even when a fully-connected network is combined with some state-preserving model such as ARIMA it still cannot match the performance of CNNs or RNNs. See Deep Learning for Time-Series Analysis.


How to correctly calculate average F1 score, precision and recall of a Named Entity Recognition system?

There is a quite detailed comparison with references here: https://towardsdatascience.com/a-tale-of-two-macro-f1s-8811ddcf8f04

Basically the two definitions are used and both can be considered valid. For the sake of clarity I would recommend mentioning which definition you are using when you report your results.


What is the difference between CountVectorizer token counts and TfidfTransformer with use_idf set to False?

Actually, the documentation was pretty clear. I'll keep it posted in case someone else searches before reading:
The TfidfTransformer transforms a count matrix to a normalized tf or tf-idf representation. So although both the CountVectorizer and TfidfTransformer (with use_idf=False) produce term frequencies, TfidfTransformer is normalizing the count.


Select samples from a dataframe in python

If I understand your question correctly you can break this problem down into two parts:
1. Figuring out which country occurs most frequently and then
2. Subsetting the pandas dataframe to that country

import pandas as pd
from scipy.stats import mode

# 1
mock_df = pd.DataFrame([{'country': 'a'}, {'country': 'b'}, {'country': 'a'}])
print(mock_df)
#   country
# 0       a
# 1       b
# 2       a
mode_country, _ = mode(mock_df.country)

# 2
mock_df[mock_df.country == mode_country[0]]

#   country
# 0       a
# 2       a
```


Modeling Encoder-Decoder according to instructions from a paper

What is the input and output? Assuming your input is a sequence of documents, you would need to convert it into some sort of embeddings (TF-IDF, any other). After that comes the RNN - which looks ok to me. The output of the RNN is equal to the number of timesteps (num of words in each doc?). For each timestep we predict using a Dense final layer. You have 9 outputs. Is it some sort of a classification across 9 different types of categories or levels of citation counts?
Overall looks ok with the above assumptions other than the fact that you are feeding raw data instead of using an embedding layer


How to calculate $\phi_{i,j}$ in VGG19 network?

In section 2.2.1 of the paper, they state that they use euclidean distance.  I'm going to take your word that there are 512 filter activations in that layer; if I'm reading this right, there aren't 512 feature spaces, there is a 512-dimensional feature space that they are calculating euclidean distance in. So your distance function between two images $p$ and $q$ is just the standard Euclidean distance formula:

$$ d(\mathbf{p},\mathbf{q}) = \sqrt{\sum_{i=1}^{512}(p_i - q_i)^2}$$

where $\mathbf{p}$ and $\mathbf{q}$ are vectors holding the corresponding filter activations of $p$ and $q$.

Edit: Above the horizontal rule is my original answer which is wrong (or incomplete). What I think is happening is that the authors are taking the euclidean distance as above for each position in the feature maps at the $i,j$ layer, and averaging those distances to generate a scalar loss value. So for a 7x7 feature map, they'd be taking 49 512-dimensional euclidean distances and averaging them to get the VGG19 5,4 loss. This is how I read equation (5) in section 2.2.1 in their paper. I think the missing piece is that the authors don't bother with the square root in the euclidean distance formula. As discussed below, I think the notation is unclear.


ValueError: Found input variables with inconsistent numbers of samples: [13366, 1735

After you apply dropna on your out, some rows are removed but the same is not done for your input. Hence, they have different number of rows, leading to your error.


My training accuray is 1.0 but the predictions on the training data are wrong

Chances are, you are overfitting your data, you need to closely monitor the validation accuracy, as if it diverges from the training one too much, then you enter overfitting territory.

Also, using just 3 pictures for training is just too little, your network won't be able to generalize properly with such an small input.


Confidence score for all observations is between 0.50 - 0.55

Random Model Classifier

It feels like there's a word missing here, you probably mean "Random Forest [model] classifier" or "Random Fields classifier"?

the confusion matrix

This is an important piece of information, because if you have a new (annotated) sample this allows you to compare a few things:

compare the distribution of the classes in the original test data and the new dataset
calculate the expected performance by class, analyze the differences if you evaluate the model against the new sample
...

I would like to know is it right to say that "model might have learn the pattern even tough all the confidence score is between 0.50 and 0.55"

It would be technically correct to say that indeed, but maybe not for the reasons that you mention (see below). While this predicted value is often called a confidence score or probability, it doesn't necessarily represents the confidence of the model in a normalized range [0,1] like one would expect, even though ideally it's supposed to. The details depend on the exact type of model, but in general in ML the model is only as good as the data it's trained with so most of the time the predicted score is just the "best guess" of the model. So it shouldn't be interpreted as "probability of class C", and anyway in the case of classification the only use of the value is to compare two instances x and y relatively to each other: if x has a higher predicted score than y, then the model considers x more likely than y to be in class C. The actual value doesn't matter and it's better not to rely on it.

Because I would like to explain that since the all the predictions score are concentrated around a single value it is not right to say that the model learnt and probably if some one would have tried to plot AUC it would have been straight line.

This conclusion is incorrect. To be clear, it might be true in some particular case, but one cannot conclude this from the information given in the question. Why? Because the ROC curve (and the AUC) don't depend at all on the actual values of the score, they depend only on the order of the instances according to the score. The ROC curve shows the performance for different theresholds separating the predicted positive/negative instances. Whether the scores range in [0.51,0.52], [0,1] or even [-1234,+5678] doesn't have any influence, it's the proportion of negative instances under the threshold and positive ones above it which determine the performance at any given point.
In case you're interested, I gave some more detailed explanations about ROC curves in this answer.


Is converting a numerical target to binary helpful?

When you convert a variable from numerical to binary, what happens is that you lose information about the magnitude of the variable and summarise it into a >=X variable.

It has advantages and disadvantages depending on the use you will give to your model:

If your model has to determine the exact amount of "how high" is your pollution, then is not a good idea to convert it because you are loosing crucial information, but if the city major declares a curfew if the pollution is bigger than x, then a binary output is not only good idea but necessary for the decision you are achieving with it.

When you convert the variable to binary, the hypothetical model you are achieving has different kind of information that the model with numerical variables:

You will not be comparing exact values (measuring RMSE), you will be measuring membership to a set (measuring AUC).
If your client wants to measure wheter you trespassed or not a threshold, you should explain to him what is achievable with that model and ask him it that meets the requirements of his job.


I think this is a reasonable question. Here is what I do:

Peak at the first few rows
Visualize the distribution of the features I care about (histograms)
Visualize the relationship between pairs of features (scatterplots)

I downloaded the abalone dataset from the UCI Machine Learning repository here. Let's say I care about how height and diameter can be used to predict whole weight. For completeness, I've included the step of reading the data from file.

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = pd.read_csv("abalone.data", header=False)
data.columns = ["sex", "length", "diameter", "height",
                "whole_weight", "shucked_weight",
                "viscera_weight", "shell_weight", "rings"]

Now we can take a peak at the first few rows:

data.head()

Now, I know that the variables I care about are floating point values and they can be treated as continuous. I want to take a look to see how these three variables are distributed:

fig = plt.figure(figsize=(20,5))
plt.subplot(1, 3, 1)
plt.hist(data['diameter'], normed=True)
plt.title("Diameter")
plt.subplot(1, 3, 2)
plt.hist(data['height'], normed=True)
plt.title("Height")
plt.subplot(1, 3, 3)
plt.hist(data['whole_weight'], normed=True)
plt.title("Whole Weight")
plt.show()

Great! Now, I know that diameter and whole weight are skewed left and right (respectively). I also know that there are some outliers in terms of height (which is why matplotlib gives me extra room to the right of the distribution). Finally, I'd like to see if I can find any visual patterns between my predictors and outcome variable. I use a scatter plot for this:

plt.figure(figsize=(15,5))
plt.subplot(1, 2, 1)
plt.plot(data['diameter'], data['whole_weight'], 'o')
plt.title("Diameter vs. Whole Weight")
plt.ylabel("Whole Weight")
plt.xlabel("Diameter")
plt.subplot(1, 2, 2)
plt.plot(data['height'], data['whole_weight'], 'o')
plt.title("Height vs. Whole Weight")
plt.ylabel("Whole Weight")
plt.xlabel("Height")
plt.show()

Here, I see there is a non-linear relationship between diameter and whole weight and I'm going to have to deal with my height outliers. Now, I'm ready to do some analysis!


ValueError: The number of classes has to be greater than one; got 1 class in MultiLabel Classification problem

Could you see how many classes are in training and in test?

y_train.unique()
y_test.unique()

It can happen that at some point you got a split that has only one label, if your data is small or heavily imbalanced there are more chances. One way to fix it is stratifying the target.

train_test_split(data[features], data[labels], test_size=0.20, shuffle=True,stratify=data[labels])


How do Classification Algorithms such as Catboost and Random Forest parse test data?

Both the models i.e. RandomForest/CatBoost are based on a Decision Tree.
A Tree splits on feature values. Check this Tree for Iris dataset. Scikit-Learn


What are the benefits and tradeoffs of a 1D conv vs a multi-input seq2seq LSTM model?

I suggest you transform your 6 sequences into 1 since they have the same lengths. You will have 1 sequence with more features for each timestamp.

It is hard to tell you which method would be better. But a LSTM model seems to be well suited for you.


Finding dominating attributes with in the clusters generated

One way would be:

Cluster data.
Within each cluster, count the number of males and females.
Label each cluster with the majority gender.

Repeat steps 2-3 with each demographic feature of iterest.


How to use TFIDF vectors with multinomial naive bayes?

When predicting the most likely class, the factorial terms are present in the probability calculation for every class, so they can essentially be ignored in the calculation. This leaves word probabilities which are raised to fractional powers, which are easily calculated. See this paper for a more detailed explanation: http://www.cs.waikato.ac.nz/ml/publications/2004/kibriya_et_al_cr.pdf.


Using LSTMs for modelling and forecasting several time series generated by the same process

You can frame the problem this way.

Consider each car as a sample of your dataset and use $Car^{i}$ to denote the $i^{th}$ sample. Then for each sample, you have the following features:

Time series of cars' condition $C_0^{i}...C_n^{i}$
Other time-invariant features like colour $A^{i}$ and engine size $E^{i}$

(I am not sure how mileage fits here, do you have a time series on this as well? Intuitively mileage changes over the life of a car so it should be time series as well)

My understanding of your question is how to incorporate those time-invariant features into an LSTM model?

Instead of only passing the $C_{t-1}$ as LSTM cell input you pass it with a vector $[C_{t-1}^i, A^i, E^i]$. Here, using Tensorflow's terminology, you input_size is 3 instead 1. $A^i$ and $E^i$ is always the same at each timestep but different across samples.


cv2.selectROI() returns a 2D rectangle with (x, y, width, height) (see the Rect constructor - that object is created from selectROI()).

So, if you want to measure the apparent width in pixels, you need marker[2].


BERT :dropout(): argument 'input' (position 1) must be Tensor, not str

Which transformer version are you using? I had to pin mine to transformer == 3.5.1 to mitigate that problem, when the hugging face team updated their transformer to 4.0 things started to break.
Hope it helps


What is the significance of underflow during parameter update using stochastic gradient descent?

Underflow is happening because the result of that calculation is a number of smaller absolute value than your computer can actually represent in memory.
Underflow is not directly related to the performance of the model (e.g., learning rate or local minimum). Underflow is the interplay between computer software and hardware.
I suggest rescaling the data. For example, batch normalization often stops underflow errors.


Individual differences in response time (RT) experiment - searching for the right test

In most individual differences research, subjects are additionally coded along some dimension, such as young/old, low/high working memory, normal/cognitively impaired. Without this, you’re simply looking at individual variation, which doesn’t seem to address any question (what is your research question?). If you simply want to look at variation in overall RT on a subject level, I’d construct CDFs of mean RT collapsing over color, or perhaps for each color separately.  This will simply demonstrate that people differ, and you’ll find the distributions adhere to roughly ex-Gaussian. If you mean to test differences between colors, you can use ANOVA or compare CDFs using K-S tests, but again there isn’t a clear research question here.


How can I detect significant changes in my data?

Looking for "outliers" will not give you the appropriate answer because you have a trend.
Moreover the trend is not linear, so the answer will depend on the trend model.

What you should do is :

(1) fit the trend. I choose a logistic function with a MAD but an exponential with LSE growth would have lead the same results

(2) compute the residuals $e(t)$ as the observed data minus the model.

(3) search for the outliers on the residuals. I tool Tuckey's fence with $|e(t)|$ greater than 1.5 times the interquartile.

The attached graph shows that obviously you should choose an exponential growth with a multiplicative error, thus minimizing the MAPE and computing the errors in percentage.


Your series is indeed homogeneously-typed and you can check it's type:

s = pd.Series(data=[1,2,3,4,5,'x'], index=['a','b','c','d','e','f'])
s.dtype
> dtype('O')

where 'O' is for "object". However, if you check the type of the individual elements of your series, they are different:

type(s['a'])
> int
type(s['f'])
> str

I think the key thing to remember is that if your Series or DataFrame Column is non homogenously-typed (eventhough technically homogeneously-typed of type "object"), then there are certain pandas functions that won't work.

Here is a better explanation from the pandas documentation's Essential Basic Functionality:

  If a DataFrame or Panel contains homogeneously-typed data, the ndarray
  can actually be modified in-place, and the changes will be reflected
  in the data structure. For heterogeneous data (e.g. some of the
  DataFrame’s columns are not all the same dtype), this will not be the
  case. The values attribute itself, unlike the axis labels, cannot be
  assigned to.

  Note: When working with heterogeneous data, the dtype of the resulting ndarray will be chosen to accommodate all of the data
  involved. For example, if strings are involved, the result will be of
  object dtype. If there are only floats and integers, the resulting
  array will be of float dtype.


Is it possible for a neural net to score as high as a different form of supervised learning?

Generally speak, no. Deep learning models struggle to compete when it comes to tabular data.

If we head over to kaggle were people compete to build the best model we find that usually the best performing non-ensemble models for this kind of data are gradient boosting trees. More specifically it tends to be either XGBoost or more often now days it is LightGBM that performs best. Both are highly optimized implementations of gradient boosting trees.

Feature engineering and parameter tuning are both important to get extra performance. But the gap does not shrink since you would get the same improvements if you did it on a gradient boosting model instead.

Actually other models tend to gain more from feature engineering than neural networks since one of the strengths of neural networks is that they perform a sort of automatic feature engineering when they are trained.

Neural networks usually shine in the domain of unstructured data such as text and images.


Logistic regression prediction changed after executed couple of time

There is some randomness in the results from selecting/shuffling data that is used in the model.

If you don't want that, you could set a fixed random_state (seed) in your model.


Relationship between VC dimension and degrees of freedom

As stated by Prof Yaser Abu-Mostafa-

  Degrees of freedom are an abstraction of the effective number of parameters. The effective number is based on how many dichotomies one can get, rather than how many real-valued parameters are used. In the case of 2-dimensional perceptron, one can think of slope and intercept (plus a binary degree of freedom for which region goes to +1), or one can think of 3 parameters w_0,w_1,w_2 (though the weights can be simultaneously scaled up or down without affecting the resulting hypothesis). The degrees of freedom, however, are 3 because we have the flexibility to shatter 3 points, not because of one way or another of counting the number of parameters.


test accuracy of text classification is too less

You should try to perform cross-validation using all your datasets. Maybe your test set has a completely different distribution than your training set (with such a small number of samples out of all possible subtitles, it is likely). Did you create this dataset?

What is the size of the final input vectors you are using and what size of vocabulary are you using for this?

At least in the case of the SVM, it seems that your model is overfitting and you need to increase the regularization.
More training data will always help.
Not sure what are your options here.
There are a lot of options based on the small number of samples, you can try using clustering (KNN, K-Mean).


Datamodel for cluster analysis terms & segmentation

clustering algo would take any data type as long as it is measurable, ideally shouldn't be string (which can be translated into vectors using topics).
If you tell me what application you are using i can guide you on how to go about it step by step


Correlation as an evaluation metric for regression

The organizers might deem the direction of predicted change more important than the magnitude, i.e., it is more important that your prediction is high when the known value is high (and vice versa) than to get as close to the known value as possible. The measurements might be noisy anyway.

One, fairly robust way of optimizing for it would be by grid-searching for the local optimum, like in this QA.

However, you should also take note that algorithms tweak ther internal parameters when fitting according to some loss function. Some algorithms accept custom cost functions and derivatives, yet some implementations don't. Information-theoretic measureas are standard in classification while MSE in regression.

Theoretically, you should be able to tell your Random Forest (or another algorithm accordingly) what the optimal split is as a function of pearson correlation.


What is the best way to deal with imbalanced data for XGBoost?

I think using something like this could help in your case.

https://github.com/dmlc/xgboost/blob/master/demo/kaggle-higgs/higgs-train.R#L17
http://xgboost.readthedocs.io/en/latest/how_to/param_tuning.html

Hope this helps at least a bit!


What features from sound waves to use for an AI song composer?

First off, ignore the haters.  I started working on ML in Music a long time ago and got several degrees using that work. When I started I was asking people the same kind of questions you are. It is a fascinating field and there is always room for someone new. We all have to start somewhere.

The areas of study you are inquiring about are Music Information Retrieval (Wiki Link) and Computer Music (Wiki Link) .  You have made a good choice in narrowing your problem to a single instrument (monophonic music) as polyphonic music increases the difficulty greatly.

You're trying to solve two problems really:

1) Automatic Transcription of Monophonic Music (More Readings) which is the problem of extracting the notes from a single instrument musical piece.

2) Algorithmic Composition (More Readings) which is the problem of generating new music using a corpus of transcribed music.

To answer your questions directly:

  I think that this would be an unsupervised learning problem, but I am
  not really sure.

Since there are two learning problems here there are two answers. For the Automatic Transcription you will probably want to follow a supervised learning approach, where your classification are the notes you are trying to extract.  For the Algorithmic Composition problem it can actually go either way. Some reading in both areas will clear this up a lot.

  What features from the sound wave I should extract so that the output music is melodious?

There are a lot of features used commonly in MIR. @abhnj listed MFCC's in his answer but there are a lot more.  Feature analysis in MIR takes place in several domains and there are features for each.  Some Domains are:

The Frequency Domain (these are the values we hear played through a speaker)
The Spectral Domain (This domain is calculated via the Fourier function (Read about the Fast Fourier Transform) and can be transformed using several functions (Magnitude, Power, Log Magnitude, Log Power)
The Peak Domain (A domain of amplitude and spectral peaks over the spectral domain)
The Harmonic Domain

One of the first problems you will face is how to segment or "cut up" your music signal so that you can extract features.  This is the problem of Segmentation (Some Readings) which is complex in itself.  Once you have cut your sound source up you can apply various functions to your segments before extracting features from them. Some of these functions (called window functions) are the: Rectangular, Hamming, Hann, Bartlett, Triangular, Bartlett_hann, Blackman, and Blackman_harris.

Once you have your segments cut from your domain you can then extract features to represent those segments.  Some of these will depend on the domain you selected.  A few example of features are: Your normal statistical features (Mean, Variance, Skewness, etc.), ZCR, RMS, Spectral Centroid, Spectral Irregularity, Spectral Flatness, Spectral Tonality, Spectral Crest, Spectral Slope, Spectral Rolloff, Spectral Loudness, Spectral Pitch, Harmonic Odd Even Ratio, MFCC's and Bark Scale. There are many more but these are some good basics.

  Is it possible, with recurrent neural networks, to output a vector of sequenced musical notes (ABCDEF)?

Yes it is. There have been several works to do this already. (Here are several readings)

  Any smart way I can feed in the features of the soundwaves as well as sequence of musical notes?

The standard method is to use the explanation I made above (Domain, Segment, Feature Extract) etc.  To save yourself some work I highly recommend starting with a MIR framework such as MARSYAS (Marsyas).  They will provide you with all the basics of feature extraction.  There are many frameworks so just find one that uses a language you are comfortable in.


How to scale exponential data for a regression problem?

Yes, there is. Instead of Min-Max scaling, that shrinks any distribution in the [0, 1] interval, you can Scale the variables (in statistics, they are called Z-scores). The formula is:

(x - mean(x)) / stdev(x)

This will zero-center the distribution, and scale all the variables to a standard deviation = 1.

I strongly prefer this technique to Min-Max Scaling, since the latter is too sensitive to outlier observations and generates problems unseen, out-of-scale datapoints.

Additionally, if your analysis allows for that, consider taking the log() of a data distribution that grows exponentially. That would coerce it to a more linear growth. (I'm not sure this is what you need, it's just a possibility.)


A way to Identify tuning parameters and their possible range

Not a complete answer, but was too long for a comment.

I always first try to see how the default parameters perform. Then from the documentation or some reading, you can see what is each parameter global influence (by influence I mean maybe increasing parameter X means complexifying the model, or parameter Y means increasing the convergence speed towards a solution). Depending on the first result you get, pick up one parameter, the one that seem to have the most influence on the model, and make it vary a bit in the way that make sense from your first results. If things improve on the validation set, keep moving the value this way, if not do the opposite. Often times you get good results without tuning every single parameter.

This is a method by hand, it is not optimal. But as you precise that you are a beginner in machine learning, I believe it is the best way to learn to "feel" what usually impact the performance of an algorithm as Xgboost and what impacts less and that therefore can be overlooked for a primary coarse tuning.

https://xgboost.readthedocs.io/en/latest/parameter.html has some nice pieces of information about what parameter impacts what. Don't hesitate to ask more precise questions about some specific parameters if you need :)


R - Error in KNN - Test and training differ

You are avoiding the error with sampling purely by chance (the words are common). Simplest way would be to merge training and test, construct a tdm on a joined set, separate into training and test again and then purge some columns (the constant ones in the train set, as they correspond to words occurring in test only => useless for training).


Is an Arma model equivalent to a 1-layer Recurrent Neural Network without activation function?

It's correct. The reason it sounds so weird is that a 1-layer-NN without activation function is simply a linear map, so it's equivalent to any linear model, the only difference being the inputs having some interpretation.

This even holds true for any NN, no matter the number of layers, without activation functions. The reason: A k-layer-NN is just k matrix multiplications (=linear maps) with activation functions in between. If you remove the latter you are left with a long chain of matrices that you can simply multiply to get one matrix that now defines your linear map.

Many deep learning researches concluded that giving the networks the possibility to create non-linear maps is what makes deep learning so powerful, even when using seemingly unsophisticated activation functions such as the ReLU.


Traditional ML Model or Deep Learning for ~200-300 samples?

The less data you have, the less complex your model can be. Otherwise you will overfit your data. There is not really a good way for me to judge what model is appropriate for you without knowing a lot about your dataset, but I doubt you will get anything sensible from 200 data points with a deep learning model. Try some simpler models like bag-of-words and see whether you overfit. If so, you will just have to collect more data. If not, try models of increasing complexity and see how accurate you can get before overfitting sets in.


One way is to encode your input in a fixed size dimension. That is, you can use an RNN, like an LSTM, with padding and its output should be the input of an ANN.


Are there any good out-of-the-box language models for python?

I also think that the first answer is incorrect for the reasons that @noob333 explained.
But also Bert cannot be used out of the box as a language model. Bert gives you the p(word|context(both left and right) ) and what you want is to compute p(word|previous tokens(only left contex)). The author explains here why you cannot use it as a lm.
However you can adapt Bert and use it as a language model, as explained here.
But you can use the open ai gpt or gpt-2 pre-tained models from the same repo
Here is how you can compute the perplexity using the gpt model.
import math
from pytorch_pretrained_bert import OpenAIGPTTokenizer, OpenAIGPTModel, OpenAIGPTLMHeadModel
# Load pre-trained model (weights)
model = OpenAIGPTLMHeadModel.from_pretrained('openai-gpt')
model.eval()
# Load pre-trained model tokenizer (vocabulary)
tokenizer = OpenAIGPTTokenizer.from_pretrained('openai-gpt')

def score(sentence):
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    loss=model(tensor_input, lm_labels=tensor_input)
    return math.exp(loss)

a=['there is a book on the desk',
                'there is a plane on the desk',
                        'there is a book in the desk']
print([score(i) for i in a])
21.31652459381952, 61.45907380241148, 26.24923942649312


The questions I ask myself when I see learning graphs are the following ones:

Is the loss decreasing and the accuracy increasing ? if yes, your network is learning and everything works fine, which is already a good new.
Have we reached a kind of plateau ? (in accuracy especially), which means learning is over. (Here maybe you could train your network a bit more, it doesn't seem to have reached a plateau yet).
What is the value of that plateau ? Final validation accuracy is the most important value we seek when learning.
Do the curve have an exponential shape ? This is a tricky one and i'm not sure everyone does this, but i like having exponential shape curves when learning (which is not possible with every optimizer), so in your case (linear curves) i would try increasing the learning rate to get a faster learning.
Is the validation accuracy decreasing at the end of the training ? in which case it means we are overfitting and need to stop learning earlier. (Not your case here)

Hope this helps, this is only my tips for analysing learning graphs and may not be what everyone ask theirself when facing these graphs.


Comparing data sets with different measurements

You're right, you cannot directly compare the scores, since they are extracted from different samples, using different metrics and parameters, etc. Is exactly as comparing apples with airplanes.

But is not so wrong to play around with the placement in the rank: you are comparing placements with placements, so it makes perfectly sense. If Sweden is always at the last place, this definitely means something.

Maybe you can also think of weight those placements by some parameter/set of parameters which takes in account the subject of the ranks.
As example, seriousness of the threats (ex. infection by AdWare can be less important than identity theft, etc), coverage of the population by the authority which published the results, reputation of the authority, etc.

You can also extract uncertainties on those ranks, maybe based on the statistical samples taken in account by the authorities, and so on.

Cheers


inconsistency occurred when using cross_val_score in python

One reason might be your train_test_split() setup. Using shuffle=False means you just use the first 80% examples of your data to train with no randomness. Have a look at your labels:

>>> y
Out:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

As you can see the iris dataset is sorted by its classes from 0 to 2. This means your train_test_split() will be unbalanced since class 2 will be underrepresented:

>>> y_train
Out:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

And even worse your test data only has one class:

>>> y_test
Out:
array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2])

On the other hand cross-validation will use a stratified split preserving the class distribution for all folds as described in the sklearn user guide:

  For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

Which I'd expect to be leading to better results. Now, also giving the train_test_split() this advantage of balanced classes (stratified labels) gives me a different result:

>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, stratify=y)

runfile(...)

>>> model.score(X_test, y_test)
Out: 1.0


You can get a dendrogram from any hierarchical clustering method. The tricky thing here is how to compute the distances between the words. If efficiency is your main concern, I would consider using HDBSCAN clustering.
The Jaro-Winkler distance was originally designed for such tasks. There is an efficient implementation in the python Levenshtein package, but it still might be too slow for computing the distance between all pairs of words.
If this is still too slow, you can opt for some more approximative methods and represented the strings as vectors - bags of characters (or character n-grams). You lose some information about how exactly characters are ordered in the string, but you will get them represented as vectors and vector comparison is fast. In this way, you can use algorithms that do not require comparing all pairs of data points such as K-Means.


Why can't I use data augmentation with a pretrained convnet?

Found the answer in stats.stackexchange.com. Hopefully this helps anyone else with the same question.

feature extraction: freezing convolutional base vs. training on extracted features


I think SMOGN will work for your problem. The method is described in a paper titled: "SMOGN: a Pre-processing Approach for Imbalanced Regression". You can find it on arXiv. There is also a python implementation called "SMOGN" which can be installed through PyPI. You can find the package description at https://pypi.org/project/smogn/


What could be a right datascience approach of doing fuzzy string matching in large amount of short text data?

You could try this:

Take distinct records from List2
Clean List1 of any extraneous characters or symbols
Use a fuzzy logic package like Python's fuzzywuzzy. It has partial matching options.


Input data of variable length - two scenarios

For scenario 1, if you have enough data for each runner, you could build separate models for them otherwise you can add runner as a categorical variable by one hot encoding your runners and then trying out your model.
For scenario 2, you can create fixed sized vectors for each string by handcrafting features, such as count of consonants, count of vowels, presence of special characters, etc.


Both approaches can be done: I recommend to validate after every batch when your are just playing around with your gradient method, to see if the validation accuracy goes up or down and to figure out how everything is going.
In this setting, you can adopt early stopping, although there are many ways to prevent from overfitting. Early stopping checks, at the end of an epoch, your validation accuracy, and saves the model if it is the best so far. If the validation accuracy does not increase in the next $n$ epochs (and here $n$ is a parameter that you can decide), then you keep the last model you saved and stop your gradient method.
Validation loss can be lower than training loss, this happens sometimes. In this case,  you can state that you are not overfitting.


Can reducing the number of classes in multi-label classification increase performance?

Yes, merging / dropping classes will tend to increase performance. If you merge classes, there will be more examples per class which will tend to decrease the variance of a model. Reducing the total of number classes has the potential to allow the model better fit the data, reducing bias. Many models are constrained by learning capacity.


Is information lost when converting an image to a single vector for analysis?

The short answer is it depends on two things: what definition of information you use and if you really use that information.

From information theory point of view, if your transformation is reversible than the information is there. This happens because you can apply the inverse transformation to recover the original. So nothing is lost. This is similar with what happens in various lose-less compression algorithms.

So, information is there, but is encoded in another place, specifically in the transformation function. As a consequence, to learn also the position you have to create those new features to express that information. Not also that when you have a classical learner this information is not understood by learner. Take for example a linear model. There is no difference between $f_1 = \beta_0 +\beta_1 x_1 + \beta_2 x_2$ and $f_2 = \beta_2 x_2 + \beta_0 + \beta_1 x_1$. So learning with a simple linear model is impossible to use directly this locality information. This happens with most learners. So, even in the original space the locality is an information in your head, and not in the data.

On the other hand if your neural net is tailored for images and expects to receive patches of images where the order of the values is important than yes, your neural net perhaps understands those relations and use them. Even so, perhaps that custom neural net has a custom way to specify the location information, so what you will have to do would be to encode this information in a format specific to that custom neural net.

On the other hand if we are talking about a standard back propagation neural net, the order is irrelevant. The model is simply not able to use the location.

As a conclusion, if you have a standard generic learning algorithm your only your only chance would be to encode this locality information somehow in the features. For example (this is a made-up example, I do not even know if that would word) one can add a new set of input variable, one for each square of 4 adjacent pixels and use the average of pixels intensity as value. You do not made that explicitly enough, but you created a new set of features which is created using the locality information.


Is a good shuffle random state for training data really good for the model?

If this is split is a train/validation split (not a hold out test set) then you should be doing cross-validation. You are going to be overly optimistic about the performance of your model for this set of features and hyperparameters if you try to split it "just right". Cross-validation will give you a more accurate portrayal regardless of your split. If this is for a train/test split (test being a hold out test set), this is a very bad practice, since you are informing your decision on how to make the split based on the performance of the test set.


Looking into the linked answer, it appears that they are directly training on X and y since they're using a GridSearchCV, which already includes a k-fold cross validation (5 fold by default). So basically you'll already have a score for the classifier by calling GridSearchCV with the defined pipeline.

That being said I'd argue that it is never really the recommended approach to directly do this without a final test step, to asses the performance of the trained model on unseen data. So even if you do a k-fold cross validation, it is advisable to leave a test set to get a final score, specially when the k-fold process involved a hyper-parameter tuning, as in this case. In such cases you  need anther validation step that is independent of the tuning.

And in relation to the second point, no, you don't need to include a StandardScaler if the data is already normalised. Though, since you're using pipeline, you might as well include all transformation logic in the pipeline, for the sake of simplicity.


Unbalanced class distributions

First, unbalanced datasets will cause your model to have a bias towards the over-represented classes. If the distribution of the classes is not very drastic then this should not cause a significant problem with any algorithm you will employ. However, as the difference between the class distribution becomes more severe you should expect to get higher false negatives for that class. Consider this, you are trying to have the model adequately identify what it means for a specific example to belong to a class. If you do not provide sufficient examples, then the model will not be able to understand the extent of the variation which exists among the examples.

If the class distribution is very different, then I would suggest anomaly detection techniques. These techniques allow you to learn the distribution of a single classification and then identify when novel examples fall within this distribution or not.

Choosing an algorithm

More classes will result in a higher dimensional output, thus contributing to the complexity of your model. For example, if you have a model which discriminates between 2 classes with a set dataset size. Then further discrimination (increasing the number of output classes) will cause the model to have higher bias. You should thus expect to see greater test error if you do not increase the size of your dataset.

If you have a set dataset X. Then you need to find the correct balance between bias and model complexity to get the optimal results. For example, a neural network based technique (highly complex) is not a good algorithm to use for a limited dataset with many output classes. However, Naive Baye's or Random Forest would be.


Normalizing a list with both positve and negative numbers to the range [-1 1]

You can do this yourself or use a library such as scikit-learn which has a MinMaxScaler.
data = [-518.8134, 480.1884, 160.4761]

# manual scaling
scaled = [-1 + 2 * (x - min(data)) / (max(data) - min(data)) for x in data]
print(scaled)
# [-1.0, 1.0, 0.3599364886029235]

# using scikit-learn's MinMaxScaler
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(-1, 1))
scaled = scaler.fit_transform([[x] for x in data])
print(scaled)
# [[-1.        ]
#  [ 1.        ]
#  [ 0.35993649]]


Regression vs. Expected value for campaign results

There are special types of regression models for these kinds of problems.

The expected loan amount for a given individual is a function of both the probability that they respond and take out some type of loan and the loan amount they take out conditional on them responding.  In mathematical terms, we can think of it as expected value of the loan amount and it can be expressed like this:

$E[loan$ $amount] = Prob(response) * E[amount | responded] + Prob(no$ $response) * E[amount | not$ $responded]$

(It seems like in your case you may be assuming that without receiving a letter an individual has a zero probability of taking out a loan, so you can drop everything after the $+$ operator, if that's the case.)

So you have to model both variables - probability and amount.  The question is how.  If you just go with the first approach you outlined, i.e., a basic regression for modeling counts like Poisson or Negative Binomial and include 0's in the model, then you will be assuming that the zeros and the nonzeros (positive values) come from the same data-generating process, governed by the same data distribution, and are influenced in the same manner by the same set of predictors.  But models like hurdle models and zero-inflated models (e.g., zero-inflated Poisson) separately model the binary outcome of whether a count variable is zero-valued or positively-valued and the conditional distribution of positive values.

(If you want to know more hurdle and zero-inflated models, here is a couple of links to Q&A's on the Cross Validated site:
https://stats.stackexchange.com/questions/81457/what-is-the-difference-between-zero-inflated-and-hurdle-models, https://stats.stackexchange.com/questions/279273/zero-inflated-distributions-what-are-they-really.)

You could also manually construct those separate binomial (logistic) and count models (using separate sets of predictors if you wanted) and try to combine the results.  One issue though is that you would need to model amounts only for those individuals who have previously responded (as you are looking for expected amount conditional on responding) and it would be difficult for you to calculate/estimate expected amounts for individuals who have not previously responded and for any new customers.


How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Here are the answers:

In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not really the order). For example: In bag of words: "I ate ice-cream" and "ice-cream ate I" are same, while this is not true for the models that treat entire sentence as ordered sequence. Thus, you cannot pick N random words in a random order.
Choosing tokens is model dependent. You can always preprocess to remove stop words and other contents such as symbols, numbers, etc if it acts as noise than the information.
I would like to clarify that lemmatizing and word-piece tokenization is not the same. For example, in lemmatization "playing" and "played" are lemmatized to "play". But in case of word-piece tokenization it's likely split into "play"+"##ing" or "play"+"ed", depending on the vocabulary. Thus, there is more information preserved.
max_length should be optimally chosen such that most of you sentences are fully considered. (i.e, most of the sentences should be shorter than max_length after tokenization). There are some models which considers complete sequence length. Example: Universal Sentence Encoder(USE), Transformer-XL, etc. However, note that you can also use higher batch size with smaller max_length, which makes the training/fine-tuning faster and sometime produces better results.
The pretrained model is trained with MAX_LEN of   512. It's a model's limitation.
In specific to BERT,as claimed by the paper, for classification embeddings of [CLS] token is sufficient. Since, its attention based model, the [CLS] token would capture the composition of the entire sentence, thus sufficient. However, you can also average the embeddings of all the tokens. I have tried both, in most of my works, the of average of all word-piece tokens has yielded higher performance. Also, some work's even suggests you to take average of embeddings from the last 4 layers. It is merely a design choice.
Using sentence embeddings are generally okay. But, you need to verify with the literature. There can always be a better technique. Also, there are models specific to sentence embeddings (USE is one such model), you can check them  out.


Does batch normalization mean that sigmoids work better than ReLUs?

See, the basic concept behind the batch-normalization is that (excerpt from a Medium article)-

  We normalize our input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning. If the input layer is benefiting from it, why not do the same thing also for the values in the hidden layers, that are changing all the time, and get 10 times or more improvement in the training speed.

Read the article here.

This is the reason why we use Batch-normalization. Now coming to your question, see as the output of sigmoid is constraints between 0 and 1, and that what is the motto of Batch-normalization.
If we use Bach-normalization with sigmoid activation, then it will be constrained between sigmoid(0) to sigmoid(1), that is between 0.5 to 0.73 ~ $frac{1}/{(1+1/e)}$.
But if we use ReLU with Batch-normalization then the output will be spread over 0 to 1 which is the good thing for us as finally, we want the output as much diverse as it can be.
So I think ReLU will be the best choice among other activations.


A couple of points I have since found myself:

I was right in suspecting that self-training could be used for PU learning. In fact, I found the original paper on PU Learning, and indeed the paper is a variation on self-training. (Oddly enough, the original authors had Positive, Unlabeled and Negative examples!)
The authors of this survey identify three families of methods: (i) two-step strategy (identify reliable negative examples in the unlabeled data and then use supervised learning), (ii) weight the positive and unlabeled
examples, and estimate the conditional probability of positive label given an example (I believe this is akin to semi-supervised self-training), and (iii) just treat the unlabeled data as highly noisy negative data.
There are also some interesting loss functions to be used with neural networks (and I imagine could be adapted for gradient boosting) described in Table 1 of this paper.


Algorithms to find average distance between data maxima

The first technique that comes to mind is cubic spline interpolation, which is good at approximating sinusoidal functions without suffering from Runge's phenomenon. Here's a visual of cubic splines applied to a similar problem:

Matlab has an implementation documented here.

Once you have a representation for the underlying function you should be able to find the local maxima easily and take the differences you are looking for.


Python library to detect a bank/financial institution name in a string

As mentioned in the comment you can use regex but you would need to define a set of rules for it. You can try with LexNLP which is trained on legal documents and use it to extract data type like address, companies and persons.


Why are ordered lists considered one dimensional?

The specific ordering is only metadata as it is not explicitly in the data itself, and the index doesn't always represent order.
If you have an additional row with the order in then it would be two dimensional. But, for example, you wouldn't be able to distinguish between ascending and descending order by looking at the index itself.
v(t) is velocity by time
data[index] is data point by location in a data object (rather than known order).
Perhaps you could argue that the index is a second dimension, but consider this, how would you represent 1D data in a computer otherwise?


Using sensor data and a know reference point infer the position of a moving robot

I see two classes of solutions to your problem:

Solutions without machine learning

The way we did this when I studied realtime systems was to analyze the data coming from the wheel sensors by hand:

First, plot the rate of rotation of the robot w.r.t. the wheel speeds. Is the rate of rotation linear with the difference in speed of the two wheels? The ratio of speeds? Try to find a relation between these variables so that you can later calculate the change in the orientation of the robot from its wheel speeds over a period of time.

Second, do the same with forward and backward movement: when the wheels are turning at rpm X, how fast does the robot move? Is it a linear relation? Find an equation for it.

Once you have these two relations defined, you can dynamically update the position of the robot by determining its direction (based on previous direction and current rate of rotation) and speed (based on wheel rpm), and adding those to the previous position.

This approach is a lot of work, but will help you understand how the wheel speeds influence your robot's movement.

Solutions with machine learning

Given the popularity of machine learning in data science, I presume it might be what you were looking for when asking your question here.

What you have is time series data with two variables: speed for two wheels.

You could drive your robot around the grid for a little while, accelerating, decelerating, turning, stopping, etc. while recording the robots position as well as its wheel speeds at regular, short intervals. This is how you would build a training dataset. You will need to find a way to record the position: perhaps you could use triangulation to locate the robot in the grid, or an overhead camera. You will end up with a sequence of data points (or multiple sequences), each with two wheel speeds (the input variables) and two position coordinates (the output variables).

You would then need to train a model on the time series data, with the position at each step of the time series as your model output (a regression with two output variables).

What kind of model would suit this purpose? It would be very interesting to see what a RNN, with LSTM or GRU units, could do in this situation. The only way to find out if it'll work, however, is to give it a try.

This approach is also quite a bit of work, and will not help you understand the impact of the wheel speeds on movement much (because the model will most likely not be interpretable).


How can I predict the post popularity of reddit.com with hidden markov model(HMM)?

I don't think it makes a lot of sense to use HMM's for this problem. What I would suggest is some kind of text-based classifier.
If you want to use a cool technique, you could use a neural network to learn based on the text of successful posts.

On the other hand, If you want to use an easy technique you could make a predictor for the popularity such as a regression model (try to predict upvotes).


Sentiment Analysis: Creating dictionary from dataset

When looking on the probability of word occurrence, you will get stop words and other popular words. You are interested in words that appear more in the comments (assumed hate related) than in normal use.

Get a neutral  resource (e.eg., a German newspaper, the German wikipedia, maybe Google ngrams for German). Compute the probability on the neutral source $P_{neutral}(word)$, the probability of the comments $P_{comments}(word)$ and look for the words of lift  $\frac{P_{comments}(word)}{P_{neutral}(word)} > 1$. These are the words that are more popular at the comments.

As @chi wrote, many repositories can both give you a head start and help you tune the needed lift threshold (you might want words that appear much more often in the comments).

After this phase you might need do do a finer analysis. For example, I guess that there will be politicians names that will appear more often in the comments. See here for a possible approach.


I believe that the following image (original link) will be helpful to understand. 1D convolutions work exactly the same way as 2D convolutions, the main difference is how the filter/mask is applied. 1D convolution is intuitively like a sliding window of a certain width.

Many packages like Keras or PyTorch have native 1D convolution function/modules, so I would recommend checking their documentation and perhaps source code for deeper understanding.


Is there a way to put a separate line between clusters for k-Means Clustering?

there are two answers to this question.

The first one is yes, you can do it with python code. From a Sklearn tuto, you can plot the decision boundary by using meshgrid:

# Step size of the mesh. Decrease to increase the quality of the VQ.
h = .02     # point in the mesh [x_min, x_max]x[y_min, y_max].

# Plot the decision boundary. For that, we will assign a color to each
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# Obtain labels for each point in mesh. Use last trained model.
Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(1)
plt.clf()
plt.imshow(Z, interpolation='nearest',
       extent=(xx.min(), xx.max(), yy.min(), yy.max()),
       cmap=plt.cm.Paired,
       aspect='auto', origin='lower')

plt.plot(X[:, 0], X[:, 1], 'k.', markersize=2)
# Plot the centroids as a white X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
        marker='x', s=169, linewidths=3,
        color='w', zorder=10)
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())
plt.show()

And there is a second answer, the analytical one:

The decision boundary is the group of points at which we can't decide the label. For the Kmeans it is the equidistant points from two centroides. A small calculation shows that this is a line. To find it's equation, you only need the coordinates of the two centroides.

After that you find the middle of the segment between those two, find an orthogonal vector to this segment going throw the middle and you have your equation. You just need to plot it and voila! Coding this don't take long, i'll try to update the answer if you want.


Variation in output of Logistic Regression when using SMOTE

I do not have enough reputation to make a comment.

For each run, you use different data for training and testing. To obtain stable results you should also fix the partition of the data into train/test set by providing random_state for the train_test_split:

X5_train, X5_test, Y5_train, Y5_test = train_test_split(X5,Y5, test_size=0.20, random_state=4)

A completely different question, however, is why with a small/large? change in training data you have drastically different classification performance.


Does adding new features that contain information derivable from current features help performance?

It depends what model will you train and depend on the new feature. However, your answer is that yes. it is possible to help the performance of that model. For example, suppose the new feature is a non-linear combination of other features (multiple of the other feature) and your model is a linear classifier. When you add the new feature might help to classify better the data depends on the context.


Extract most informative parts of text from documents

What you're describing is often achieved using a simple combination of TF-IDF and extractive summarization.

In a nutshell, TF-IDF tells you the relative importance of each word in each document, in comparison to the rest of your corpus. At this point, you have a score for each word in each document approximating its "importance." Then you can use these individual word scores to compute a composite score for each sentence by summing the scores of each word in each sentence. Finally, simply take the top-N scoring sentences from each document as its summary.

Earlier this year, I put together an iPython Notebook that culminates with an implementation of this in Python using NLTK and Scikit-learn: A Smattering of NLP in Python.


Is this attribute numeric or categorical (ordinal)? Help!

They are not categorical as they have a meaningful ordering that you likely want to use. The first is usable as is as it is roughly fraction religious times 10. Yes it's ordinal but happens to be just a rescaled continuous feature.
The second is ordinal and so you don't quite want to use it as a continuous feature as that doesn't capture the difference in scale that they indicate. It might work OK, but you could also translate each number to the midpoint of corresponding range.