loss/val_loss decrease but acc/val_acc are consistent

In your code:

model.compile(optimizer = 'adam', loss = 'mean_absolute_error', metrics=['accuracy'])

You are using accuracy as a metric, metrics=['accuracy']. This does not work when you are doing regression (which you are doing). Regression is when you have continuous (floats) labels.

So instead your code should look like:

model.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics=['mean_absolute_error'])

Then when you train you should look for both loss and your metric to be decreasing during training. If they are, then your network is behaving normally.


Understanding Logistic Regression Cost function

The way my intuition works for logistic regression is simple, say you are trying to classify if you have a picture with a dog on it, if that is the case, you output 1 (true), if not, you output 0 (false)

The cost function essentially represents the diff between what your model output (imagine, you have a dog in a picture, and your model outputs 0.75 instead of 1, your cost is 0.25).

Now, this is not a very mathematical way to put it, but at least it has helped me a lot to understand logistic regression.


Combining binary features with textual ones

hstak is a good solution. Using this I combined these features.


No, according to your description the syslog is not a time series data.

Time series data means to have a numerical value at each time stamps. For instance, if your log records the CPU usage (a number between 0 to 100%) then it is a time series data.

If you log is a sequence of text message, like the /var/log/syslog in Linux, then it is not a time series data.


How to use testing data set to measure recommender system algorithm

I guess the right answer might be just predict all 10 items User A has ratings for. And compute MAE using all of them.

Update: Just learnt that approach above is called "Judging Model Quality by Residuals", there are others like Cross Validation


Calculating image text orientation in scanned documents

You might be thinking of this article, which mentions some other approaches too!


Applying convolutional neural network over text documents using 1-D tf-idf feature vectors

In TFIDF representation "neighborhood" is not interpretable. The 1st number in a document vector has just as much relation to the 2nd and to the 100th.
While in case of a standard time series or sequence analysis "neighborhood" is interpretable.
On the other hand, if you could re-order the TFIDF representation in a way, that makes sense from the analysis perspective (eg: ordering based on the "sentiment" of the word), maybe 1D CNN would show some good results.


On L2 normalized data it is an easy and good exercise to prove that they are equivalent.

So you should try to solve the math yourself.

Hint: use squared Euclidean.

Note that it is common with tfidf to not have normalized data because of various technical reasons, e.g., when using inverted indexes in text search. Furthermore, cosine is faster on very sparse data.


Supervised learning vs reinforcement learning for a simple self driving rc car

I'd suggest you to try a hybrid approach:

First, train your car in supervised fashion by demonstration. Just
control it and use your commands as labels. This will let you get all
the pros of SL.
Then, fine tune your neural net using reinforcement learning. You
don't need extra sensors for that: the rewards may be obtained from
distance sensors (larger distances = better) and from the speed
itself. This will give you the pros of RL and train your NN to the correct goal of driving fast while avoiding obstacles instead of the goal of imitating you.
Combining both approaches will get you the pros of both SL and RL
while avoiding their cons. RL won't start from random behavior, just
small gradual deviations from what you tought the NN. A similar approach was applied successfuly by Google DeepMind with AlphaGo.
You can always include explicit rules on top of this. Implement them with high priority and call the NN only when there is no explicit rule for current situation. This is reminiscent of the Subsumption Architecture.


machine learning mode are suitable for using word features as x, and y as click rate of text

You can build a dictionary of character sequences (words) and for each instance of text you will count the occurrence of these words. You can either use groupings of characters n-grams or words them selves using bag-of-words.

n-grams

n-grams is a feature extraction technique for language based data. It segments the Strings such that roots of words can be found, ignoring verb endings, pluralities etc...

The segmentation works as follows:

The String: Hello World

2-gram: "He", "el", "ll", "lo", "o ", " W", "Wo", "or", "rl", "ld"
3-gram: "Hel", "ell", "llo", "lo ", "o W", " Wo", "Wor", "orl", "rld"
4-gram: "Hell", "ello", "llo ", "lo W", "o Wo", " Wor", "Worl", "orld"

Thus in your example, if we use 4-grams, truncations of the word Hello would appear to be the same. And this similarity would be captured by your features.

Bag-of-words

Bag-of-Words builds a dictionary of the words it has seen during the training phase. Then using the word the frequency of each word in the example a vector is created. This can then be used with any standard machine learning technique.


Is LSTM or pretrained BERTForMasked LM usable for predicting changed word in a sentence using a small dataset? (2000 samples)

Both bidirectional LSTM (like ELMo) and BERT seem appropriate for this kind of task. Whether one or the other performs best can only be known by testing.
If you use BERT, ensure to apply the typical measures to avoid overfitting. If you apply LSTMs, you will probably need regularization measures.


Why is $Y=\beta_0 x^{\beta_1} e$ a linear model?

The term "linearity" is context-dependent, so a linear regression model is not necessarily the same as a linear function.
A linear function is classified as such via the superposition principle, requiring both additivity and homogenity. We can generalize linear maps with to multivariate functions simply:

Additivity: $f(\vec{x}_1+\vec{x}_2)=f(\vec{x}_1)+f(\vec{x}_2)\enspace \forall\ \vec{x}_1,\vec{x}_2\in \Bbb{R}^n$
Homogeneity: $f(\alpha \vec{x})=\alpha f(\vec{x})\enspace \forall\ \vec{x}\in \Bbb{R}^n$ and $\alpha\in \Bbb{R}$

So the equation $f(\beta_0,\beta_1)=\beta_0 x^{\beta_1}e$ is a nonlinear function because it is not additive nor homogeneous.
However, w.r.t. linear regression, linear models are of the form $Y=\beta_0+\beta_1f_1(x_1)+\beta_2f_2(x_2)+\cdots+\beta_nf_n(x_n)+c$, regardless of whether any $f_i$ is a nonlinear map.
So, after a nonlinear transformation such as natural logarithm, you can consider the resulting regression model as the form $Y'=\beta_0'+\beta_1ln(x)+1$ where the primes denote the natural log transformed variables. This demonstrates linearity between $Y'$ and the parameters $\beta_0'$ and $\beta_1$.


Check If Answer for a Question is Correct by Similarity

This problem can be solved better if you also include the background text from where the questions have been picked up. Then, firstly train word embeddings on the background text. Further, generate sentence compositionality of the questions as well as their answers using a Recursive Neural Network or some variant. After all these steps you can compute the similarity between compositions of the new answer and given answers to know if it is correct or not.


Why does the same algorithm give very different metrics on similar datasets?

Simple answer: the datasets are different. Apparently, your approach worked well for A but not for B. From this you might infer, that B and A are not so similar after all. I suspect that B has a more complicated relation between the X and targets y. So maybe try boosting on B?!


Please let me know if I am on the right track to being an NLP Expert

Here are a couple of general NLP resources:

How To Do Things With Words by Peter Norvig
NLP sections of Intro to Artificial Intelligence from Udacity
Neural Network Methods in Natural Language Processing by Yoav Goldberg & Graeme Hirst


Filtering outliers in Apache Spark based on calculations of previous values

This is actually a general problem with time-series data: you have some logic to implement based on one or more values in the series.  You always have two choices:

Feed the time series through some module that calculates as each data point arrives
Use the "spreadsheet method" to calculate a series of columns eventually arriving at the goal

The advantage of the first approach is you can use the same module to process your real-time data.  The advantage of the second approach is that it's very fast and usually easier to implement.

Since you're already in a Spark Dataset, here's the strategy:

Calculate a speed column: $p_t - p_{t-1}$ where $p$ is the position
Calculate a "jump" column: 1 if the speed is over a certain threshold, -1 if under, 0 otherwise
Calculate a "jumpsum" column: the cumulative sum of the jump column
Bad data will have a jumpsum of 1; filter them out

Here's how you do it:

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions._
import org.apache.spark.sql.SparkSession

val ss: SparkSession = SparkSession.builder.getOrCreate()

// note the file must be on each executor in the same directory
val ds = ss.read
   .option("header", "true")
   .option("inferSchema", "true")
   .csv("file:///home/peter/data.csv")

val w = Window.partitionBy().orderBy("datetime")
val threshold = 10
def jump(v: Double): Int = if (v > threshold) 1 else if (v < -threshold) -1 else 0
val sqlJump = udf(jump _)

val cleanDS = ds
    .withColumn("speed", $"position" - lag($"position", 1).over(w.rowsBetween(-1, -1)))
    .withColumn("jump", sqlJump($"speed"))
    .withColumn("jumpsum", sum($"jump").over(w.rowsBetween(Long.MinValue, 0)))

Here's what the output Dataset looks like (I didn't remove the bad rows so you can see the calculation):

+--------+--------+-----+----+-------+
|datetime|position|speed|jump|jumpsum|
+--------+--------+-----+----+-------+
|       1|       1| null|null|   null|
|       2|       1|    0|   0|      0|
|       3|       1|    0|   0|      0|
|       4|       1|    0|   0|      0|
|       5|       1|    0|   0|      0|
|       6|       2|    1|   0|      0|
|       7|       1|   -1|   0|      0|
|       8|       1|    0|   0|      0|
|       9|      46|   45|   1|      1|
|      10|      45|   -1|   0|      1|
|      11|      48|    3|   0|      1|
|      12|      45|   -3|   0|      1|
|      13|       1|  -44|  -1|      0|
|      14|       2|    1|   0|      0|
|      15|       1|   -1|   0|      0|
+--------+--------+-----+----+-------+

The "data.csv" is just the first two columns of that Dataset:

datetime,position
1,1
2,1
3,1
4,1

...etc.

All that's left to do is filter out jumpsum === 1.


You should ask yourself - are you teaching an algorithm to play chords or to play music? In addition, what are you trying to predict here?

It seems to me that you need to create input data that is a series of chords and your label is the next chord in the tune. So you should design a neural network that takes in a series of chords and can tell you the next chord in the sequence, add that back to the input sequence and pick the next chord, add that back to the input sequence, etc, etc. Next thing you know, you have a neural network that can play music.


I think you might have confused the concept of training set, validation set, and actual input. The training set helps you build a model; the validation test let you validate that performance of your model. (sometimes people also use real life testing test to ensure the model is not overfitting) Once you have a model created, your model will be like a black box that takes your input (i.e., could be a single sample) and give you a classification result. So it really doesn't matter if your only 1% of your input data is a car sound; if the model is built properly it will be able to give you a good prediction.


Firstly the function returns the variables u and indices:

u contains the unique elements sorted. In other words no element is repeated (the number 2 does not appear twice) and the elements will be listed from the smallest to the largest value
indices is the same size as a and basically it contains the index in u you should used to recover a. So when they give you [0,1,3,2,1], in this case the number 3 refers to the 3rd index of u = [1, 2, 3, 5] which in this case is 5

You can see how u and indices can be used to recover a by running the following code:
for element in indices:
      print(u[element])

This also explains your last question: u[indices] it basically gives you back a. Note that in your question u[indices] should return array([1, 2, 5, 3, 2]). I assume you made a typo


Can someone explain the following error in my python code?

Seems that remove border is not defined. You have to define the function before used.

I do not know where the string error comes, is not clear to me. If you post the full traceback it will be clearer.

Finally your label is not show because you have to call the method plt.legend()


Can we implement random forest using fitctree in matlab?

I would highly recommend doing some research into the architecture of random forests.  There are many sites that provide in depth tutorials on RFs (Implementation in Python).

Quick  explanation: take your dataset, bootstrap the samples and apply a decision tree. Within your trees, you  want to randomly sample the features at each split. You should not have to build your own RF using fitctree however.  You don't want to control each individual tree in the forest. This introduces bias, and the point of the RF is that by bagging many trees, you remove the risk of overfitting.

Define your hyperparameters and let the algorithm do it's thing.  Carefully cross-validate to ensure you are not under-fitting.


Multivariate Regression Error “AttributeError: 'numpy.ndarray' object has no attribute 'columns'”

Using .values on a pandas dataframe gives you a numpy array. This will not contain column names and such. You do this when setting X like this:

X = dataset[['Read?', 'x1', .. ,'x47']].values

But then you try to get the column names from X (which it does not have) by writing X.columns here:

coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])

So store your column names in a variable or input them again, like this:

coeff_df = pd.DataFrame(regressor.coef_, ['Read?', 'x1', .. ,'x47'], columns=['Coefficient'])


Either format will work with R and Python, though you'll need a library for MS Access, which is the least common option. If any of these formats is the native format of the data, I'd go for that, which will avoid weird transformation artefacts. If neither of the formats is, I'd choose CSV, which would allow me to use a text editor if needed. However, if one of the columns contains user generated texts, it might be worth the trouble to choose Excel, that would limit the chance that the export would get botched.

It is way easier to get your hands on a CD-reader. Your data doesn't sound that impressively large that it warrants using tape.


Word2Vec uses the bag-of-word model as input which means that you can use whatever alphabet. Building a bag of word just consists of attributing one feature per word in the entire dictionary of your corpus. You can do this with a cyrillic alphabet as with other alphabet.


How to select the best parameters for GridSearchCV?

That is indeed a drawback with grid search strategy, since you must know in advance each one of the possible combinations to try out, and that might be not optimal neither to get the best evaluation metric value nor in computation performance.
You have other interesting strategies, not exhaustive hyperparameter search, for instance random search or based on bayesian tuning, for a more efficient search and being a "more clever" search strategy in the second option.
You can have a look at HyperOpt library with several optimization algorightms (see also this link for a practical use case), and more recently Keras released a nice keras tuner (which I love by the way).
You can also have a look at this answer for a worked out example on a XGB model using Hyperopt, and this one for using keras tuner.
You can also check the keras tuner wrapper for sklearn models: https://keras-team.github.io/keras-tuner/documentation/tuners/#sklearn-class


The PyTorch remembers a computation graph. It means that for each value you get after the forward pass it remembers which operation you have applied and for all basic operations, it knows how to calculate the gradients. So, that's why in PyTorch you get the optimization for free -- at least, for common neural nets. The nn.Parameter method specifically tells to track gradients and change them during optimization.
The approach works well in the particular case for the most part, but there are two not-so-common steps in bayes by backprop:

For each neuron we sample weights. Technically, we start with sampling from $N(0, 1)$ and then we apply the trainable params. The specific values we get from $N(0, 1)$ are kind of extra inputs and for some operations, they change the gradients.
Loss usually depends only on outputs and true labels, but for the KL term, you need to calculate the closed-form value based on weights themselves. You would probably need to explicitly collect the KL loss term from each layer.

Everything else should be handled fine by pytorch


Difference in model performance measures of train and test data sets

Your model suffers of a slight overfitting, however it doesn't seem too dramatic.

Performance on train set are always better than test set if random sampled (when you have statistically significant volumes)

Maybe you can reduce the gap of performance by controlling the CP parameter, try setting a higher cp when you prune the tree (like 0.01) or by using the parameter maxdepth that prune according to the length of the tree.


Classify sentences containing typos into groups

If I understand correctly, you're looking for string similarity. There are several techniques available, the most simple is "edit distance" (aka levenshtein distance), which is the count of the minimum insertion/deletion/substitution/transposition operations needed to get from one string to the other.

For your particular task, I suspect "jaro-winkler similarity" would be better. JW is similar to ED, but was specifically designed for "entity resolution" (i.e. "record linkage"), which it looks like is what you're trying to accomplish. You can see a short demonstration of how this would work here


In Tensorflow 1.*, myvar will not be collected even if you lost track of the variable pointing to it, till the Python process terminates.

In Tensorfow 2.* it will be collected by the garbage collector.

For details, see the blog by Tensorflow, under the "No more globals" section.


How to perform convolution with kernel bigger than image?

Is your question how can we perform convolution operation with kernel bigger than input image?

If yes then answer is padding. TL;DR you increase the size of the original image with boundary pixels so that kernel can "fit"


What is purpose of partial derivatives in loss calculation (linear regression)?

The underlying idea behind machine learning is to come up with more or less complicated algorithms such that, given a set of input data, one is able to produce some sort of output; this output in turn depends on some parameters (upon which the model is specified). The objective is to choose the aforementioned parameters so that the algorithm output is as close as possible to the actual result. Now let $y_i$ and $f(x_i, \beta)$ be the actual and the predicted value, respectively, (in correspondence of the variable $x_i$): the previous sentence translates into trying to choose the parameters $\beta$ (whatever they mean) such that the error you commit is the smallest, namely $L(y | x, \beta)$ is minimised, where the function $L$ is any way we decide to estimate the error between predictions and actuals. In the literature $L$ is referred to as loss function and for most practical purposes, especially for polynomial models like linear regression, it reduces to the sum of squares
$$
L(y|x) = \sum_{i=1}^N (y_i -f(x_i,\beta))^2.
$$
For each choice of parameters $\beta$ and function $f(x_i)$ the above takes different values; we are looking for, once we decide to fix the form of $f$, the set of $\beta$ such that the above is the smallest. Assuming the loss function is differentiable, local minima must satisfy the condition such that the collection of partial derivatives with respect to the variable in question (in this case $\beta$) must vanish; as such, at the end of the day one comes down to essentially taking derivatives and equating them to zero.

In case of linear regression one assumes $f(x_i, \beta) = \sum_{j=1}^M x_i^j \beta_j$: plugging this expression into the loss function and taking derivatives gives back the familiar expressions for the coefficients that one learns in school. Likewise for more complicated models: the form of the function $f$ might be more complicated, there might be analytical problems related to the computational minimisation of the loss, there might be a bunch of more parameters connected to each other in a somewhat complex way (for instance for neural networks), nevertheless the underlying argument still holds.


differences between feature weighting and feature selection

Feature selection is choosing among your features what are the most important ones.

Feature weighting is from the features that you have selected ranking the ones that contribute most to the model.

Just to keep a basic and simple response.


Since what you are describing sounds like some sort of auto-encoder, you can use this blog on how to use convolution layers in Keras.

Most of the logic is straightforward: If you wish to reconstruct your input, you need to your output to have the same spatial dimensions and the same number of channels as your input. As in most neural network architectures, you will gradually decrease the spatial dimensions of your input and increase the number of filters, and if you wish to reconstruct your input you will use the same concept of layers, but in reverse order, using deconvolution layers.

As to whether you need to use deconvolution or what is the optimal number of channels, layers and symmetry is entirely up to what you are trying to accomplish and the specific details of your problem.


VC dimension of hypothesis space of finite union of intervals

VC dimension is defined for a hypothesis space $H$, e.g. a set of binary classifiers $C \rightarrow \{0, 1\}$. For example, hypothesis space
$$H=\{{\Bbb 1}_{x \le \theta}: \theta \in {\Bbb R}\}$$
has VC dimension $1$, because for any $C=\{a<b\}$,  it does not contain a classifier that gives $\{a \rightarrow 0, b\rightarrow 1\}$.

For example, a classifier from $H$ would be $f(x)={\Bbb 1}_{x \le a}$ that gives  $\{a \rightarrow 1, b \rightarrow 0\}$.

From C to H

As you have illustrated in the comments, we can build a hypothesis space $H$ from $C$ as follows:
$$H=\left\{{\Bbb 1}_{x \in C}: C = \left\{\bigcup_{i=1}^{k}(a_i, b_i): a_i, b_i \in {\Bbb R}, a_i < b_i, i=1,2,..,k\right\}\right\}$$
Meaning, each classifier in $H$ is a union of $k$ intervals that labels a point inside the union as $1$ and outside as $0$.

VC dimension of this $H$ is $2k$:

For VC $\geq 2k$: Let $A$ be an arbitrary set , and $A \rightarrow \{0, 1\}$ be an arbitrary labeling. By going from minimum to maximum member of $A$, we can cover all adjacent $1$s with one interval, and only need to use another interval when there is a $0$ barrier. Therefore, we need $k$ intervals to cover $k$ isolated regions of $1$s. Furthermore, a set with  $2k$ members has at most $k$ isolated $1$s (since to have $k+1$ isolated $1$s there should be $k$ $0$ barriers in-between), and thus, needs at most $k$ intervals.
For VC $< 2k+1$ by contradiction: for any ordered set $A_{2k+1}=\{a_1<...<a_{2k+1}\}$, there is labeling $a_k \rightarrow 1_{\text{k odd}}$, i.e.  $\{a_1 \rightarrow 1, a_2 \rightarrow 0,...,a_{2k+1} \rightarrow 1\}$ with  $k+1$ isolated $1$s which cannot be covered with $k$ intervals.


I'll go through your questions one by one:

  is feature selection more important in KNN than in other algorithms?

I don't think it is more important for kNN than for other kinds of algorithms.

  If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction worse, right?

Correct. Neural Networks are "smarter" algorithms, they have internal weights that adjust to minimize a cost function. Less important features will be attributed comparatively lower importance with respect to highly predictive variables. This doesn't happen in kNN, in which prediction is based exclusively on distance between datapoints - and no information about relative importance of variables can be deduced from it.

  In a less extreme example, what if a feature is weakly predictive? Rather than normalize all my features so they have an equal weight, wouldn't I want to make the highly predictive features have more weight than less predictive ones?

sklearn allows to manipulate kNN weights. But this weights distribution is not endogenous to the model (such as for Neural Networks, that learn that autonomously) but exogenous, i.e. you have to specify them, or find some methodology to attribute these weights a priori, before running your kNN algorithm. If you can do that, and you have good methodological reasons for it, then changing variables' weights can improve your model, but I'd be careful about it (you can easily overfit).


Using Low Frequency Labels with High Frequency Features

One way is to transform the daily data into dynamic quarterly data, eg by averaging over a moving window (that represents a quarter) and similarly for other statistics.
These windows can be overlapping, eg by a factor $o$%, no need to be non-overlapping.
This way the data are transformed to quarter period statistics while not actually reducing amount of data (but simply transformed).
Then you feed your regression model these quarterly data instead of the daily data in order to get quarterly results.
It is clear that to get results over quarter periods one needs inputs that represent quarter periods in the first place. So this is a natural way to achieve it.


tensorflow: is there a way to specify XLA_GPU with tensorflow?

try 'device:XLA_GPU:0'

with tf.device('device:XLA_GPU:0'):
  a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')


MinMaxScaler returned values greater than one

I am not sure why your MinMaxScaler didn't work, but here is a function that should scale your data into the desired range:

def rescale(data, new_min=0, new_max=1):
    """Rescale the data to be within the range [new_min, new_max]"""
    return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min

Looking at the documentation of the MinMaxScaler, it seems my function above it the same as their method.

You could break your code down a little to explicitly comppute each step on its own line. This might help find the origins of your problem. I tried it out and got the expected results:

In [1]: import numpy as np

In [2]: from sklearn.preprocessing import MinMaxScaler

In [3]: x = np.random.randint(0, 10, (10, 10)).astype(np.float)

In [4]: x                                    # generate random data in range [0, 9]
Out[4]:
array([[ 1.,  4.,  5.,  4.,  6.,  1.,  8.,  1.,  8.,  9.],
       [ 3.,  1.,  4.,  4.,  6.,  2.,  5.,  1.,  0.,  8.],
       [ 2.,  0.,  6.,  1.,  5.,  2.,  5.,  8.,  8.,  4.],
       [ 8.,  9.,  2.,  8.,  5.,  6.,  0.,  5.,  0.,  5.],
       [ 1.,  3.,  2.,  2.,  3.,  2.,  4.,  1.,  7.,  5.],
       [ 7.,  0.,  8.,  8.,  3.,  6.,  6.,  6.,  4.,  3.],
       [ 4.,  3.,  4.,  4.,  7.,  6.,  4.,  5.,  6.,  7.],
       [ 9.,  0.,  8.,  9.,  7.,  1.,  2.,  2.,  4.,  6.],
       [ 7.,  4.,  2.,  8.,  6.,  5.,  2.,  9.,  9.,  9.],
       [ 7.,  6.,  9.,  2.,  9.,  0.,  1.,  5.,  7.,  3.]])

In [5]: scaler = MinMaxScaler()              # defaults to range [0, 1]

In [6]: scaler.fit(x)                        # compute the scaling factors
Out[6]: MinMaxScaler(copy=True, feature_range=(0, 1))

In [7]: scaled_data = scaler.transform(x)    # scale the data

In [8]: scaled_data.shape                    # still the same shape
Out[8]: (10, 10)

In [9]: scaled_data.min()                    # min and max are 0 and 1 as expected
Out[9]: 0.0

In [10]: scaled_data.max()
Out[10]: 1.0


How to use decode_predictions() for non-Imagenet models..?

In Deep learning when you are performing prediction you will get prediction in your case it is an array with 9 probabilities in them. So first perform following operation on them.

import numpy as np

prediction = model.prediction(test_data)
# prediction will contain [[0.6, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]]

prediction = np.argmax(prediction[0])
# Now predition hold the index of the (0.6) i.e max probability value

Now after that you should have a dictionary which contains key as 0, 1, 2, .. 8 and values as classname1, classname2, ...classname9

This is the way you will get the class name as output


Why is "next state" kept in RL experience replay?

True, but if you take a look at the equations of Q learning or Advantage functions in Policy Gradients you will see that the expected value of the next state is being used. For this, you need to know in which state you have landed from state $s_t$. For example, you can approximate your future reward from state $s_{t+1}$ by using $Q(s_{t+1})$. To do this you need to give to your network as input the $s_{t+1}$.


Linear Regression with Category variables

We just encode the categorical variable to some sort of numerical representation (like one-hot encoding)

The choice of representation matters, because it has to preserve the properties of a categorical variable: one-hot-encoding is a standard option, but directly encoding categorical values as integers is a mistake because it introduces order where there is none.

the use-cases of such a method: under what kind of circumstances would using categorical data to predict a numeric value be useful?

There are many applications for which regression from categorical data is useful. For example text data is often represented as "bag of words" (BoW), where each word is a categorical variable. Some tasks involve predicting a numerical target from some text input, for example:

Predicting the grade of an essay
Predicting a score representing the likelihood of plagiarism (e.g.Turnitin)
Predicting sentiment of a text on a scale from 1 to 5

There are many other examples, and not only from text data.

And what type of data format should I have my data in before doing the encoding? (does having two columns with one numeric results and the other the corresponding category work?)

The data can have as many independent variables (features) as needed. Since the categorical variables are typically one-hot-encoded, what matters is the number of features after encoding. It's often necessary to preprocess the data in order to avoid overfitting (typically rare values should be discarded).

I would also like to know the different ways we can visualize and analyse the results of our model (and its predictions), especially if we have a sizeable amount of categorical variables.

As far as I know there's no specific method for analyzing the result of the model, it's the same principle as with numerical features. If one wants to analyze the impact of an individual feature on the performance of the model, a simple technique is to train the model with/without this feature and compare the results.


What is the best algorithm/solution for predicting the following?

Very interesting question indeed.

I will dare to give you an answer, but I could be changing it if I find a better reasoning about the problem after watching the data.

Your problem is perhaps a problem of multi(many)-label classification, where your label is the country and your data is what you have as "patience", "risktaking", etc.

You could work with random forest or decision trees but on the dataset containing all the individuals (not the summarised by country).

Why on this dataset and not on the summarised? Because the summarised is summarised (!), and only has the data for the mean of "risktaking" (for example) and does not take into account the internal variance of risktakingness inside the countries.

Is important to take this into account, because is possible that risktaking has a variance so large inside a country that is makes the variable meaningless to select a nationality for a person and this is not seen in the summarised database.

If is mandatory that you run the model in the summarised dataset first, then the solution is this:

You should run a tree where your node_size parameter is 1, because your summarised dataset has "one individual (country) per label", this is a extreme case where you need a node leaf for every country.


There are a number of different ways to impute data when you have missing values. Far to many people just delete those records, which often contain useful training data.  Further, some people advocate for simple methods like using the mean of the feature when it is missing.  I don't like this method as it seems too simplistic.

Here is a good presentation detailing a couple of methods and particularly drawing attention to imputation through maximum likelihood.

However, my favorite method is collaborative filtering, which is detailed in this paper. The nice thing about collaborative filtering is that it is easy to write from scratch and is also very popular, so is usually contained in most machine learning libraries.

Also think about taking a look at the imputation techniques that might be available in the specific machine learning library that you may be using.

Finally, a word of caution... imputing data adds some linear dependence, so you should expect that you might have to increase your regularization slightly in order to avoid high variance scenarios. I would experiment with (cross validate) removing features, removing records, then imputing data using collaborative filtering and see what produces the best results.

Hope this helps!


Tweet Classification into topics- What to do with data

As far as I'm aware there is no correct/standard way to apply topic modelling, most decisions depend on the specifics of the case. So below I just give my opinion about these points:

I have removed, before cleaning the data (removing mentions, stopwords, weird characters, numbers etc), all duplicate instances (having all three columns in common), in order to avoid them influencing the results of topic modelling. Is this right?
Should I, for the same reason mentioned before, remove also all retweets?

In general there is no strict need to deduplicate the data, doing it or not would depend on the goal. Duplicate documents would affect the proportion of the words which appear in these documents, and in turn the probability of the topic these documents are assigned to. If you want the model to integrate the notion of popularity/prominence of tweets/words/topics, it would probably make sense not to deduplicate and keep retweets. However if there is large amount of duplicates/retweets the imbalance might cause less frequent tweets/words to be less visible, possibly causing less diverse topics (the smallest topics might get merged together for instance).

Until now, I thought about classifing using the "per-document-per-topic" probability. If I get rid of so many instances, do I have to classify them based on the "per-word-per-topic" probability?

I'm not sure what is called the "per-document-per-topic" probability in this package. The typical way to use LDA in order to cluster the documents is to use the posterior probability of topic given document (this might be the same thing, I'm not sure): for any document $d$, the model can provide the conditional probability of every topic $t$ given $d$. The sum of this value across topics sums to 1 (it's a distribution over topics for $d$), and for classification purposes one can just select the topic which has the highest probability given $d$.

Do I have to divide the dataset into testing and training? I thought that is a thing only in supervised training, since I cannot really use the testing dataset to measure quality of classification.

You're right, you don't need to split into training and test set since this is unsupervised learning.

Antoher goal would be to classify twitterers based the topic they most are passionate about. Do you have any idea about how to implement this?

The model gives you the posterior probability distribution over topics for every tweet. From these values I think you can obtain a similar distribution over topics for every tweeter, simply by marginalizing over the tweets by this author $a$: if I'm not mistaken, this probability $p(t|a)$ can be obtained simply by calculating the mean of $p(t|d)$ across all the documents/tweets $d$ by author $a$.


You could convert your decision tree model to the PMML format. In Java you could use JPMML to parse/read the model and predict.


To ‘correctly’ predict the weather would require a clairvoyance that our best meteorologists have yet to divine.  In other words, they’re often quite wrong.

Weather prediction is very much a data limited problem. If you had access to each and every variable that contributes, you’d do a great job. The problem is one only has access to a subset, and the mechanics are still not fully understood. I’m not a meteorologist but I also suspect the sensors we have are sparse. I’d want temp, humidity, wind speed, and barometric pressure at 100 meter intervals both parallel and perpendicular to the earths surface, plus historical data, rate of change, ocean temp and currents, and the airspeed velocity of every unladen swallow in the area.


Micro Average vs Macro Average for Class Imbalance

The question is actually about understanding what it means to "take imbalance into account":

Micro-average "takes imbalance into account" in the sense that the resulting performance is based on the proportion of every class, i.e. the performance of a large class has more impact on the result than of a small class.
Macro-average "doesn't take imbalance into account" in the sense that the resulting performance is a simple average over the classes, so every class is given equal weight independently from their proportion.

Is it actually a good idea to "take imbalance into account"? It depends:

With micro-average, a classifier is encouraged to focus on the largest classes, possibly at the expense of the smallest ones. This can be considered a positive because it means that more instances will be predicted correctly.
With macro-average, a classifier is encouraged to try to recognize every class correctly. Since it is usually harder for the classifier to identify the small classes, this often makes it sacrifice some performance on the large classes. This can be considered a positive in the sense that it forces the classifier to properly distinguish the classes instead of lazily relying on the distribution of classes.

One could say that it's a kind of quantity vs. quality dilemma: micro-average gives more correct predictions, macro-average gives attention to actually distinguishing the classes.
Very often one uses macro with strongly imbalanced data, because otherwise (with micro) it's too easy for the classifier to obtain a good performance by relying only on the majority class. Your data is not strongly imbalanced so it's unlikely this would happen, but I think I would still opt for macro here.


How can I replace NaN values with other values in the CustomerID column if they have a same InvoiceNo?

Assuming that each invoice id has only 1 unique customer id something like this should work:
df["CustomerID"] = df.groupby("InvoiceNo")["Customer ID"].apply(lambda x: x.ffill().bfill())

The code above simply makes groups based on the customer id and forward an backward fills all NA values.


Twitter Data-Analyse: What can I do with the data?

Community analysis implies graph analysis.
here is a short list of things you can work on:

People often reshares tweets among a certain social group. Minimum-cut method, Girvan–Newman and Modularity maximization are someof the starting algorithms to extract these type of substructures.
You can try and find different hierarchies among the groups sharing a particular topics
You can try and analyse the lifetime of tweets for particular topics (survival analysis)

Analysing tweets is closer to graph analytics rather than NLP.
Here is a great overview on community analysis. For coding and algorithms, please check graphX Spark library. If your data is not too large, networkX is easier.
For survival analysis, lifeline is one of the easier options.


Function $L_2(x):=\left \|x \right \|_2$ is a norm, it is not a loss by itself. It is called a "loss" when it is used in a loss function to measure a distance between two vectors, $\left \| y_1 - y_2 \right \|^2_2$, or to measure the size of a vector, $\left \| \theta \right \|^2_2$. This goes with a loss minimization that tries to bring these quantities to the "least" possible value.

These are some illustrations:

$L_p$ norm: $L_p(x) := \left \|x \right \|_p = (\sum_{i=1}^{D} |x_i|^p)^{1/p}$,
where $D$ is the dimension of vector $x$,
Squared error: $\mbox{SE}(A, \theta) =\sum_{n=1}^{N} \left \| y_n - f_{\theta}(x_n) \right \|^2_2$,
where $A=\{(x_n, y_n)_{n=1}^{N}\}$ is a set of data points, and $f_{\theta}(x_n)$ is model's estimation of $y_n$,
Mean squared error: $\mbox{MSE}(A, \theta) =\mbox{SE}(A, \theta)/N$,
Least squares optimization: $\theta^*=\mbox{argmin}_{\theta} \mbox{MSE}(A, \theta)$$=\mbox{argmin}_{\theta} \mbox{SE}(A, \theta)$,
Ridge loss: $\mbox{R}(A, \theta, \lambda) = \mbox{MSE}(A, \theta) + \lambda\left \| \theta \right \|^2_2$
Ridge optimization (regression): $\theta^*=\mbox{argmin}_{\theta} \mbox{R}(A, \theta, \lambda)$.

In all of the above examples, $L_2$ norm can be replaced with $L_1$ norm or $L_\infty$ norm, etc.. However the names "squared error", "least squares",
 and "Ridge" are reserved for $L_2$ norm. For example for $L_1$, "squared error" becomes "absolute error":

Absolute error: $\mbox{AE}(A, \theta) =\sum_{n=1}^{N} \left \| y_n - f_{\theta}(x_n) \right \|_1$,


how to do the imputation for categorical feature with a missing rate?

You can turn it into a one-hot encoded feature with an added class of 'Missing', depending on the cardinality (how many categories are there). If the cardinality is too high, you will need to use other techniques for high cardinality features but you can still have 'Missing' as an additional category.


WGAN is too slow what are some ways to tweak for speed ups?

If you are using WGAN with Gradient Penalty, I think the Framework you are using is the limited factor since computing all the gradients will take time.

If you are using WGAN with Gradient Penalty one way to get faster results is to omit the gradient penalty and just do weight clipping as mentioned in original WGAN Paper. But be careful in Improved WGAN (with gradient penalty) they showed, weight clipping can lead to bias in the discriminator.

But for my experience with GANs WGAN GP gave the overall best results and I will spend the time training for it.

But also your training routine could be part of the problem since lot of reference implementations try to get an rate of which the generator updated, e.g. 1 generator update for 10-100 discriminator updates (which in some scenarios totally makes sense). So there are several factors for slow training but most important is that gradient penalty is computational expensive.


How to resize MNIST images to fit AlexNet model

You can use tf.image.resize, as follows:
(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()

print(x_trian.shape) # (60000, 28, 28)

# train set / data
x_train = np.expand_dims(x_train, axis=-1)
x_train = tf.image.resize(x_train, [32,32]) # if we want to resize

print(x_train.shape) # (60000, 32, 32, 1)


What is the main concept of using lexical,linguistic, semantic or syntactic approach in NLP for cyberbullying

I suggest to check out some introductory books on NLP, e.g. Natural Language Processing with Python. This is a very accessible and practical introduction and useful even if you won't be working in Python. Another, more detailed text book is Speech and Language Processing by Jurafsky and Martin.

You need to understand the basics first, and these are independent of your particular application (cyber bullying). And I think this is too much knowledge to reasonably fit into a response here. But just to give you an idea:

Lexical: information about the words (meaning and structure)

Syntactic: the structure of the text (how words make up phrases, and how phrases make up sentences, the rules that govern that process)

Semantic: about the meaning of text (as it is constructed from the meaning of the words and their composition).

These are usually part of a pipeline, which means you need start processing from the words, then look how they form sentences, and then you can look at the meaning.

Your understanding of POS-tagging seems off the mark: it is the process of assigning words in a text their part of speech (POS), e.g. verb, noun, adjective etc. As such it is part of the syntactical processing (but requires lexical knowledge too), but is also useful for semantic analysis further down.


Is there a way to weight instances based upon the values of a feature?

Try reading up on Loss/ Cost functions. Samples that are closer to your model will be assigned a smaller cost while others a higher one.

https://en.wikipedia.org/wiki/Loss_function


It automatically determines the amount of mappers by the number of partitions your data is in. You can call getNumberPartitions on your data source (RDD/DataFrame) to see how much it is and use repartition for scaling this up or coalesce to scale this down (you can use repartition for this as well but this is slower). Repartitioning is expensive however and should be avoided when unneccesary.


sklearn.metrics.average_precision_score getting different answers for same data but different formats

By explicitly giving both classes, sklearn computes the average precision for each class.  Then we need to look at the average parameter: the default is macro:

  Calculate metrics for each label, and find their unweighted mean...

If you switch the parameter to None, you get

average_precision_score(y_true, y_scores, average=None)
# array([0.58333333, 0.33333333])

whose second entry agrees with the answer for the positive class, and whose average gives you the other output you're seeing.

As for hand calculations:
For the positive class, with a threshold in $(0.8,1]$ we get zero positive predictions, and so 0 recall and 1 precision (by convention).  With the threshold in $(0.4,0.8)$ we get one false positive, one false negative, and one true negative, so 0 recall and 0 precision.  With the threshold in $[0, 0.4)$ we get one true positive and two false positives, so 1 recall and 0.333 precision.  So the table from the linked question in this case is

    R     P
1   0.0   1.0
2   0.0   0.0
3   1.0   0.333

Finally, the average precision computation is
$$(0.0-0.0)\cdot 0.0 + (1.0-0.0)\cdot 0.333 = 0.333$$
Somewhat degenerate example, but it checks out.

For the other class:
$(0.6,1]$ gives zero "positive" predictions, so 0 and 1 again.
$(0.2,0.4)$ gives one true positive, one false positive, and one true negative.
$[0,0.2)$ gives two true positives, one false positive.  So

    R     P
1   0.0   1.0
2   0.5   0.5
3   1.0   0.666

and average precision is
$$(0.5-0.0)\cdot 0.5 + (1.0-0.5)\cdot0.666 = 0.58333$$


Features selection in imbalanced dataset

What this shows is that the protocol is not a very discriminative feature:

the probability of class 1 given http is 10/(109+10)=0.084
the probability of class 1 given https is 180/(180+1560)=0.103

If these conditional probabilities were very different this feature would be more helpful to predict the class, but they differ only slightly. Note that the feature might still be useful, but it doesn't have a huge impact on its own. In case you're interested to know if the difference is significant (i.e. not due to chance), you could do a chi-square test.

Does it make sense to say that the most websites having class 0 have an https protocol, even if I have a dataset with class imbalance?

It is factually correct, but most websites having class 1 also have https so it's not a very useful information (and on its own this information might be confusing for some readers).

For a model, I would consider resampling techniques. Should I work on this after the resampling, or it would make sense to check features importance with other tests (e.g., Pearson correlation, if it his appropriate in this case)?

Feature selection can done either before or after resampling, it doesn't matter. The two things are independent of each other because the level of correlation between a feature and the class is independent from the proportion of the class.
I don't think Pearson correlation is good for categorical variables. I think conditional entropy would be more appropriate here (not 100% sure, there might be other options).


Censored output data, which activation function for the output layer and which loss function to use?

We can't tell you what loss function to use.  That is based on business needs.  In particular, what is the cost of being wrong?  Is the cost of being wrong proportional to relative error?  absolute error?  Something else?  That will drive the choice of loss function.  You should try to choose a loss function where the value of the loss function is proportional to the cost of the error (e.g., the monetary cost to your company).

Using ReLU as the activation function in the final layer would be sensible given that your data is censored to replace all negative outputs with 0.


What model should I use for multiple time series input

You mention having 2 types of data:

Water temperature with a minute-granularity
Climate data taken 20 days prior

I'm not sure what you mean with "joining the series into one row/series", but I don't think that it make sense to combine these two types as the time-frame is just too different. In this case it would probably be better to combine different layers to deal with the discrepancy. I would use a RNN layer for the water temperature and experiment with different layers (for example regular dense layers) for the remaining input, treating it as fixed inputs (like the location variable) instead of dynamic time-step input like the water temperature.

Since the climate data is 20 days prior, I don't think that it'll really influence the water temperature on a minute basis; the climate data is more like a foundation, similar to  the location variable.


Linear regression: variables with high P-value

In general, there is no clear and easy way for deciding which features to include in a model.

That being said, there are different strategies you can use to process features in an efficient way:

Domain knowledge

One of the most important aspects when determining important features is the knowledge of the specific domain related to your dataset. This might mean reading past research papers that have explored similar topics or asking key stakeholders to determine what they believe the most important factors are for predicting the target variable.

So, these methods depend on your knowledge from the domain.

Wrapper methods

These methods determine the optimal subset of features using different combinations of features to train models and then calculating performance. Every subset is used to train models and then evaluated on a test set. Sp, these methods could end up being very computationally intensive, though they are highly effective in determining the optimal subset. It is not suggested to use these methods with large feature sets because they are computationally expensive.

For this case you can use Recursive Feature elimination (RFE)

Filter methods

In these methods, feature selection methods carried out as a pre-processing step before even running a model.

They work by observing characteristics of how features are related to one another. Different metrics can be used to determine which features will get eliminated and which will remain. They also return a feature ranking that tells you how independent variables are ordered in relation to one another. They will remove the variables that are considered redundant.

One example of this method is "VarianceThreshold" from sklearn

Embedded methods

These methods are included within the actual formulation of the machine learning algorithm. The most common type of embedded method is regularization such as lasso.


Forecasting via LSTM or XGBoost... is it really a forecast or

However, when it came to forecasting, the author held out portion of the data and then used that data to predict future values... In my mind, it is not really forecasting (something that is done via ARIMA, VAR, etc - where you specify number of time periods and then don't specify anything else and the model gives you some forecasting for the future)

Generally speaking, I think the term "forecasting" can apply to any problem where the goal is to predict future events/values.

You are describing a subclass of forecasting problems called "univariate time-series forecasting".  In the typical case of univariate time-series forecasting, a model is built using only historical observations of the target variable.  ARIMA and VAR models are commonly used for these problems. (Although there are almost always multivariate extensions of these models.)

There's also "multivariate time-series forecasting", where the time-series includes more than one time-dependent variable, and each variable might depend on both its past values and the past/present values of other variables.  Weather forecasting is a good example of this type of problem.

  [To what extent are LSTM or XGBoost ] used in forecasting?

As you have correctly pointed out, models like XGBoost are only useful in cases where you have additional inputs other than historical observations of the target.  (LSTMs can be actually used with or without additional inputs.)

It's hard to say for sure how common XGBoost or any other model is in industry, but there is a
pretty
huge
body
of
research
on forecasting with exogenous inputs.

I once worked on a project where the goal was forecasting solar irradiance at a solar farm.  Naturally, there's a significant amount of seasonal variation in solar irradiance.  But local weather conditions like cloud cover, fog, and temperature also have a big impact on irradiance, so a univariate model is not sufficient.

We trained XGBoost, Random Forest, SVM, and deep learning models to forecast future irradiance.  The input to these models were forecasted weather varaibles from one of NOAA's numerical weather models, and I think this is a very common approach.

  [If] I need to forecast something in the future and have no clue about the other input values, just stick to traditional VAR model?

Yep, if you don't have any information about other variables, then you're restricted to univariate time-series methods.  But even if you don't know the other inputs with 100% confidence, you may be able to find reasonable predictions that could be leveraged to build a more sophisticated model :)


I do not understand the prediction result of the CNN model

Without knowing more about the training data it's impossible to explain the prediction results properly. One way to interpret the results is that the model is producing a probability distribution, over 3 classes, for each image you feed. This would be the case if your training data consists of one-hot vectors where exactly one element is 1 and the other two are 0.

The output of the model is your final layer after the final activation function is applied, so the prediction has the same shape of your dense_2 final layer. The None in the first dimension indicates that the size of that dimension at runtime is dynamic, and in this case the first dimension represents batch elements. Hence the output is an array of shape [batch_size, 3] and you have one 3-element vector for each of the 4 example images fed into your network.

In your comment you indicated that your final activation is the softmax function, which means that each 3-element vector can represent a probability distribution over 3 possible outcomes (notice that each vector sums to one: by definition softmax normalizes its output).

If you're trying to predict classes for each image, which appears to be the case, the predicted class label integer would be the index of the 3-element vector that has the highest probability. Typically these integers represent something semantically relevant to your application.

With a softmax output you still have to apply a decision rule on the output to actually retrieve the predicted class. For example, if your possible classes are ["dog", "cat", "mouse"] and your example image outputs a vector [.06, .9, .04], then you could choose the "predicted class" to be "cat". Or you could incorporate the predicted probability distribution into a more elaborate probabilistic model.


Machine Learning: Why do the error in cost function need to be squared?

The simple answer is that it's conviences rather than necessities.  You're more than welcome to take the absolute value and in many cases it may be better to do so.  Squaring a function makes the math a bit happier and easier and for proofs has desirable properties.  While you can basically do the same proofs with an absolute value function, you may have to handle certain edge cases and just amounts to more writing.


Although building neural network models is admittedly still an art rather than a science, there are some (unwritten) rules, at least for initial approaches to a problem, such as yours here (I guess).

One of them is that dense layers with 50,000 nodes are too large, and AFAIK I have never seen such large layers in practice; multiply this x3 (layers), and no wonder your code takes forever.

I would certainly suggest to experiment with a dense layer size between 100 - 1000, and even start with less than 3 dense layers. Reducing your 1st CNN layer to 32 is also certainly an option.


I faced same issues and searched solution in many places. But apparently, it's because I put wrong label in my dataset. It should be 0,1,2,... n but I made 1,2,3,....n. It should be started from zero. After I changed the label, it works well now.


How to define tidy data if there is repeated measures?

As you correctly observed, there is no assumption that an observation must be defined by a single key (variable). In this example an observation must be defined by a pair country + year, hence the correct tidy version in table1. It's not a complex case at all, this is very common and sometimes with more than two variables.

In general "tidying" a dataset increases the number of rows and often decreases the number of columns. A way to see that your second table is not tidy is that it would require new columns for every new year added to the dataset. As you noticed another indication is simply that it requires variables values in the columns names, which is a very bad design idea in general.

This being said, tidy data isn't a magical solution to every data design problem: it tends to demultiply the number of rows to an extent which makes it impractical in some cases.


How to extract most occuring words based on month & what tool to use?

If you are new to data science and data munging, this could be kind of a tricky task, but a good one to get your feet wet.  Many programming languages have the capability to do this (R, Python, Matlab, etc.).  I use R primarily, so I'll give you a brief heuristic for how I'd approach the task in R.  Perhaps looking into these steps will get you started.

Install R
Install some packages that will help you along ('tm' for text mining,'dplyr' for cleaning/organizing your data, and perhaps also 'lubridate' for working with dates/times)
Read in your data from your source, be it a text file, spreadsheet, or some database (if a database, you'll have to conquer connecting R to said database too)
You want to do a word frequency analysis for each month. How you accomplish this will depend on how the data is organized, which I do not know, but it would involve first rounding all your dates to month (using lubridate's 'floor_date()' function is one way), then parsing the text for each month into a corpus that can be analyzed (using package tm).
Finally, for each month I would make a table counting words, sorting by frequency.  That would give you, for each month, the top 'trending' words. To discount words like 'the' and 'a', I might also use some of the tools in the 'tm' package to clean things up.
Note that in #5 I said 'words', not 'terms'.  If you want to account for terms consisting of > 1 words, you'll have to 'tokenize' them, but that's beyond the scope of this very brief intro.

As with many data science tasks, there are many ways to attack this; the above is but one of many possibilities.

Hope that helps.


corpus development for plagiarism detection

I want to have a local database of corpus of the whole internet

Are you Google? If not storage might be an issue ;)

The PAN series have run various tasks related to plagiarism detection in the past: https://pan.webis.de/tasks.html#task-originality. I think they provide annotated datasets and they used to provide a live search engine.


Train and predict on a varying number of inputs - time based events

When dealing with sequence data it is usual to have sequences with different sizes. One answer for your problem might be to using padding in your events, in this way you keep a fixed number of events for all your data.
It happens in the same way with NLP applications, let's set number of events to be five.
So you have the following two rows in your data set:
UserID | time     | eventType | eventData
1        08:00:00      A           X
1        08:01:00      A           X
1        08:02:00      B           X
1        08:03:00      C           Y
1        08:03:00      A           Z
1        08:00:00      A           X
2        07:00:00      A           Z
2        07:01:00      B           X

So, you have for userId = 1 six events and for userId = 2 two events. Using paddings = 5, you will add 'Null'/Zero data for user 2 and you will truncate one event for user 1. Usually, you keep the last ones, so your data will become:
UserID | time     | eventType | eventData
1        08:01:00      A           X
1        08:02:00      B           X
1        08:03:00      C           Y
1        08:03:00      A           Z
1        08:00:00      A           X
2        07:00:00      Null        Null
2        07:00:00      Null        Null
2        07:00:00      Null        Null
2        07:00:00      A           Z
2        07:01:00      B           X

But what about the null values? You can use GradientBoosting algorithms that deals with null values by default. Or a missing imputation stretegy such as the most common value or a new class indicating null value.
Then, to finish, you can transform your data to have each user a row in your dataset:

Notice that your 'window' size is a hyperparameter and it is a good practice to define it via cross-validation.
I hope it helps.


What is the difference between trax vs tensorflow?

Trax is a high-level interface to make it easy to apply existing models (like ResNet, LSTM, Transformer). Trax focuses almost exclusively on deep learning.
TensorFlow is a data flow computational framework that can be applied to a wide variety of problems. TensorFlow allows fine grain control over every step in the modeling process. TensorFlow is often used for deep learning, but given its flexibility it can be used for any numerical computation.


Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

There can be two distinct reasons to use instances annotated with the gold-standard class, i.e. the true answer for the target application:

In order to perform proper evaluation your test set must contain the gold-standard labels. The principle of evaluation is to measure by how much the predictions deviate from the truth, but without the truth the performance that you obtain on the test set is meaningless for the task that you are doing.
In order to train a supervised or semi-supervised model, the training set must contain the gold-standard labels.  Semi-supervised methods offer some options to adapt a training set to a different task.

You can't rely on a model if you can't evaluate it at least on a small sample, so yes you probably need to manually annotate a subset of the data. It's only after that you can start thinking about how to improve performance.


logistic like curve fitting using machine learning

So my question is, is logistic regression only used in classification tasks?

Name Logistic regression indicates that we could use this algorithm for regression problems.

  Or can it be used for curve fitting?

It could be used to curve fitting.

  If not what are the other available techniques to fit a logistic like curve to a set of data points?

Try numpy or scipy:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html

https://lmfit.github.io/lmfit-py/model.html


In Pytorch, if I have a 2D tensor, how to iterate over this tensor to get every value changed


You are using the function softmax_cross_entropy_with_logits which, according to Tensorflow's documentation, has the following specification for logits,

logits: Per-label activations, typically a linear output. These activation energies are interpreted as unnormalized log probabilities.

Hence, you should pass the activations before the non-linearity application (in your case, softmax). You can fix it by doing the following,

def neural_network(data):
    hidden_L1 = {'weights': tf.Variable(tf.random_normal([784, neurons_L1])),
                'biases': tf.Variable(tf.random_normal([neurons_L1]))}

    hidden_L2 = {'weights': tf.Variable(tf.random_normal([neurons_L1, neurons_L2])),
                'biases': tf.Variable(tf.random_normal([neurons_L2]))}

    output_L = {'weights': tf.Variable(tf.random_normal([neurons_L2, num_of_classes])),
                'biases': tf.Variable(tf.random_normal([num_of_classes]))}

    L1 = tf.add(tf.matmul(data, hidden_L1['weights']), hidden_L1['biases']) #matrix multiplication
    L1 = tf.nn.relu(L1)

    L2 = tf.add(tf.matmul(L1, hidden_L2['weights']), hidden_L2['biases']) #matrix multiplication
    L2 = tf.nn.relu(L2)

    logits = tf.add(tf.matmul(L2, output_L['weights']), output_L['biases']) #matrix multiplication
    output = tf.nn.softmax(logits)

    return output, logits

Then, outside your function, you can retrieve the logits, and pass it to your loss function, as in the example bellow,

output, logits = neural_network(x)
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits,
                                                             labels=y))

I remark that you may still be interested in the outputs tensor, for calculating your network's accuracy. If this substitution doesn't work, you should also experiment with the learning rate parameter on your AdamOptimizer (see the documentation here).


Can I use categorical and numerical data variable at the same time in randomForest in r?

In general you can always transform categorical data into numeric data. Many experienced STATA users need to do routinely. If you have gender variable coded as "male/female" you need one binary gendervvariable recorded as 0/1. If you have four levels variable such as main role in a football team "goalkeeper/defense/midfield/attack"  you could take midfield as reference and generate the goalkeeper, midfield and attack variables all coded as 0/1.
In caret, the dummyVars function allow to create easily dummy variables.
Here an example with cylinders transformed as factors and then dummyfied:

 library(caret)
 mtcars$cyl <- as.factor(mtcars$cyl)
 dmy<- dummyVars(" ~ .", data = mtcars, fullRank=T)
 dummy.mtcars <- data.frame(predict(dmy, newdata = mtcars))
 head(dummy.mtcars)


ValueError: Found input variables with inconsistent numbers of samples

This error usually indicates your train and test in the train_test_split() function call are different sizes. You may need to reshape one to get them to match. Look at the shapes of train and test to see what is the problem.


Is it normal for some classifiers to change their prediction performance from fixed amount of data each time I run them to train?

How are you dividing? You say randomly, but is this being redone each time your analysis is run? If it is, then this is perfectly normal, expected behavior. You're getting a different prediction performance because you have a different random training set each time.

You could also be using a stochastic method, that doesn't give the same result each time, say regression fit using gradient descent versus least squares. Since you didn't give any more details, I can't be more specific than that.


How to adjust the Regression with ANN for last part of function

I see you use are sequential model. Could be a task for an LSTM architecture.
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

Maybe you are overengineering the thing. Judging from the dataplot, a simple OLS regression splines model may also fit well.
https://www.analyticsvidhya.com/blog/2018/03/introduction-regression-splines-python-codes/


Improving misclassification for one class in a multi-class classification task

The main problem is that there are too many Escas in your dataset. If you look at the confusion matrix, the Esca column gets predicted (wrongly and correctly) much more that the others. This is clearly a symptom of a skewed data set.
Try augmenting your images to generate a larger 'super-dataset'. Then sample a 'sub-dataset' from that 'super-dataset', such that it has an equal distribution across all classes. Train on the 'sub-dataset'.
If you can't augment your data to have a better distribution across the 4 classes, here are some ideas:

Modify the loss function to more aggressively penalize Black Rots classified as Escas.
Split into two networks; the first differentiates between
Leaf Blight-Healthy-Black Rot/Esca, and the second differentiates between
Black Rot-Esca.


How to convert longitude and latitude in time series data from daily to weekly?

I don't know exactly whether it is useful for your case, but you can use last day of the week, so coordinates of the object at end of the week.
I copied your example data to a text file, read it with Pandas, and resampled data from daily to weekly by getting coordinates of the last day of each week.
df = pd.read_csv('untitled.txt',
                 infer_datetime_format=True,
                 parse_dates=['date'], dayfirst=True)
df.set_index('date', inplace=True)
df.resample('W').last()


validation/training accuracy and overfitting

I'm not sure what you mean by 'intrinsic'.

A higher training accuracy means you have found a model of the training data that doesn't describe the validation data equally well.  Because of this, any statistically significant difference in training-validation accuracy is due to overfitting.   Figuring out what 'statistically significant' means in this context is not totally trivial, unfortunately, but the general point still stands.


Python XGBoost predict_proba returns very high or low probabilities

If you are after well calibrated scores (that is, the scores outputted by your model can be interpreted as probabilities in some sense, at least according to what you have observed empricially in your training set) then accuracy is not the right function to optimize (i would argue that it is never a good function to optimize, but I won't get into this). Accuracy can be optimized by providing scores that are not necessarily reflective of the empirical probabilities observed in your dataset: ex: suppose the true label = (1, 1, 0, 1) and you have two classifiers (0.51, 0.51, 0.3, 0.51) vs (0.9, 0.9, 0.1, 0.9). Both have the same accuracy assuming 0.5 threshold but clearly very different scores.

Switch your objective to log loss which is optimized only when you feed it well calibrated/true underlying probabilities. Furthermore, look into probability calibration (like platt/isotonic/multiple width binning) if you find that log loss performance is not satisfactory.


R Combine Multiple Rows of DataFrame by creating new columns and union values

What you are asking is how to convert data from "long" form to "wide" form.

You can just use the reshape() function which is in the stats package.

#' I constructed your data frame here
x<-data.frame(ID=c(1234,1234),
          APPROVAL_STEP=c("STEP_A","STEP_B"),
          APPROVAL_STATUS=c("APPROVED","APPROVED"),
          APPROVAL_DATE=c("23-Jan-2019","21-Jan-2019"),
          APPROVER=c("John Smith","Jane Doe"))

#' Now to reshape the data

library(stats)
reshape(x,
    timevar="APPROVAL_STEP",
    idvar="ID",
    sep="_", direction = "wide")

Your output should look like.

  ID APPROVAL_STATUS_STEP_A APPROVAL_DATE_STEP_A APPROVER_STEP_A APPROVAL_STATUS_STEP_B APPROVAL_DATE_STEP_B
1 1234               APPROVED          23-Jan-2019      John Smith               APPROVED          21-Jan-2019
  APPROVER_STEP_B
1        Jane Doe

The column names is a cosmetic issue, but that's the gist of it.

Hope this is helpful!


Mutual information (MI) quantifies the amount of information needed to express one variable with the help of another variable. If two variables are independent of each other, their MI is zero, else it will be greater than zero. MRMR, a mutual information based feature selection uses MI, which considers a feature effective if it has maximum MI with its class label(maximum relevance) and minimum MI with rest of the features(minimum redundancy). This combination of maximum relevance and minimum redundancy will ensure better performance with smaller feature dimension.


"probabilities of finding a neighboring word given a word"

here you refer to the Skip-Gram architecture, where given the center word you predict the surrounding words.

This extract from these notes might clarify your question.   Note that by assuming the conditional independence the total probability factors into a product.

  "As in CBOW, we need to generate an objective function for us to evaluate the model. A key difference here is that we invoke a Naive Bayes assumption to break out the probabilities. If you have not
  seen this before, then simply put, it is a strong (naive) conditional independence assumption. In other words, given the center word, all output words are completely independent."

Maybe this article can also help, though it is about negative sampling it is a very clear exposition.


Statistically Robust Distance Measure/Metric for comparing more than two network data series

Just to clarify the question:
Do the lists describe different graphs, or do you need the similarities to learn which lists refer to the same graph?
Is data tri- or duplicated within single lists or are lists duplicating other lists? Would you consider removing redundant metrics?

Sorry, this was rather a comment than an answer, but I am not yet entitled to comment here, so I am starting the discussion like this.


Using SVM Classifier with C=0 and C=infinity ,what would be the effect on classifying this data?

C value is like lambda, the L2/L1 regularization hyper parameter, but in reverse manner. Whenever C is large, means there is high probability that your model overfit the data in hand. Whenever it is small, your model endure some errors in order to avoid producing small margin and consequently you will have a model less prone to overfitting. The first one is called hard SVM and the latter one is soft SVM. In your case, I guess there will be no difference because your model is linearly separable. If X4 was between X2 and X3 you would have different result for each hard and soft. Soft and hard differ in cases which the data is not linearly separable or maybe they are separable but the margin would be narrow or wide if SVM discards some samples. Soft tries to make the separator space, margin, as large as possible by discarding those minority data which hinder a large margin to be crated but hard tries to separate the data to reduce the error as much as possible. In your case, you will get same result.

For illustrating more, look at the left figure. It tries to discard the sample which hinders the large margin to be created but large margin, considers all the samples.


Association Rules with Python (coded dataset)

Create a dictionary that contains the coded variables as keys and the item names as values.
So it would look like:
dicty = {4: "bread", 7: "milk", 9: "toothpaste"}

Constructing dictionaries in python is really easy if you have them in a table or excel spreadsheet.
dicty = {i:j for i,j in zip(coded_list,normal_list)}

where coded_list is the list of variables in numbers, and normal_list is the list of variables in their categorical names.
Once you have a dictionary you can simply convert them like this:
name = dicty[9]

and it should return toothpaste for name.


Find errors if any in tensorflow computation graph

According to me, reshaping of the input in perfectly ok.
If you are changing the shape of the parameter just to fulfill the requirements of the method parameter or for further processing of the logic, it is perfectly ok, as long as you know what you are sending and how it will be processed by further code.

But in feed_dict, input and placeholder dimensions should be same otherwise you will get the compilation error.

  Note: In case you are not sure about the exact dimensions of your
  placeholder, you can use [None] as a dimension value while declaring a
  placeholder


What cost function and penalty are suitable for imbalanced datasets?

If you have an imbalanced dataset you usually want to make it balanced to begin with, since that will artificially affect your scores.
Now, you want to be measuring precision and recall, since those can capture a bit better the imbalanced dataset biases.
L1 or L2 won't perform particularly better in a balanced or unbalanced dataset, what you want to do is call elastic nets (which is a combination of the two) and do cross validation over the coefficients of each of the regularizers.
Also, doing grid search is very odd, you are better using just cross validation and see what parameters work better.
They even have ElasticNetCV, which does that part for you


Trouble performing feature selection using boruta and support vector regression

Without the full code, there are few things to check but I would try as follows:
Note that you are passing svm_model to BorutaPy and according to your code you should pass the fitted object that is svm_model
from sklearn.svm import SVR

svclassifier = SVR(kernel='rbf',C=1e4, gamma=0.1)
svm_model= svclassifier.fit(x_train, y_train)

from boruta import BorutaPy

feat_selector = BorutaPy(svm_model, n_estimators='auto', verbose=2, random_state=1)
feat_selector.fit(x_train, y_train)
feat_selector.support_
feat_selector.ranking_
X_filtered = feat_selector.transform(x_train)

EDIT:
After reading the implementation of Boruta and since this is based on tree models.
 Parameters
    ----------
    estimator : object
        A supervised learning estimator, with a 'fit' method that returns the
        feature_importances_ attribute. Important features must correspond to
        high absolute values in the feature_importances_.

SVM does not have the attribute feature_importances_, so Boruta can only receive tree models like DecisionTrees, RandomForest, XGB, etc.
If you want to use SVM anyway I would recommend to change the feature selection algorithm to PermutationImportance, which is quite similar way of computing importance base on random repeated permutation, but in this case you will have to provide a metric to measure the decrease on performance when a feature is shuffled.


Best approach for image recognition/classification with few training data

Your data seem to be very small to be used for training a CNN. I do not fully understand what does it mean that an image "belongs" to "Twitter" and not to "Google".

Nevertheless, my best advice to you is to use simple sklearn classifiers and gather some data that does not belong to any of your options to be used as the None class. The None class should include a similar amount of data as the other classes and then you can train any classier and check the scores.

Regarding the over-fitting situation, you should consider doing cross-validation (because your data is too small for hold out set). Try doing a 5-fold cross-validation and assess how good your model performs on average.

Good Luck!


Query related to Matlab Neural Network toolbox

I don't see it explicitly discussed in the documentation, but it appears that the numbers on the right side of the screen are the stopping criteria.  That run appears to have stopped because the error on the validation set increased six times (explained here).  I'm guessing it would also stop after 1000 epochs, when 0 MSE was achieved, the gradient was below 1e-6, or after 4 resets.

Thus the number to the left of the bar should be the MSE on the epoch with the best validation error*.  I'm guessing the value of the blue bar is the lowest that MSE got during the iterations that were actually increasing the validation error.

From the docs:
"In the default setup, the training stops after six consecutive increases in validation error, and the best performance is taken from the epoch with the lowest validation error."


Problem when cherry picking actions - Proximal Policy Optimization

I believe that alphazero handles illegal actions by masking out the illegal logits before feeding to the final softmax output.


Calculation of VC dimension of simple neural network

I do not believe this is correct.  The entire network will represent a piecewise-constant function with at most $k+1$ pieces, and has VC dimension $k+1$.
Each hidden neuron is a step function, and together there are at most $k$ jump points among them.  Taking a linear combination of those, we still cannot create any new jump points, so at the output neuron before activation, we have a piecewise-constant function with at most $k$ jump points, and arbitrary constant values on the $k+1$ intervals between them.  After the activation, it's just piecewise constant with values 0 and 1 on at most $k+1$ intervals.


Information retrieval / slot filling / NLP

This is usually handled as a two step process:

intent detection
slot filling

Intent detection is basically just a kind of classification. So if your program has multiple kinds of questions it can be asked you build a list of examples for each and train a classifier.

Slot filling is typically modelled as a sequence labelling problem, so you can look into seq2seq.

You can find more detailed information in any tutorial on how to make a chatbot.


Will Keras fit( ) function automatically shuffles the input dataset by default?

Yes, by default it does shuffle.
Here is the documentation.
The default call signature:
fit(x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,
    validation_split=0.0, validation_data=None, shuffle=True,
    lass_weight=None, sample_weight=None, initial_epoch=0,
    steps_per_epoch=None, validation_steps=None, validation_freq=1,
    ax_queue_size=10, workers=1, use_multiprocessing=False)

where the description for shuffle is:

shuffle: Boolean (whether to shuffle the training data before each epoch) or str (for 'batch'). This argument is ignored when x is a generator. 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not None.

This means it will not shuffle if you have set a value for steps_per_epoch.


find the difference between two columns in specific rows

for i in range (99,len(df),200):
    try:
       df1=df.loc[i+200,'acctimestamp'] - df.loc[i+200,'gyrtimestamp']
       print(df1)
    except:
       print('End')

Is what you're looking for? Step is 200. So i will take 299 -> 499 -> 699 etc.., and you're taking only the rows between i ( 299 ) and i + 200 ( 499 ) and so on.


copy data between two dataframes in python

If you want to add the data from df1 to df2 you can use pandas.merge. Given that you want to keep all records from df2 and only add data from df1 for the identifiers in df2 you can use the following syntax:
import pandas as pd

pd.merge(df2, df1, how="left", on="id_column")


How to split train/test data 50% by class and grouping by Object ID in R?

Instead of:

dataTrain <- EXTRACTED[trainIndeks,]
dataTest <- EXTRACTED[-trainIndeks,]

try:

dataTrain <- EXTRACTED[ID %% 2 == 1,]
dataTest  <- EXTRACTED[ID %% 2 == 0,]

dataTrain will contain only odd ID, dataTest only even ones.

Using this method you may lose the balance of CLASS distribution between dataTrain and dataTest, but you may improve it by replication or removing of a few records, or by checking the distribution in other splitting, e.g.

dataTrain <- EXTRACTED[ID %% 4 < 2,]
dataTest  <- EXTRACTED[ID %% 4 > 1,]

EDIT:

For random changing the split, you can use:

set.seed(123)
N <- 10
#N <- round(max(EXTRACTED$ID)/10) # for more random grouping
p <- 0.5 # train/(train+test) ratio
idx <- sample(0:(N-1),round(N*p))
dataTrain <- EXTRACTED[ID %% N %in% idx,]
dataTest  <- EXTRACTED[!(ID %% N %in% idx),]


Is it a good practice to evaluate model performance by comparing the metrics of rescaled (inverse transformed) predictions and true target values?

The error as a percentage of the true value in each case is probably the same - it just seems bigger because the standardised values are a lot smaller than the unstandardised ones (standardised values will be mostly between -1 and 1 and the unstandardised values will be whatever they are in your dataset). Try measuring the fit of your linear model using Pearson’s correlation coefficient, checking how your predicted values correlate to the true values. This should be the same for both the scaled and unscaled cases and will give you a better idea of whether your model is good. You could also plot your predicted values against the true values in a scatter plot so that you can see roughly what’s going on.


The aim of Doc2Vec is to produce document level embeddings, thus even if words are sentence-separated if you include them in the same document it has to be considered part of the same semantic source for word similarities. If you don't want such a behavior, you might want to separate your documents in a sentence-level and maybe aggregate the embeddings later in different groups to form your initial paragraphs (depending on your plans)?


In the context of Kaggle, it means LeaderBoard (emphasis mine).


MLP typically refers to a type of neural network called a 'Multi-Layer Perceptron'. As you can read on the Wikipedia page, these neural networks consist of neurons organized in layers. Each neuron has a (typically fixed) function that transforms its weighted input to produce the output for that particular neuron known as its activation function.
A 'network architecture' refers to the way the neurons are laid out in layers in the network, sometimes including the activation function. In the given example, there are 120 neurons in the first input layer. These are connected to 7 neurons in the next layer, which in turn are connected to 24 neurons in the last output layer.
The 120 input neurons in the linked article appear to include both history, i.e. a sliding_window as well as some known future information (temperature, from weather forecasts it seems).


one hot encoding target variable in tree and non tree (knn) methods

The answer depends on the library you are using. If you are using scikit - you wouldn't need to one hot encode the targets. Scikit handles it automatically.
If you were using keras to build a neural network, you might want to use one hot encoded labels because the built in loss function in keras (e.g categorical crossentropy) expects labels to be one hot encoded.
So both approaches work and ate a function of the library or tools you use.


How do you evaluate ML model already deployed in production?

After the models are deployed in production, I'd monitor the following:

(1) The same metric you used to evaluate the performance of your model, in some cases it is accuracy, or it could be precision, recall, RMSE. I'd plot a daily time series charting the metric and see that it is still performing above a satisfactory threshold. There might be seasonality within the calender, the model performs well around certain months and not so well in other months. I'd compare the performance against the test/validation sets of the same months to account for seasonality.

(2) Apart from looking at the performance of the model, especially if one is working with shared computing resources, I'd also recommend keeping a close tab on the data aggregation runtime, model runtime, success rates of models runs over the past period.


The affect of bootstrap on Isolation Forest

This is well explained on the original paper Section 3.
As well as in the Supervised Random Forest, Isolation Forest makes use of sampling on both, features and instances, so the latter in this case helps alleviate 2 main problems:

Swamping

Swamping refers to wrongly identifying normal instances as anomalies. When normal instances are too close to anomalies, the number of partitions required to separate anomalies increases – which makes it harder to distinguish anomalies from normal in- stances.

Masking

Masking is the existence of too many anomalies concealing their own presence.

Contrary to existing methods where large sampling size is more desirable, isolation method works best when the sampling size is kept small. Large sampling size reduces iForest’s ability to isolate anomalies as normal instances can interfere with the isolation process and therefore reduces its ability to clearly isolate anomalies. Thus, sub-sampling provides a favourable environment for iForest to work well. Throughout this paper, sub-sampling is conducted by ran- dom selection of instances without replacement.


Gini impurity in decision tree (reasons to use it)

Misclassification error will not help in splitting the Tree.
Reason-We consider the weighted dip of error from parent Node to the child node and misclassification error will always result in 0(Other than pure splits).
Let's consider an example
Data = 1, 1, 0, 1, 0, 1, 0, 1, 0, 1
Parent Classification error= 4/10 = 0.4
Parent Gini Impurity = 1-(0.4x0.4+0.6x0.6) = 0.48
Case - I
Split - 1, 1, 0, 1  Vs 1, 0, 1, 0, 1, 0
Classification error= 0.4x0.25 + 0.6x0.5 = 0.4, split not possible
Gini Impurity = 0.45
Case - II
Split - 1, 1, 1, 0, 0  Vs 0, 1, 0, 1, 0
Classification error = 0.5x0.4 + 0.5x0.4 = 0.4, split not possible
Gini Impurity = 0.48
Case - III
Split - 1, 1, 0, 0, 1, 1  Vs 1, 0, 1, 0
Classification error = 0.6x(2/6) + 0.4x0.5 = 0.4, split not possible
Gini Impurity = 0.477
Pure splits
Split - 1, 1, 1, 1, 1, 1  Vs 0, 0, 0, 0
Classification error = 0, split possible but no further splits
Gini Impurity = 0
Reference-
Sebastian Raschka Blog


Neural network for MNIST: very low accuracy

What loss function are you using? It looks to me like you're using squared error loss (am I right?). This might work, but consider using cross entropy loss instead, which is more well suited for classification problems.

Also, by using the logistic function as activation function in the last layer, you're treating the problem as ten separate binary classification problems. While this also might work, since the problem is a multi-class classification problem, you should probably change the activation function in the last layer to softmax.

One error I spot in your code is that you're not taking into account the derivative of the nonlinearity in the last layer. To fix this, you should change

delta3 = x3 - Y[:,j];

to

delta3 = (x3 - Y[:,j]) .* sigmoidGradient(z3);

similar to the way you calculate delta2.


SKLearn - Understanding Discrepancy Between LogisticRegressionCV classification_report and scores_

You are training and evaluate on the same data.
You should split data to train set(to train) and test set(to evaluate) to get proper results.
Kinda:

decoder.fit(X_train, y_train)
y_hat = decoder.predict(X_test)

print(classification_report(y_test, y_hat))
print(confusion_matrix(y_test, y_hat))


Is it correct to create topic models using both train and test data?

You should only use your training set in this context though it may benefit you if you used an additional cross-validation set for feature selection. If you use training data in your feature selection step you will optimistically bias your model and expose it to overfitting because of the data leakage. A similar scenario is also described in this post.


When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ordinality.

LabelEncoder can turn [dog,cat,dog,mouse,cat] into [1,2,1,3,2], but then the imposed ordinality means that the average of dog and mouse is cat. Still there are algorithms like decision trees and random forests that can work with categorical variables just fine and LabelEncoder can be used to store values using less disk space.

One-Hot-Encoding has the advantage that the result is binary rather than ordinal and that everything sits in an orthogonal vector space.  The disadvantage is that for high cardinality, the feature space can really blow up quickly and you start fighting with the curse of dimensionality. In these cases, I typically employ one-hot-encoding followed by PCA for dimensionality reduction.  I find that the judicious combination of one-hot plus PCA can seldom be beat by other encoding schemes.  PCA finds the linear overlap, so will naturally tend to group similar features into the same feature.


100 data points is a very very low number of points. I am quite surprised that the algo managed to learn something meaningful (well unless you have very obvious relationships between the features and the target). I'll suggest you look at what the logistic regression learned exactly (what coefficients are deemed significant). There is an important risk that you are overfitting on your very very small dataset. And you can't really catch that so your are just left with checking if the rationale of your model make sense.
Yes, you should probably tell your boss that this is not enough data points.


What you want to achieve with this validation strategy is a robust estimate of what combination of hyperparameters is good enough for your final model, so:

for each combination of hyperparameters, you carry out k trainings (as you ask with your last question), following this schema:

source of info

once you have this k-trained models (i.e. for one hyperparams combination) you find the mean and standard deviation of the k sub models

you repeat this process for all the hyperparameters combinations you wnat to try out

Once you select the model hyperparams which provided the best mean value (and std) of the desired evaluation metric, you retrain with all the training dataset (the green section in the image), and evaluate it with the never-seen-before blue test data.
Luckily for you, this whole process is automated with helpers like thos one from scikit-learn, which already retrains the final model for you, and is accessible via the best_estimator_ attribute.
If you want a bit more detailed documentation about this cross-validation, you can have a look at this link.


finally worked something out though I wonder if it is the best solution?

dt = pd.DatetimeIndex ( dt ).astype ( np.int64 )/1000000

cheers.


How to use different classes of words in CountVectorizer()

First of all your question is about stemming words as mentioned in the other answer which can be found in any Python NLP library such as Spacy or NLTK.

The other point to mention here is that despite the other answer, what libraries has as Stop Words list is not actually stop word! Do no remove them! In NLP stop words should be extracted based on working corpus not based on a predefined list. In practice removing this kind of stop words usually reduces the performance on specific domain corpuses.

The Third point is that depending on the classifier and loss function you use, TF-IDF might be better than Count Vectorizer. I suppose it works better specially if Log Loss is the cost function but I am not sure. Just give it a try.


Given that two images are similar, how do I find and classify the difference between them?

In order for you to do pairwise difference, you need to do the following:

Generate Ground truth data by drawing bounding boxes on elements you need comparison for and label them i.e if you want to compare text on the hard cover, you need to mark the text region and add a label to it like font type.
Once you have ground truth data, you can then use object detection networks like different variants of Yolo to identify multi class objects in each document.

This should provide you a starting point to solving the problem at hand.


Treat similar observations in a classification problem

Wouldn't remove similar looking observations unless you have a strong reason to do so. By deleting similar looking observations you may be adding bias into the underlying distribution responsible for generating the data.
Your model may be misled into learning a biased distribution and that may affect final performance.
To start with use the entire dataset as is. If the size is a concern, then sample a random subset of the data to build your model.


Keras -- Transfer learning -- changing Input tensor shape

You can do this by creating a new VGG16 model instance with the new input shape new_shape and copying over all the layer weights. The code is roughly

new_model = VGG16(weights=None, input_shape=new_shape, include_top=False)
for new_layer, layer in zip(new_model.layers[1:], model.layers[1:]):
    new_layer.set_weights(layer.get_weights())


To One-Hot-Encode or not to One-Hot-Encode?

An intuitive explanation, why we should encode categorical features, is that otherwise there will be absolutely another sense of "closeness" between features of the same type. The model will treat this features as continuous and as a result if you have two points in your feature space (p1 with CountryCode 61 and p2 CountryCode 962) and then you add the third point p3 with CountryCode 81, then model will miss out, that it can't evaluate distances between 61, 962 and 81.