The setting:
We have a neural network $\phi_{\mathbf{w}}:\mathbb{R}^{n} \rightarrow \mathbb{R}^{m}$ with weights $\mathbf{w} \in \mathbb{R}^{q}$. A loss function $\hat{L}: \mathbb{R}^{m} \times \mathbb{R}^{m} \rightarrow \mathbb{R}$ evaluates the quality of a prediction.  If $x \in \mathbb{R}^{n}$ shall be mapped to $y \in \mathbb{R}^{m}$ by the neural network, the loss is given as $\hat{L}(\phi(x),y)$.
For a fixed dataset $D \subset \mathbb{R}^{n} \times \mathbb{R}^{m}$, we obtain the empirical error
$F(\mathbf{w}):= \sum_{(x,y) \in D} \hat{L}(\phi_{\mathbf{w}}(x),y)$.
Then $F: \mathbb{R}^{q} \rightarrow \mathbb{R}$.
Now $F$ is minimized using backpropagation.
Let us try to define the vanishing gradient term. I am not sure if there is a proper definition, but I would say we have a vanishing gradient at $p$ if $0 <||\nabla F(p)|| \leq c$ for some small $c$.
Raised questions:

If the gradient is almost zero due to the vanishing gradient, does that mean the current solution is very close to the optimum ? So we can stop iterating..

Why is it bad to have "vanishing gradients" ?

Adressing Question 1
Recall from school that if a functional $F$ has a local optimum at $p$, then $\nabla F(p) = \mathbf{0}$ and $D^2 F(p)$ definite.
If $D^2 F(p)$ is positive definite ($x^T D^2 F(p) x > 0$, for all $x$, where $D^2 F(p)$ is the Hesse matrix), then $p$ is local minimum.
If $\nabla F(p) = \mathbf{0}$ and $D^2 F(p)$ in definite, then $p$ is a saddle point.
In particular this shows that having a zero gradient does not always imply that the position is a local optimum.
(In case of $q = 1$ and $F$ being two times differentiable, $F$ has a local optimum at $p$, if $F'(p) = 0$ and $F''(p) \neq 0 $. )
We can also construct a function that can have arbitrary small gradient while being far away from the minimum:
Consider the  function $f_{c}(x) = \max\{0,cx\}$ with $c>0$. Then $\min_{x \in \mathbb{R}} f_c(x) = 0$. For any $p>0$, we have $f'_{c}(p) = c$.
As an example let $p = 10^{9999}$ and $c = 10^{-90}$. Then, the value $f_{c}(p)$ is far away from the minimum, still for the gradient $f'_{c}(p) = 10^{-99}$ holds, which shows that a small gradient does not imply that the current point is close to the optimum.
Adressing Question 2
Note that performing backpropagation is  performing the gradient descent algorithm.
Now to address the section questions, there are two directions (an analytical answer and a numerical answer).
The analytical answer would be that a vanishing gradient is nothing special that needs to be considered.
If the step size is choosen appropriately, it can be shown that the sequence of iterates $(p_k)$ is either finite with $\nabla F(p) = 0$, or it is an infinite sequence, and $\lim_{k \rightarrow \infty} \nabla F(p_{k}) = 0$, so that each limit point is a stationary point. This will work independent of any "vanishing gradients".
However, if we consider the question from the numerical aspect, there are certain issues.
1.) There is a machine epsilon $\epsilon$ so that updates with values smaller than $\epsilon$ cannot be performed numerically in a computer. This effectively means that the algorithm converges to some point if $||\nabla F(p)|| \leq \epsilon$.
2.) Even if the values are bigger than $\epsilon$, a "small" gradient vector results in very slow weight updates.
3.) The vanishing gradient problem may arise for example if the sigmoid function is used as activation in a deep neural network. This can be understood from the chain rule.
Let us compute a simple example where we have neural network consisting of $L$ layers, and each layer consists of a single neuron, without any bias. As activation function, we use the sigmoid function $\sigma(t) = \frac{1}{1+e^{-t}}$. Then $\sigma'(t) = \sigma(t)*(1-\sigma(t))$.
The output of the i-th layer is given as
$f_{i}(w) := \sigma(w o_{i-1})$, where $o_{i-1}$ is the output of the $i-1$-th layer.
Let us ignore the loss function and assume that $F$ is exactly the neural network.
The output at layer $i$ is denoted as $o_{i}$ so that we have $o_{i} := \sigma(w_{i} o_{i-1})$ and $o_{1} = w_{1}$.
Then $F(w_{1},\ldots,w_{L}) = o_{L} = \sigma(w_{L} o_{L-1})$.
According to the chain rule, we have $\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w) = \frac{\mathrm{d} \sigma}{\mathrm{d}w_{1}}(w_{L} o_{L-1}) = \sigma'(w_{L} o_{L-1})  \frac{\mathrm{d} w_{L} o_{L-1}}{\mathrm{d}w_{1}} = \sigma'(w_{L} o_{L-1}) w_{L}  \frac{\mathrm{d}  o_{L-1}}{\mathrm{d}w_{1}}$.
Repeatedly applying the chain rule results in:
$\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w) = \prod_{i = 2}^{L} w_{i} \prod_{i = 2}^{L} \sigma'(w_{i}o_{i-1})$.
Since $\sigma(t) \in [0,1]$, we have $\sigma'(t) \in [0,1]$. In particular, we often want $\sigma$ to output either $0$ or $1$. However, if $\sigma(w_{i}o_{i-1})$ is close to $0$ or $1$, then $\sigma'(w_{i}o_{i-1})$ will be close to $0$.
Now if one (or multiply) numbers are close to zero in $\frac{\mathrm{d}F}{\mathrm{d}w_{1}}(w)$, we obtain a very small number, which results in numerical issues (due to reaching machine epsilon), causing a very slow update during the algorithm.


How to obtain original coefficients after performing linear regression on normalized data?

Coefficients of the linear regression for unnormalized features

If parameters in the normalized space are denoted as $(\theta_0', \theta_1', \theta_2')$, parameters in the original space $(\theta_0, \theta_1, \theta_2)$ can be derived as follows
$$\begin{align*}
y' &= \theta_2'x_2'+\theta_1'x_1'+\theta_0'\\
\frac{y-\mu_Y}{\sigma_Y} &= \theta_2'\frac{x_2 - \mu_{X_2}}{\sigma_{X_2}} + \theta_1'\frac{x_1 - \mu_{X_1}}{\sigma_{X_1}} +\theta_0' \\
 y &= \overbrace{\left(\frac{\sigma_{Y}}{\sigma_{X_2}}\theta_2'\right)}^{\theta_2}x_2+ \overbrace{\left(\frac{\sigma_{Y}}{\sigma_{X_1}}\theta_1'\right)}^{\theta_1}x_1
 + \overbrace{\sigma_Y\left(-\theta_2'\frac{\mu_{X_2}}{\sigma_{X_2}}-\theta_1'\frac{\mu_{X_1}}{\sigma_{X_1}} + \theta_0'\right) + \mu_Y}^{\theta_0}
\end{align*}$$

Generalization to D features

$$\begin{align*}
\theta_d = \left\{\begin{matrix}
 \sigma_Y \left(\theta_0' - \sum_{i=1}^{D}\theta_i'\frac{\mu_{X_i}}{\sigma_{X_i}} \right) + \mu_Y& d=0\\
\frac{\sigma_{Y}}{\sigma_{X_d}}\theta_d' & d > 0
\end{matrix}\right.
\end{align*}$$

A trick

For visualization, we can plot the plane in original, un-normalized space without changing the parameters (Thetas). We only need to re-label (re-scale) the plot axes.

For example, a point $(x_1', x_2', y')$ in the normalized space corresponds to point $$(x_1, x_2, y)=(\sigma_{X_1}x_1'+\mu_{X_1}, \sigma_{X_2}x_2'+\mu_{X_2}, \sigma_{Y}y'+\mu_{Y})$$

in the original space. So you just need to rename the plot axes from $(x_1', x_2', y')$ to $(x_1, x_2, y)$.

Note that $y'=\theta_2'x_2'+\theta_1'x_1'+\theta_0'$ is still calculated using normalized features.


Can you provide examples of business application of vector autoregressive model?

Caveat: I have a doctorate in economics and that is why I knew how, and where, and when to apply this type of model.

Sure, I used a vecm model last year to figure out how many credit cards get compromised per month given how much activity we see. (I work for a financial institution).  We were looking for a long-run relationship and a short-run relationship. From this model, a survival analysis, and a couple of other techniques we were able to determine the optimal time to set the expiration date on a card. I applied this technique because I was afraid I was getting the wrong answer, namely because the number of transactions and number of fraud transactions were cointegrated.(All of the models I tried gave similar results which was comforting). Given that we are talking about fraud strategies I don't feel comfortable giving more details than this, hopefully, that gives you a flavor of what I was trying to accomplish. Basically, I was trying to do a robustness check.

It isn't easily applied in a business context because this model isn't intuitive, it isn't easily explained, and it takes great skill to interpret it. The "mathematical" answer probably has something to go like this, ask an average data scientist what the impulse-response function for their model looks like and you will get a blank stare. Ask them what the FEVD looks like for their model, and you will get a blank stare. Data scientists tend to be trained in non-linear models (read machine learning), not cointegrated time-series models. It isn't that they aren't capable, they've just never been trained to think in those terms. I know technically not math, and technically that is an opinion but it is probably the truth.

Also doing the math, I can probably get a better fit out of a non-linear model like an LSTM network(?). So if I only care about pure prediction, it wouldn't be worth my time. Where VAR and VECM shine is that although complicated, they are essentially linear, and therefore, fairly interpretable. So perhaps what you are looking for is a business setting that requires causally valid, interpretable, multivariate time-series models. If you look hard enough, I'm sure you can come up with a couple of interelated KPIs being tracked in your business where it would make sense.

But in my experience, if you need a fast dirty answer, forgive me econometrics teachers everywhere, you can just plug the differenced series into a linear regression. (Warning not best practices, but it does alright).


Export weights (formula) from Random Forest Regressor in Scikit-Learn

The SKompiler library might help:

from skompiler import skompile
skompile(rf.predict_proba).to('excel')

Check out this video.


What are the best ways to use a time series data for binary classification

From your comment, I understand that you are trying to solve the binary classification problem using your aggregated data and you are getting very poor results when you simply use the mean.

Depending on specifics of your data and the shape of your time series, there are several alternatives that you could try. Note, that you might need (significantly) more than just a single number per time series to solve your problem.

In addition to the mean, you could use the quantiles or some other summary statistic, like standard deviation, min, or max.
You could try to sample the data, i.e. instead of taking the entire time series, pick only the values that are minutes, hours or days a part. Or pick only mid-day values. The frequency of the sampling depends on your data.
Or just pre-aggregate by calculating averages for every hour, day, month, etc.
Additionally, you could calculate the periodicity of your time series and use it as a new feature.
Or calculate some trends.
Try to fit some standard time series models to your data, e.g. ARIMA and use the coefficients as informative features.
Last but not least, use the domain knowledge re what could be relevant feature for your classification problem: the biggest jump (max first order difference), change of regime, etc.

Edit
I’d pick at least 10-20 features per time series generated as described above and apply logistic regression with LASSO or even xgboost.

After selecting 10-20 features per time series you also could try PCA to reduce the dimension.


Expanding my comment:

One can use a regression method (eg linear, non-linear, symbolic, ..) to try to fit a curve to the data.

Outliers can be removed in 1 or 2 steps:
2.a. Remove some points before applying regression, eg by testing how much away they are wrt to std.
2.b. Remove outliers after fitting the curve and measuring distance of point from curve and discard if above a certain threshold.

[UPDATE]
In curve fitting using regression, one rarely has knowledge of the actual curve, instead one assumes the curve belongs to some class of functions and tries various options.
Furthermore, there are what are called symbolic regression methods, which assume even less about the actual curve and can even find functions in implicit form, although more tedious.
Finally, some pre-processing to remove outliers before applying regression (eg discard points very "far", density-based approaches, ..) can certainly improve the outcome based on reasonable domain-based assumptions.
The above will work for most of the cases presented in question. This is an ill-posed inverse problem, so one should make some assumptions and have reasonable expectations.
[UPDATE 2]
If your data is only 2D image data and you simply need to get the curve as an image (and not need to infer any analytical form of the curve), then doing a combination of blur/lowpass filtering and edge detection, can be an option. OpenCV, for example, has methods to achieve that.
References:

An edge detection based noise removal algorithm

We present an image denoising method using the edge map of an image.
The denoised image is considered as a linear combination of the
observed image and its average value, where the coefficients are
controlled by a local edge detector. The parameters are set on
suitable values related to noise energy computed by the curvature norm
of the original image. Implementation can be done in a single
iteration and the speed of the process is reasonably high. Noise
reduction quality of the introduced method is compared with Wiener and
Total Variation based filters for some images. The method appears to
be easy, fast and useful for very noisy images. The differences
between our method and the patent 6229578 “Edge Detection Based Noise
Removal Algorithm” are explained.

How to remove the noise without destroying the main edge?

Canny Edge Image - Noise removal


Can I use sentiment analysis techniques and text processing for detection social bot in online social networks?

Yes, because it has been done before:

http://dl.acm.org/citation.cfm?id=2808779


Visualization of an Order Verified by Wilcoxon

I do not have a good way to answer your question since I do not know who your audience is, what actions/understandings are you trying to drive from your analysis, etc.

I think that is where you need to start. If your audience is executives in charge of revenue, perhaps showing the difference in revenue where the top n preferences are in a car vs n-1, n-2, ....  Or frequency of the top preferences.

Given the audience and the action/understanding you may not need to show all 42 preferences in the mainline of your presentation.

You can also look for inspiration on your audience/action from traditional statistical methods for this type of analysis.  Namely conjoint part of choice modeling. Take a look at some of those public analysis or classes and maybe an idea will take hold that is similar to your audience/action.

This is more of a path forward rather than an answer but I do not see the question as answerable right now.


The problem is that you say "apply(lambda x:[i.split()[0] for i in x])"
As soon as you say apply, x is your list. So you can say following
"apply(lambda x:x[0].split(" ", 2)[0])"
Meaning you say take first element in the list, than split on " " and in two parts. And than take the first word (part) with the last [0]


First as "timleathart" mentioned, you need to fix your code by changing this line :

X_test = sc.fit_transform(X_test)

to:

X_test = sc.transform(X_test)

For your question :

you have already the number of samples (training_samples) used for training. so all you need is to iterate over the y_pred and save the values in new column in the dataset starting from "training_samples" as row index.


Ideal aggregation function for Partially Connected Neural Network (PCNN)

What you call sum product is the same as a dot product of two vectors of equal length, $(w_{0j},w_{1j},\dots,w_{nj})$ and $(x_0,x_1,\dots,x_n)$.

Based on your diagram, your difficulty is that you want to be able to define connections between neurons that are not in consecutive layers (in other words, the connections might skip layers). I assume that there are no directed edges (synapses) that would go in the opposite direction (from layer $n$ to layer $m$ for some $m<n$). In this setting, you have to forward propagate information to layer $n$ from input $X$ and layers $1,2,\ldots,n-1$. I don't know when and how often you want to 'update' your network topology. Do you want to change it during training?! (Otherwise you'd fix a partially connected NN and you wouldn't write update in your question.)

For forward propagation in a fixed network topology, I would associate to neuron $j$ a vector of weights $(w_{ij})_{i\in I_j}$, where $I_j$ is the set of neurons that bring information to $j$ (those $i$ where there exists a directed edge from $i$ to $j$). Then you could either cherry-pick the neurons $i$ from the set of all neurons (by maintaining a list of indexes to find them in the vector of all neurons) and put them into a vector $x$, which is now as long as $(w_{ij})_{i\in I_j}$, and you can use dot product on the two vectors. (With numpy for Python, numpy.dot would work.) You would need to do this separately for all $j$ in the given layer. As these are receiving information from differing numbers of inputs and neurons, they indeed cannot be arranged into a matrix/tensor, not even receiving neurons of the same layer.

Or you could force all potential preceding neurons into one vector. Neuron $j$ of layer $n$ can only receive information from neurons $x=(<$input nodes$>$, $<$neurons of layer $1>$, $\ldots$, $<$neurons of layer $n-1>)$. This is a vector, and you can define a vector $w$ of equal length for $j$, which would indeed have many zeros, and $x$ and $w$ can be multiplied. For all neurons of layer $n$, you could stack these $w$ as row vectors on top of each other into a matrix/tensor of size (the number of neurons in layer $n$) times (the size of input $+$ the number of neurons in layer $1$ $+\ldots+$ the number of neurons in layer $n-1$) and use np.dot between this matrix and the $x$.

How to do training (backpropagation) with such topologies is something you have to consider a bit. I suspect it will work without much adjustment compared to a fully connected neural network without layer skipping. In the second option, you would only need to zero out the weights which must be kept zero after each backpropagation update. This indeed carries overhead if you're not careful. But if you maintained a binary ($0$ or $1$) 'mask' matrix of which weights are live (which correspond to existing connections), then you would only ever update these weights in the backpropagation.

Additionally, you may want to introduce a bias term $w_{-1j}$ (also known as intercept term, a constant offset) as if $x$ had an extra dimension that is kept constant $1$. After the aggregation with the sum product, you should also pass the result through a non-linear activation function before entering the value into the neuron in the next layer.


can someone give me an idea how to extract information for text such as invoice bills using natural language processing

You might be looking for regex. With this you can parse the text based on your specified search patterns.

Some info:
https://en.wikipedia.org/wiki/Regular_expression

With python you can follow this documentation in order to build your regex expressions:
https://docs.python.org/3.8/library/re.html


Does testing a training dataset guarantee successful results?

This is correct, there's no guarantee at all, not even a high probability. As usual It depends on the type of model, the data, the number and distribution of the classes.
However there's of course a higher chance that the instance would be correctly classified. That's why one shouldn't use a test set containing training instances to estimate the performance of the model, since there's a high risk the performance would be overestimated (data leakage).


Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

This was caused by numerical precision of floating numbers. It becomes really apparent as we change one of input values, by a tiny $\epsilon$ which (in my example above) emedially affects the cost function.

The tricky thing was - because the cost function is Mean Square Error (MSE), and there are no other layers in my network, we indeed can use any epsilon to estimate the slope. Even ridiculous epsilon will work, and will be numerically more stable, explaining why discrepancy seemed to get better. That's just how $y=x^2$ works.

In practice, when doing gradient-checking, I switched to another cost function, which is linear (instead of MSE). This improved the numerical stability, allowing me to use epsilon that is 10 times smaller:

$$C=\sum^r\sum^n(obtained-expected)$$

the gradient is then simply:

$$\frac{\partial C}{\partial (obtained)_{rn}} = 1$$

I don't use this cost in production, just for purpose of gradient-checking.

So yeah, this explains why large epsilon was improving the discrepancy.

Edit:

Notice that this linear-cost function must not be used if you have a softmax right before it. That's because the sum of softmax is always 1.0

In that case you must use the MSE


Netflow anomaly detection python packages

I could not find any specific package in the internet that does the same thing as netflow anomaly detection, but I found out how to train some standard algorithms that solve the same problem. My understanding is that the problem is "network intrusion detection" or "anomaly detection in network traffic to detect network intrusion"

So creation of a model can be divided into several parts:

choosing an appropriate algorithm
understand how to transform the data for the algorithm
finding the data set
training the model

The above tasks in more detail:

algorithm: k-NN or Random Forest. According to the paper Machine Learning DDoS Detection for Consumer Internet of Things Devices k-nearest neighbor is a pretty precise algorithm in network anomaly detection.  Nearest neighbor algorithms are present in scikit-learn python package (link). Random forest classifier performed even better. scikit-learn also has a random forest classifier (link).

Here is the precision table from the first paper. KN is k-NN. RF is Random Forest. NN is Neural Network. LSVM is a variation of Support Vector Machine algorithm.

How k-NN for DDoS works is explained in this paper: Proactive Detection of DDoS Attacks Utilizing k-NN Classifier in an Anti-DDos Framework.
According to the above mentioned papers and other sources the following datasets are used for training:

KDCup1999 This is a network intrusion database
MIT Lincoln Lab, 2000, DARPA intrusion detection scenario specific datasets Contains two DoS attack scenarios
Reasearchgate question on available datasets

On this link you will find how to train a k-nearest neighbor in scikit-learn.

Here is an example code from the link:

>>> X = [[0], [1], [2], [3]]
>>> y = [0, 0, 1, 1]
>>> from sklearn.neighbors import KNeighborsClassifier
>>> neigh = KNeighborsClassifier(n_neighbors=3)
>>> neigh.fit(X, y)
KNeighborsClassifier(...)
>>> print(neigh.predict([[1.1]]))
[0]
>>> print(neigh.predict_proba([[0.9]]))
[[0.66666667 0.33333333]]


What is advantage of using the theory of point processes instead of probability distribution functions?

Machine learning is about finding a model for your data and sometime the Point process is a better model than uniform or normal probability distribution.


Optimization methods used in machine learning

In my reply here

Does gradient descent always converge to an optimum?

it is explained that standard gradient descent works well because backtracking gradient descent works well (proven in our recent paper mentioned in the post) and in the long run backtracking gradient descent behaves like the standard gradient descent.

The main issue with other methods, I think, is that they require too strong conditions for convergence or no convergence is proven at all. In both cases, these make them less applicable to realistic applications.


Is it valid to include your validation data in your vocabulary for NLP?

Yes.  That should be fine.  What you suggest makes sense and should not bias the results.  The reasons you give are good ones.  For any reasonable classifier, if the value of an attribute is always zero in the training set, that should cause the attribute to be essentially ignored.

There is a simple test to let you confirm this.  You can try, for each document in the validation set, zeroing out the entries in the feature vector that correspond to words that were not present in any document in the training set, and see if that changes the classification.  If it doesn't, then you know that your method has had no effect and hasn't introduced any bias.

Here is another alternative, if you prefer:

Conceptually, in an ideal world the vocabulary for each fold would only include words in the training set.

As a matter of implementation, it's certainly possible to implement that in a more efficient way than re-generating the vocabulary and re-generating the feature vectors for each fold.  As long as your implementation generates the same vectors, it doesn't matter how it obtains them.

Thus, you could have an implementation that works like the following:

Generate a "superset vocabulary" that includes all of the words found anywhere in the dataset (not just the training set).
Generate a sparse feature vector for each document, based on the "superset vocabulary".  I suggest that you represent these in an efficient way, e.g., as a Python dictionary that maps from word to count.
For each fold:

Split into training and validation sets.
Generate a subset vocabulary, containing only the words in the training set.  Probably there is a small set $S$ of words that are in the superset vocabulary but not the subset vocabulary, so I suggest storing that set $S$.
For each document in the training set and validation set, generate a derived feature vector for the document, using only the subset vocabulary.  I suggest generating this by starting from the feature vector for the superset vocabulary, then removing the words in $S$.  This should be more efficient than regenerating the feature vector from scratch.
Now train and validate these derived feature vectors.

This is equivalent to the "conceptually ideal" approach, but will run faster.

That said, I think your proposal is the pragmatic one, and in practice should yield equivalent results, and be even faster still.


Hyperparameter tunning for Random Forest- choose the best max depth

In general, the max depth parameter should be kept at a low value in order to avoid overfitting: if the tree is deep it means that the model creates more rules at a more detailed level using fewer instances. Very often some of these rules are due to chance, i.e. they don't correspond to a real pattern in the data.
Overfitting is visible in your graphs from the quite high difference between training and test set performance. One can observe that the performance on the test set increases with the first few values of depth (I'm guessing until around 5 or so), and it starts decreasing after that. So the optimal point (performance and no overfitting) is the point that you mention, it's after this point that the model is overfit.
The stable part of the graph is probably due to another parameter (e.g. min. number of instance per leaf) which stops the model from overfitting even more. If the model was free to overfit as much as it wants, it would probably reach max performance on the training set and very low performance on the test set.
Note: I'm confused why you're using both accuracy and MAE, normally the former is for classification and the latter for regression.


XGBoost Feature Importance, Permutation Importance, and Model Evaluation Criteria

So your goal is only feature importance from xgboost?
Then don't focus on evaluation metrics, but rather splitting.
I would suggest to read this. Using the default from tree based methods can be slippery.


It uses Xavier initialization. If you don't initialize it, by default it uses this method for initialization. If you want to know how to perform that you can use here. In the first link at the end, link of the paper is also available.

This Xavier initialization (after Glorot’s first name) is a neat trick that works well in practice. However, along came rectified linear units (ReLU), a non-linearity that is scale-invariant around 0 and does not saturate at large input values. This seemingly solved both of the problems the sigmoid function had.


I can think of two options:

Train a custom supervised tagger for your data, typically with a sequence labeling method such as CRF. This will require a quite large amount of annotated data (and a specific formatting), but if done well it should give you quite accurate results.
Use manually defined patterns based on keywords (such as "My * and I", "family", "friends") directly associated with a predefined category, and match these patterns in the data. You can do several iterations of defining new patterns for cases that are not matched, thus refining progressively. Depending on your data and how far you go with this you should be able to correctly match most cases, possibly reaching as good accuracy as a tagger for much less work.

Btw be careful how you represent your data: the same user id might not always give you the same group of people since they can go to a restaurant one day with their family, the next day with their wife, the day after with their mistress, etc. Also a "family of 2, expecting" usually becomes "a family of 3" after 9 months ;)


sklearn.GridSearchCV predict method not providing the best estimate and accuracy score

Summarizing your results - your trained a model using gridsearch.
accuracy score on the train set is ~0.78.
accuracy score on the test set is ~0.59.
Rephrasing you questions: why do my model performance on the test set is worse than on my train set?

This phenomena is very common - and I can think of two potential explanations:
1) Overfitting: your trained model had learned the 'noise' in the train set and not the actual pattern.
 Then when you use your model to predict on the test set, it predicts the noise he had encountered(which is not relevant for the train set - thus lower accuracy).
2) Train set and data set are not generated from the same process/describe different parts of it. In this case - the pattern learnt by the trained model is not relevant to the test set and accuracy will drop.
This may happen in situations where the train/test split is done without considering the actual underlying process. For example - an image classification problem where you model whether this picture is an animal or not and the train data has only human and dogs sample and the train set has only fish and airplanes pictures. This is less likely to be the explanation, since your using grid search, which adds randomness to this splitting.


How to figure out if two sentences have the same meaning with AI?

This corresponds to an NLP task called paraphrase detection. It's an active area of research, as far as I know there's no ready-to-use system able to perform this task very well, but there are probably a good few methods and prototypes around. A quick search gives these links for example:

https://aclweb.org/aclwiki/Paraphrase_Identification_(State_of_the_art)
https://arxiv.org/abs/1712.02820


memory error while converting images into an array

You can actually compute how much memory it will take to hold 57,000 images in memory (it is a lot!). You are also holding them twice: once in the features list and then trying again in npfeatures. This second part will make a full copy of the entire features list. Hence why it runs out of memory there.

Here are some starting steps that should help you understand the limitations of your approach and perhaps get a working method:

1. You don't need this line: x = np.expand_dims(x, axis=0):

They do that in the Keras documentation, because you need a single image to have an additional dimension for the batch size. You do not need that because you append many images to a list (features), and the length of that list is the same thing, the batch size (the number of images).

2. Do your own scaling

In the case f VGG16 (and 19 I believe), the preprocess_input method simply scales pixel values between -1 and +1. You can probably do this a little more light-weight yourself. You can do the scaling on a numpy array like this:

x = (x / 127.5 - 1)

... and now remove the line with preprocess_input

3. Break it down into smaller pieces:

You can then either feed them directly into you models (if that is possible), or save the npfeatures to disk, one chunk at a time.

4. Try seeing how much memory your machine has while running this script:

Add a simple line after features.append(x), like this:

print("Loaded {} images"format(len(features)))

If you are running this on a Linux machine or Mac OSX, try using a tool like htop in terminal. There is also system monitor... same for Windows. You should be able to see the memory consumption grow until the point your script crashes. Now you know how many images you can do in one cycle - the last printed number before the crash.


How to read Several JSON files to a dataframe in R?

First, you can use the full.names parameter to list.files() to get
the full path added to each file.

temp <- list.files(path, pattern="*.json", full.names=TRUE)

Next, there are issues with the data since they contain NULL values
which throws off a quick-and-dirty solution. So, we have to take each
list element and convert any NULL to NA.

Finally, we can use the handy purrr::map_df() to take the whole list
of lists and turn them into a data.frame:

movies <- purrr::map_df(temp, function(x) {
  purrr::map(jsonlite::fromJSON(x), function(y) ifelse(is.null(y), NA, y))
})


What regression to use to calculate the result of election in a multiparty system?

Robert is right, multinomial logistic regression is the best tool to use. Although you would need to have a integer value representing the party as the dependent variable, for example:

1= Conservative majority, 2= Labour majority, 3= Liberal majority....(and so on)

You can perform this in R using the nnet package. Here is a good place to quickly run through how to use it.


Based on your explanation, $z$ is not a parameter so your function can be simplified like this:
$$f(x,y,a) = x*y*z(x)*a\text{, where } x,y \in \mathbb{Q}\cap[0,1000], a\in\{2,3\} $$
Note that $z(x)$ is a piecewise function. Also $a$ can have only two values so $f$ can be divided into $f_2$ and $f_3$, each of these having only two arguments so it simplifies things.
I don't know Tableau or Powerbi so I can't advise about that. I use R and I played a bit with your function. For a global visualization I came up with a kind of heat map like this:

In case you're interested, I did it like this with R:
library(ggplot2)

myrandomdata <- function(n) {
  a2 <- data.frame(x=runif(n,0,1000),y=runif(n,0,1000),z=20,a=rep(2,n))
  a3 <- data.frame(x=runif(n,0,1000),y=runif(n,0,1000),z=20,a=rep(3,n))
  df <- rbind(a2,a3)
  df$z[df$z<=500] <- 10
  df$z[df$z<=200] <- 5
  df$z[df$z<=100] <- 2
  df$f <- df$x *df$y * df$z * df$a
  df
}

df <- myrandomdata(50000)
ggplot(df,aes(x,y,colour=f))+geom_point()+facet_grid(a~.)+scale_color_gradient(low="blue", high="red")

(using random sampling is probably not the standard way but it was the easy way)


multi items forecasting: issue with storing results

You can store your data using pickle and then load it whenever you want.
import pickle

forecast1 = [1,2,3]
forecast2 = [4,5,6,6]
pickle.dump([forecast1, forecast2], open("forecasts.p", "wb"))

forecast1, forecast2 = pickle.load(open("forecasts.p","rb"))

Or you can directly store your pandas frames as .csv:
df.to_csv("forecasts.csv, header=True, index=False)


Only reason for such a long subset time is that you're running out of memory and the OS starts to swap. AFAIK base data.frame will create a copy of the data in memory before writing it to file.

What you can do is use either dplyr or data.table packages as they manage some of these steps without copying the data.


Random Forest Overfitting, issues with mtry=1?

While the bagging of random forests is meant to reduce overfitting, they generally will overfit more than e.g. a parametric model like logistic regression.

Having a larger gap between training and testing scores is not necessarily a problem; you may still prefer the model if its testing score is higher than other models'.  In a purely statistical perfect world, the test score is all you should care about.  In real problems, there are some other things to consider: a more-overfit tree model is likely less interpretable, potentially takes longer to train and make predictions, may be more susceptible to concept drift, etc.

You should, as you have, see if reducing the model's capacity can increase your testing score (and maybe see if doing so can reduce the train-test gap even if it costs a little on test score, see above).  With min.node.size, I would suggest using a fraction of the training set size rather than the small fixed integers that are default in ranger.  Note that ranger does also have a max.depth parameter if you'd prefer to fix that directly.  Reducing mtry is also a good approach, but 1 strikes me personally as too random, and I'd worry that you'd need a ton of trees to stabilize results; using mtry in combination with a tree complexity parameter hopefully produces a higher test score without too outrageous of a train-test gap.


I have a similar use-case and a working product based on tensorflow object-detection api and pytesseract for OCR. On top of the extracted text, I perform regex for validation of the extracted information and cleaning it to meet requirements of other processes.

Steps:
1. Annotate images with some tool like labelimg.
I annotated a set of 1K images, similar to yours, with 23 different classes. The dataset is unbalanced with some classes appearing almost in every image to some classes appearing in only as few as 60. However, there are ways to ensure that this imbalance does not affect the performance of the network.
2. Choose a model from tf model zoo (I use this frcnn model) and retrain the last two layers using transfer learning.
3. Export the inference graph, perform object detection to identify the region of interest, and run OCR on the region of interest to extract the text.
I'd recommend storing the extracted data in a dictionary with class of the object as key and the extracted text as value.
4. Finally, have regex validate the text in the extracted field and perform any manipulation/transformation that is necessary.

The trained model can be deployed to production with help of tfserving. The same trained network can be deployed into a mobile app as well - look for tutorials on tensorflowlite for this.

Hope my answer helps you! I had a tough (but interesting) time gathering the knowledge required to get a production grade product that currently serves hundreds of request everyday. I would recommend reading completely all the links I have shared in this answer, and feel free for more questions. Good luck!


What is the best practice for data folder structuring?

What are the best practices you apply? What would you recommend?

My practice is "kind of structured but quite different every time", and I don't recommend it ;)

I suspect that I'm not the only one but I don't have any stats.

Thanks for the link to Cookiecutter, this looks interesting! After reading a bit about it, it looks to me like a key criterion is this: "anyone should be able to reproduce the final products with only the code in src and the data in data/raw."

Based on that I would argue that:

  You have acces to SQL databases and make a no-modification extract -> Still into Raw I guess?

Yep, because this data cannot be obtained by processing some other part of the raw data.

  Because of very large databases, you create a semi-complex SQL query as base for a feature -> Is this Raw or Interim?

Assuming this query is run on the raw sql database, the resulting data goes to interim, because it can be re-generated by running the query again.
The query itself should be stored in src/features I think.


Understanding Layers in Recurrent Neural Networks for NLP

Its not like it just understands grammar.

In LSTMs the network tries to preserve the hidden states over time. By doing this they try to learn long-term dependencies in the language and relationships between words at variable distances.

LSTM does this by using its three famous gates.

Forget gate -  Tries to remember only the important features and relationships overtime.
Input gate - Adds new information to old cell state at each time time step.
Output gate - Produces new output by taking into account old cell state and output at each time step.


How to sum all price values having in count the quantity of each one

You could always call on the pandas DataFrame's columns and work with that.

values = df['price'] * df['quantity']
sum(values)

if you want more information, I recommend https://stackoverflow.com/questions/14059094/i-want-to-multiply-two-columns-in-a-pandas-dataframe-and-add-the-result-into-a-n


A simple approach could be the following: suppose $i \in \{0,1\}^d$ is the vector you want to predict which of the $0$ entries could be $1$ and $j \in J$ the rest of the feature vectors. Take the $k$ nearest neighbors, under some suitable distance (Jaccard, Hamming, Manhattan distance). For each $0$ entry the probabilities could be the percentage of the $k$ nearest neighbors that have $1$ in the corresponding entry.

This problem has been extensively study in the collaborative filtering community. The best known example being the Netflix Prize. This blog post provides a nice explanation of this approarch for binary data.

Another, more involved, approach is matrix completion, in particular check this reference. If you are into deep learning check this.


How I can compute Matthews Correlation Coefficient?

Here are the high-level steps:

Load the .csv from disk into an array in memory.
Threshold the probabilities to convert them to predicted target values.
Call sklearn.metrics.matthews_corrcoef.


Unpopular opinion: Second quickest way to overfit (next to data-leakage) is hyper-parameter optimization.

Why? You are assuming you wont have covariate-shift, while in most of the cases you can bet on it. Hence optimising too much on train (available data) will be ruin.

The most reasonable assumption (that we have to make sure it stands) is that 60% of the data is representative of the 40% and also approximately of the future unseen data hence all of the 150 trees should catch all of the neccesarry information.


Python inserts newline by writing to csv

Check the source code of http://www.the-numbers.com/movie/budgets/all/5401

In that table, every second row is an empty <tr> tag.


How to backpropogate Convolution layer padding inputs with respect to output derivative

You have derivatives for a 16x16 image including derivatives for padding - but these are worthless. Padding is not a trainable parameter. It is also not a function of previous layers which means computing padding's derivatives is not necessary to do backpropagation for all layers. For these reasons you can just cut out the middle 14x14 matrix and pass it to the max pool layer.
To visualise it even better imagine that you have 1D convolution. Your maxpool outputs 14 values and you pad it to the left and to the right with a 0. It should look like this.

Now notice that during backprop when you have calculated your 16 derivatives to reverse concatenation you simply split your matrix to 1-14-1 and pass 14 variables to maxpool. You can pass the leftover derivatives to padding if for example they're trainable parameters but it usually isn't the case.
That's why you don't have to calculate padding derivatives at all.


Queries regarding feature importance for categorical features

From what I understand, you have 185 features and out of them you want to choose the 80 most informative/ important?

It really seems like the main focus of all algorithms for decision trees, which is how to
decide which attribute is the best for a split? I suggest you do some reading in the context of decision trees. For example: DECISION TREES - Presentation

What you are looking for is an Impurity-Measure which can help determine the Goodness-of-Split due to some discrete attribute (basically measuring information gain):

There are 3 measures which are used most commonly - Gini index, miss-classification error and entropy.


How to decide the number of primary components for PCA

There are two ways to perform the PCA:

Compute the eigenvalue decomposition of the covariance matrix $\Sigma$
Compute the singular value decomposition of the data matrix $X$

Numerically, you can do both by calling svd() on either of them, as for positive semi-definite matrices (like $\Sigma$) svd() gives you the eigenvalue decomposition.

There is a difference, though, when it comes to interpreting the result:

The singular values in s are the eigenvalues of $\Sigma$, i.e. the variances along the PCs
The singular values in s are the singular values of $X$, i.e. the square roots of the variances along the PCs

In sklearn they go with method 2. Hence they need to square the singular values to compute the coverage. In coursera they go with method 1 so no need to square s. In the slide you show and in the video you linked they just sum the values up.

Without having run your code, my guess would be that if you change the line

calculated_coverages = ((s ** 2) / (len(s) -1)).cumsum()

to

calculated_coverages = (s / (len(s) -1)).cumsum()

you will get better results.

Addendum: On second thought I'm not sure how StandardScaler() impacts the results of the PCA either. When comparing, make sure that it is applied in both your implementation of the PCA and the implementation provided by sklearn (and maybe leave a comment if this mattered or not, pretty please ;)).


Change number format of confusion matrix

Add fmt = ".1f" parameter in sns.heatmap().

You can rewrite your code as follows to get all numbers in scientific format.

sns.heatmap(cm,annot=True, fmt=".1f")

Refer this link for additional customization.


bokeh library internally uses _glyph_function function to plot, if you take a look at their source code and which takes help from basic numpy, scipy library for defining arrays and other stuff and this so goes for curve smoothing too. So better do achieve this via scipy.


The fact that pieces of data arrive one by one does not define the minibatch size. You can have a buffer of size N (with either FIFO or with any other eviction policy that suits the statistical properties needed) and sample minibatches of size M out it every time you want to update the autoencoder.

Depending on the buffer eviction policy and the sampling strategy, this may also help to avoid local autocorrelation.

Note: this answer was originally a comment to the OP's question.


Long term time series forecasts with small dataset

You can use reinforcement learning, in instance mbrrl2 package.

As temperature is very stational dependent variable. You should have at least 1 year, better 2 years, to make some predictions.

Other way to solve this, is having a collection of historical temperatures, clustering them, and matching your dataset with the closest cluster.


We can do LDA via the lda function from the MASS package. The reduced dimension data can be computed as follows:

# Model Discriminant Analysis
library( MASS )
model = lda( class ~ ., data = X_train )
# Ploting LDA Model
projected_data = as.matrix( X_train[, 1:18] ) %*% model$scaling

You can then feed the projected_data into another supervised learning method.

Credit: The code is obtained from here.


How to perform node classification using Graph Neural Networks

I'm not sure if that's what you need but this library seems pretty similar to your problem: https://github.com/Accenture/AmpliGraph/


how to create new columns in pandas using some rows of existing columns?

Instead of perhaps iterating of each row and filling the gaps as required, I would suggest trying to do it via indexing. The solution is:

df['category'] = df.where(~df.id.isnull())['item'].ffill()

Here I break down my solution to help you understand why it works.

Imagine your dataframe is called df. I created a small version of yours as follows:

In [1]: import pandas as pd

In [2]: df = pd.DataFrame.from_dict(
            {'id':   [1, None, None, 2, None, None, 3, None, None],
             'item': ['CAPITAL FUND', 'A', 'B', 'BORROWINGS', 'A', 'B', 'DEPOSITS', 'A', 'B']})

In [3]: df       # see what it looks like

Out[3]:

 id          item
0  1.0  CAPITAL FUND
1  NaN             A
2  NaN             B
3  2.0    BORROWINGS
4  NaN             A
5  NaN             B
6  3.0      DEPOSITS
7  NaN             A
8  NaN             B

I get the dataframe back where the id column is not null (~ reverses the isnull()).  On the resulting dataframe, I take only the item column (using [item]) and then fill the missing gaps, using the previous valid value in that column.

In [4]: df['category'] = df.where(~df.id.isnull())['item'].ffill()

In [5]: df
Out[5]:
    id          item      category
0  1.0  CAPITAL FUND  CAPITAL FUND
1  NaN             A  CAPITAL FUND
2  NaN             B  CAPITAL FUND
3  2.0    BORROWINGS    BORROWINGS
4  NaN             A    BORROWINGS
5  NaN             B    BORROWINGS
6  3.0      DEPOSITS      DEPOSITS
7  NaN             A      DEPOSITS
8  NaN             B      DEPOSITS

The trick is to understand this part: df.where(~df.id.isnull())['item']

It returns really the whole dataframe, with the values where ~df.id.isnull() is True. Then only the item dataframe. The result is this:

In [6]: df.where(~df.id.isnull())['item']
Out[6]:
0    CAPITAL FUND
1             NaN
2             NaN
3      BORROWINGS
4             NaN
5             NaN
6        DEPOSITS
7             NaN
8             NaN

Now it should be clear why the final .ffill() works as we would like. It forward fills the missing values, using the last known valid value.


Remove indiferent respondents in survey data

I'd recommend converting your responses to the Likert Scale and going from there.

It's a widely researched area so there should be plenty of research on how to best deal with 'so-so' responses.


How do I validate this Kalman model for estimation of undocumeted covid cases?

To summarize, the notebook describes three key components:

The state, which completely describes the system at any given time. A SEIR model by definition has four variables in the state: susceptible, exposed, infected, and reported.
The dynamics, which describe how the states evolve over time.
The measured parameters for the dynamic equations: $\beta$, $\mu$, $\theta$, $Z$, $\alpha$, and $D$. Note, these parameters are (for our purposes) constant.

The tutorial walks you through measuring the parameters. The logic is:

You know your state for 30 consecutive days
You know that one day progresses to the next through the dynamics equations
So, you can solve for the parameters that maximize the accuracy of your model

Now that you have the parameters, your goal is to make predictions for the future. This is "just" a matter of plugging your parameters into the dynamics equations and doing the math.
Now solving six coupled differential equations is rarely easy, but they've done much of the heavy lifting for you. In particular, the function transition_fn is pretty much exactly what you want: it takes the current state, current time, and some "extra" information (defined in the tutorial) and produces the prediction for the next state.
So I think what you need to do is:

Figure out how to call transition_fn. You'll have to define state_params and extra in exactly the right format. The parameters you calculated in the tutorial will become part of extra
Start by using the same time period that the tutorial uses. This is "cheating" a bit, since you used the data to derive your parameters and now you're using the parameters to predict the data. But this is a good "sanity check": your predictions should definitely be accurate here, so you'll be able to identify any problems.
Then, use a different time period to assess whether the model is useful.


Studying machine learning algorithms: depth of understanding vs. number of algorithms

I would recommend limiting yourself to a few tried and trusted algorithms. I would not recommend Elements of statistical learning ( as a first book).  It is too theoretical, aimed  at graduate students, with exercises asking how to prove X or Y...  I think ISL is more appropriate, with more practical advice ( in any case both books are free as pdf downloads).

Besides statistics, I would make sure you are comfortable with experimental design/AB tests, and with Business Intelligence/Visualisation.


There are a lot of things one could do with that information.

Lat. and long. are the most precise data one can get on location, but that data itself may not say much about anything, really.

When I read this:

  predict the location of a storm at a given time

the first things that come into my mind are: what would define each coordinate point? You could get terrain height, closeness to some geographic accident... Things that would actually impact on the weather, more precisely, storms.

This is all intuitive. I'm not sure this can help in any way, but hopefully, it will at least help to think of something actually useful.


To get location based tweets, you have to specify a location circle with center (lat and long) and radius using reverse_geocode.

There is no way to find tweets by setting a polygon or drawing a border.


Loss of MSE always be 0 when keras for topic predict

First, is your output a one-hot vector of predicted classes? i.e: class one is [1, 0, 0, ...] and class two is [0, 1, 0, 0, ...].

If so, then using softmax activation at the output layer is acceptable and you are doing a classification problem. If you are doing a classification problem (one-hot output) you cannot use MSE as the loss, use categorical cross-entropy.

Softmax scales the output so that the number given is a predicted probability of a certain class. Wikipedia here: https://en.wikipedia.org/wiki/Softmax_function

If you are expecting the output vector to be real numbers then you need to use linear activation on your output neurons.


Word2Vec how to treat words that aren't in the vocabulary

The words Going, Gone, Goes are considered to be similar in only one context i.e. they all have the same root word Go. This is known as Stemming/Lemmatization, both of which are special cases of normalization process. So, in order to assign some value to these words, you have to first lemmatize them. In order to get more details about this, I am attaching two links which can be helpful in your case:

http://textminingonline.com/dive-into-nltk-part-iv-stemming-and-lemmatization

https://stackoverflow.com/questions/771918/how-do-i-do-word-stemming-or-lemmatization


How should I expose data from my app for data scientists?

Usually, people are indeed quite indifferent to the format, as long as there is an easy way in which they can transfer it to their favorite format.

CSV is a very common format, since most tools can load it.
Note that there are some dataset whose CSV representation is inconvenient (e.g., the text has many commas, data set in which the type is importnat yet hard to deduce).

As for the storage of the format most people are even more indifferent to the source too. Web interface is the common option.

If you use a relational database you gain few advantages:

The dataset is structured so you are protected from wrong structuring problems.
The dataset can be naturally updated (e.g., you can keep appending records to it on a daily basis).
You can allow the users load only part of the dataset, which is convenient with large dataset (e.g., give me just America users from the last month).
In case that you are willing to enable that, you can let the users work on your database directly (e.g., as Google Big Query host github data)

To summarize, the location of the information is not important.
Unless you need one of the advantages provided by a database, use a CSV.

Need a database:

Online datasource like Amazon Redshift
Some other RDBMS available online (e.g. a dedicated postgres installation)
Big query

The raw data is enough

CSV files available on S3
CSV files available for download in a web interface
Dumps into Google sheets

General true

Doesn't matter as decent data scientists can easily handle and automate anything (anything reasonable but yet, it is nice that you are looking for the most convenient format).


How to subset rows from a data frame with comparison operators in R

You want

data_frame[data_frame$X1 > 30, c("X1","X2","X4")]

that will just print it, you probably want to update data_frame or store it in something else:

data_frame = data_frame[data_frame$X1 > 30, c("X1","X2","X4")]

also you probably want to try asking this on StackOverflow, or reading a bit more basic R documentation because it should be well covered. Its a bit simple to be "data science".


CNN 1D for time series. Value Error. I did read all the possible answers in StackOverflow regardiing this problem

Your first error is because Keras wants the batch size as first dimension. Given that your Y has 95751 observations, your X must be reshaped in order to have 95751 records in the first dimension, as you did in the second part.

The code doesn't work because Keras has already put the first dimension (None = batch_size) for you (see model.summary()), so you don't need to specify that in input_shape. You need to change this

input_shape=(95751, 4)

into this

input_shape=(4, 1)

Update after comment - working example

Here's a working example of your code.

First I define X and Y following your shapes. Here I use random numbers given that I don't have your data

data1cnn = np.random.rand(95751, 4, 1)
y = np.random.rand(95751,)

I check the shapes to be sure that they are the same as the ones you have provided in your second part of the question

print('X and Y shape')
print(data1cnn.shape, y.shape)

Output

X and Y shape
(95751, 4, 1) (95751,)

Looks correct. Now I define the model: everything is the same as in your code but input_shape=(4, 1)

modelcnn = Sequential()
modelcnn.add(Conv1D(filters=64, kernel_size=2, activation='relu',
input_shape=(4, 1)))
modelcnn.add(MaxPooling1D(pool_size=2))
modelcnn.add(Flatten())
modelcnn.add(Dense(50, activation='relu'))
modelcnn.add(Dense(1))
modelcnn.compile(optimizer='adam', loss='mse')

I print the summary

print('Model summary')
print(modelcnn.summary())

Output

Model summary
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_1 (Conv1D)           (None, 3, 64)             192
_________________________________________________________________
max_pooling1d_1 (MaxPooling (None, 1, 64)             0
_________________________________________________________________
flatten_1 (Flatten)         (None, 64)                0
_________________________________________________________________
dense_1 (Dense)             (None, 50)                3250
_________________________________________________________________
dense_2 (Dense)             (None, 1)                 51
=================================================================
Total params: 3,493
Trainable params: 3,493
Non-trainable params: 0

Then I train it using the same command you wrote in your question but verbose=1 to print some information during training

modelcnn.fit(data1cnn, y, epochs=1000, verbose=1)

and it trains without a problem given that I get as output

Epoch 1/1000
95751/95751 [==============================] - 6s 60us/step - loss: 0.0844
Epoch 2/1000
95751/95751 [==============================] - 7s 76us/step - loss: 0.0833
Epoch 3/1000
95751/95751 [==============================] - 6s 59us/step - loss: 0.0832
Epoch 4/1000
95751/95751 [==============================] - 6s 66us/step - loss: 0.0831
Epoch 5/1000
95751/95751 [==============================] - 5s 54us/step - loss: 0.0831
Epoch 6/1000
95751/95751 [==============================] - 5s 54us/step - loss: 0.0830


Random Forest variable Importance Z Score

Please allow me to cite the site of creators of the RandomForest algorithm
Out of bag error

In random forests, there is no need for cross-validation or a separate test set to get an unbiased estimate of the test set error. It is estimated internally, during the run, as follows:
Each tree is constructed using a different bootstrap sample from the original data. About one-third of the cases are left out of the bootstrap sample and not used in the construction of the kth tree.
Put each case left out in the construction of the kth tree down the kth tree to get a classification. In this way, a test set classification is obtained for each case in about one-third of the trees. At the end of the run, take j to be the class that got most of the votes every time case n was oob. The proportion of times that j is not equal to the true class of n averaged over all cases is the oob error estimate. This has proven to be unbiased in many tests.

Variable importance and Z-score

In every tree grown in the forest, put down the oob cases and count the number of votes cast for the correct class. Now randomly permute the values of variable m in the oob cases and put these cases down the tree. Subtract the number of votes for the correct class in the variable-m-permuted oob data from the number of votes for the correct class in the untouched oob data. The average of this number over all trees in the forest is the raw importance score for variable m.
If the values of this score from tree to tree are independent, then the standard error can be computed by a standard computation. The correlations of these scores between trees have been computed for a number of data sets and proved to be quite low, therefore we compute standard errors in the classical way, divide the raw score by its standard error to get a z-score, ands assign a significance level to the z-score assuming normality.


How to turn a series that contains pandas.datetime.timedelta to floats e.g. hours or days?

One way to do this would be to use Series.dt.seconds and ´Series.dt.days´ and  multiply with a factor for the desired unit:

(Series.dt.seconds/3600) + (Series.dt.days*24)  # for values with [hours]


Product classification in e-commerce using attribute keywords

The problem type you're dealing with is referred to as multiclass classification. Not all algorithms are suited to handle it, but tree based methods and neural networks are popular choices. If you need it to run quickly and probability calibration isn't too important, Naive Bayes also works quite well for some data sets. To see an example of a dataset of this type, check of the Kaggle Spooky Author Identification competition. The published kernels give some good examples of feature engineering and modeling choices.

(I'm assuming the category label is unique. If there can be more than one label per record, it's called multilabel classification, which I would handle by building a separate binary model for each of the 39 labels in your data set. For an example dataset, check out the Kaggle Toxic Comments competition.)

As for modeling with your data in specific, the structure you have it in now seems a bit odd to feed to a model. A training data set should have each row represent one record, and each column represent a feature with the value in the column describing the record, whereas in your format, any cell phone should have information regarding each value in the corresponding mobile Attribute_Names. When you get the raw input 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable Upto 256 GB 5.5 inch Full HD Display', how is that transformed into a format that can be fed into the model?

Also, where are the Attribute_Names, Attribute_Values coming from, are they manually specified? If so, that is limiting the performance potential from a model since there could be additional words in the data the model could detect if left to generate features on it's own. For a good modeling flow, the training data should contain the inputted raw text, then process the text to generate features, then feed into the actual model to output a label. So one row of raw training data set would be:

------------------------------------------------------------------------
| Description                                            |   Category  |
------------------------------------------------------------------------
| 'Samsung Galaxy On Nxt 3 GB RAM 16 GB ROM Expandable   | Mobile      |
| Upto 256 GB 5.5 inch Full HD Display'                  |             |
------------------------------------------------------------------------

You would then turn the text into features using some sklearn pre-processors. Count Vectorizer or TF-IDF Vectorizer are popular choices. You can also create your own features from keywords from the Attribute key and values you already have, such as creating a Brand indicator column if the words Samsung, Nokia or OnePlus appear in the text, but I would never use only manually specified features when modeling with text features.


Choosing the periodicity in a SARIMA model

The choice of periodicity in a ARIMA model is depending on the seasonality in your time-series data. You should plot your time-series and see if you have a "curve"-pattern that occurs regulary.

Based on the "distance" (hours in your data set) you can make a guess what the right choice of periodicity is.


Cross validation schema for imbalanced dataset

I'm not sure if there's a question here, but I'll add some comments.
Firstly, if you can get it in the wild, always work with balanced data. However, if you are going to manually create a "balanced" data set yourself, make sure that the selection criteria that you use to create that data is appropriate. As an example, choosing the 100k most recent positive and negative outcomes may not be appropriate because the time frame of the positive outcomes may extend well beyond that of the more common negative outcomes. So in this 200k data set, your negative 100k outcomes may relate to data from the last year while the data relating to your 100k positive outcomes may relate to the last ten years.
Secondly, if you are going to balance your data be aware of how the balancing technique works and try to understand its weaknesses / limitations. Be mindful that rebalancing a data set will result in a new data set, and remember that you will have to check that the new data set is still appropriate to use. As an example, you will need to check that the distribution of input variables is still roughly the same as before. Applying this thinking to your options above, can you be certain that the data in each fold will be roughly similar?
Lastly, if you are going to use a modelling framework which can handle imbalanced data then make sure you understand why it can handle the imbalanced data. In particular, if the framework applies some weighting / balancing technique in the background you should be aware of this and be able to explain it.


TSFRESH is already supporting Time Series Forecast.
See details and example here and here


Setting "missing" distance values to zero when training a neural network

The short answer is "it depends"

Is zero better than one or infinity?  It depends on the range of x-y coordinates that you use.  It also depends on your output.  If you go from (x,y) to something like z = x^2, then you're placing your "null" at a local minimum. So, if it's a car, I'm assuming you're using a forward-facing camera.  Do you really want your algorithm to see a bunch of "cars" bunched up at the origin?
By the way, where is your origin?  Upper left (like most image indexing)? Lower left (like a cartesian plane)? Center (if so, what direction is positive x and y?)

As for re indexing, what you can do is look at the previous frame (or frames) where your 32 objects are indexed, along with their position and velocity.  Then, look at the current frame and compare each object's position to the positions in the previous frame.  Assign the objects in the current frame the same id as the object nearest it in the previous frame.  You can break ties with velocity.

following up on your comments:

a neural network can learn any arbitrary function to any arbitrary precision.  Eventually.  If you want certain behavior at (0,0) that is dramatically different from behavior at (0+ε,0+ε), then your network will take a long time to converge.

I suggest seeing what kind of results you get by converting nulls to zero and then comparing it to other techniques.  Maybe replace null with the average of all the other cars, or set it immediately behind your ego vehicle (since cars primarily move forward, your network will probably put less importance on cars behind the ego car).


How to deal with ternary Output neurons in the Output classification layer of a simple feedforward Neural Net?

The typical approach for this kind of scenario is to handle the output as a discrete element and have each output be a probability distribution over the discrete output space, that is, for each output you generate 3 numbers between 0 and 1 where the sum of them adds up to 1. You normally obtain this by generating for each position a vector of 3 elements in $\mathbb{R}$ instead of a single element and then passing it through a softmax function.
The typical loss function for that is the categorical cross-entropy, which receives the probability vector (although depending on the implementation and the parameters, it may receive the vector before the softmax, which are called the "logits"), and the index of the expected discrete output.
This is the usual approach for multi-class classification that you will find in most tutorials, like this one.


Separate business emails from personal emails

Thank you for clarifying your question.
So, from what I have got, you want to predict user demographics from the e-mails (i.e. the text).
If you knew some of the demographics for a same group of emails, you could do a supervised task by:

Using a Recurrent Neural Network (RNN) or LSTM (https://towardsdatascience.com/understanding-rnn-and-lstm-f7cdf6dfc14e), you can encode the emails by feeding each email into the network as sequence of tokens. From this we get a "hidden representation". From this "hidden representation", we then decode this with a standard neural network, which then outputs demographics (whether it be age, socioeconomic status, etc.)

If you don't know demographics, then you could build customer profiles by agglomerative clustering:
Here, you use something like Doc2Vec (https://medium.com/wisio/a-gentle-introduction-to-doc2vec-db3e8c0cce5e) to encode the emails into vectors in semantic space and then use agglomerative clustering (https://www.datanovia.com/en/lessons/agglomerative-hierarchical-clustering/) over these vectors to find clusters of users.
The above is an unsupervised approach and you can also decide after looking at the resulting dendrogram, how many customer profiles you think are necessary.


Ambiguity in Perceptron loss function (C. Bishop vs F. Rosenblatt)

I managed to find the  Bishop's version by unearthing! the Rosenblatt's 1962 Principles of neurodynamics, Page 110 book, so the Wikipedia's version must be the alternative one.

It is worth noting that the book also has a chapter on error back-propagation (page 292), which resembles the Wikipedia version, but I think it is not exactly the same.


What method is recommended after outliers removal?

based on this information I would recommend to use

Wilcoxon signed-rank test

or

paired Student's t test

This depends on your sample size and distribution. To test if your data set is normally distributed you can use the Jarque-Bera test.

I didn't work with Matlab yet, but  I guess all the tests should be implemented in Matlab.

From that point you could evaluate the impact of the condition on response time.
Visualize your data (X on Y) - Scatter plot / Heatmap (if multivariate data).

From here you could start building a model

starting with linear regression
going into more advanced models/approaches -- random forests, bootstrapping etc.

Added

If you only have binary variables (condition / no condition) - you need to set up dummy variables to perform a regression model (1 / 0) . Data then could look like:

 obs  response-time   any_condition  condition_1   condition_2   condition_3
  1   12.54           1              1             0             0
  2   19.34           0              0             0             0
  3   13.32           1              1             1             0
  4   14.7            0              0             0             0

If you have multiple different "state" conditions, I would recommend to use a control variable ("any_condition") - to see if conditions (not specifying which, have an impact) on the response time.

I hope that helps.


A confusion matrix gives you the following:

[TP, FP]
[FN, TN]

where TP = 'true positives'; FP = 'false positives'; FN = 'false negatives'; TN = 'true negatives'.

You can read more here: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

By taking TP+TN and dividing by TP+FP+FN+TN, you can get the classification accuracy of your model. In your case, that means (9779+148)/(9779+107+2227+148) = about 81%

More details:

This type of confusion matrix is used for binary classification.

A 'true positive' in this case means a true instance of class 0; that is, the model predicts that a given example belongs to class 0 and it really does belong to class 0.
A 'true negative' means a true instance of class 1.
A 'false positive' means the model predicts that the example belongs to class 0 when it really belongs to class 1;
A 'false negative' means the model predicts that the example belongs to class 1 when it really belongs to class 0.


There are very few resources that justify number of cells proportional to input. The intuition though is clear from colah's blog. The longer the sequence you want to model, the more number of cells you need to have in your layer. For e.g. if you are using the LSTM to model time series data with a window of 100 data points then using just 10 cells might not be optimal. The goal of any RNN (LSTM/GRU) is to be able to encode the entire sequence into a final hidden state which it can then pass on to the next layer. So even if you look at the older NMT models (sequence to seq for language translation), at a very basic level, it had a layer of an lstm encoding all info from the sequence that was passed to it and then using it to kick start the decoding. So basically, if the number of units are relatively lesser, the ability to encode all the info MIGHT not be optimal.

Having said that, we regularly use

a) multiples of 32 (https://svail.github.io/rnn_perf/) so inspite of their explanation being vague at best, the way to look at it is that when u declare an output size of say, 100, the RNN will generate square matrices of 100 x 100 with weights in them (that will be adjusted during back prop to give you a final model) and that matrix multiplications of such a matrix will be unwieldy as opposed to a matrix thats a multiple of 32 ( this is totally my intuition again, please correct, if im mistaken)

b) also if you use more than a certain number of hidden units, you will end up with the vanishing gradient problem (exploding gradients typically dont occur due to relu activation functions that keep activations between 0 and 1). Hence you are better off going "deeper" than "wider" , where you can stack 2-3 layers of 128 hidden cells "vertically", instead of using 512 hidden cells in one layer (this , i have actually experienced in almost all forms of experiments with text as well as image data)

im hoping this clarified matters somewhat


Matrix based Visualization Meaning - Assocation Rules

An association rule is (usually) of the format X -> Y, meaning that if X happens then Y is likely to happen. In the traditional example of supermarket, a possible rule would be something like Bread -> Butter. In this case, X is called the antecedent, or LHS (left-hand side), and Y is the consequent, or RHS.

This specific chart has the number of items in the LHS as it's horizontal axis, and the number of items in the RHS as the vertical axis. The colors represents the metrics of support and confidence, as can be seen in the legend.

Example information that can be read from this chart is that the support is higher for lower number of items in the LHS, because the left side of the chart looks red-ish and the right side looks blue-ish, matching our intuition (more items in the LHS is more restrictive and thus less likely to happen). The confidence doesn't seem to be related to any of the axes in particular.


Regarding 1) Usually you would check this based on "tests" like the VIF (see this post for details). I personally would not be too worried with a correlation of 0.5.
Regarding 2) Not really since you would need to "decorrelate" the variables and you can only do linear transformations (when you want to preserve the variable(s)). What you can do is to make projections, e.g. by principle components, but this will totally change the variable set.


XGBoost Classification Probabilities higher than RF or SVM?

Rather than answering why XGBoost give very confident predictions, I will answer why random forest and SVM give not-so-confident predictions.

Random forest probability estimates are given by the percentage of the forest that predicted a particular class. For example, if you have $100$ trees in your forest and $81$ of them predict some class for some example, the probability estimate for that example belonging to that class is calculated to be $\frac{81}{100} = 0.81$. Because of the random nature of the ensemble members, it's very unlikely that each individual tree will end up with the correct prediction, even if the majority do. This makes probability estimates from random forests shy away from the extreme ends of the scale.

SVM is a slightly different case, because they are unable to produce probability estimates directly. Typically, Platt scaling (essentially logistic regression) is used to scale the SVM output to a probability estimate. This has the added benefit of calibrating the probability estimates, meaning the predicted probability is quite accurate - in other words, if a probability of $0.8$ is given for a prediction, it actually has approximately an $80\%$ chance of being correct. For a problem like this where there's a lot of noise (underdog teams do win sometimes, and there's a lot of evenly matched games that are hard to predict), these predictions will tend not to be overconfident.

I don't have a good reason as to why XGBoost is possibly overconfident, but it has been observed in the past that additive boosting models tend to provide distorted probability estimates without applying post-training calibration e.g. here, here & here.


With your current model, you will indeed get a single scalar output. This is because the last layer is Dense with just 1 unit.

In other words, with this model you can map a sequence to a single value that may represent, e.g., the label of the series or its next value depending on what target you train the network on.


Is parsing files an application of machine learning?

I would answer the question at two levels. The first level is "can it be done using machine learning?" I would say that machine learning is essentially about learning. So given that you prepare sufficient examples of sample documents and the output to expect from those documents, you can train a network to learn the structure of documents and extract the relevant information. The more general form of extracting information from documents is a well-researched problem and is more commonly known as Information Retrieval. And it is not limited to just machine learning techniques, you can use Natural Language Processing tools as well. So, in its general form, it is actually being done in practice. Coming to the second level, "should you be doing it using machine learning?". I would agree to what @NeilSlater said. The better and more feasible approach would be to use good programming practices so that you can reuse parts of your parser as your dataset evolves.


Decision Trees are definitely easier to overfit than Random Forests. The averaging effect (see bagging) is meant to combat overfitting.

Other than that I think the default parameters will overfit.

Example:

from sklearn.tree import DecisionTreeRegressor

# Create a dataset
x = np.linspace(0, 10 * np.pi, 50).reshape(-1,1)
y = x + 3 * np.sin(x)
noise = np.random.random(50).reshape(-1,1)
noise -= noise.mean() # center noise at 0
noisy = y + noise * 2

# Define a Decision Tree (with default parameters)
dtr = DecisionTreeRegressor()
dtr.fit(x, noisy)
y_dtr = dtr.predict(x)

# Draw the two plots
plt.figure(figsize=(14, 4))
ax1 = plt.subplot(121)
ax1.plot(np.linspace(0, 10 * np.pi, 100),
         np.linspace(0, 10 * np.pi, 100) + 3 * np.sin(np.linspace(0, 10 * np.pi, 100)),
        color='gray', label='desired fit', zorder=-1, alpha=0.5)
ax1.plot(x, y_dtr, color='#ff7f0e', label='decision tree', zorder=-1)
ax1.scatter(x, noisy, label='data')
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_title('Model Overfit')
ax1.spines['right'].set_visible(False)
ax1.spines['top'].set_visible(False)
ax1.yaxis.set_ticks_position('left')
ax1.xaxis.set_ticks_position('bottom')
ax1.legend()

ax2 = plt.subplot(122)
ax2.plot(np.linspace(0, 10 * np.pi, 100),
        np.linspace(0, 10 * np.pi, 100) + 3 * np.sin(np.linspace(0, 10 * np.pi, 100)),
        color='gray', label='desired fit', zorder=-1, alpha=0.5)
ax2.plot(x, y_dtr, color='#ff7f0e', label='decision tree', zorder=-1)
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('Same graph')
ax2.spines['right'].set_visible(False)
ax2.spines['top'].set_visible(False)
ax2.yaxis.set_ticks_position('left')
ax2.xaxis.set_ticks_position('bottom')

ax2.legend()

Running the code below will produce the following figure:


What is the possible range of SVR parameters range?

I support vector regression the inverse regularization parameter $C$ can be selected from the interval $[0,\infty)$. In which $C=0$ means that we are very heavily regularizing and $C\to \infty$ no regularization.

The parameter $\varepsilon$ is also from the interval $[0,\infty)$. In which $\varepsilon=0$ forces the regression to penalize every point that is not exactly on the regression line. Whereas $\varepsilon > 0$ allows an indifference margin around the regression in which a deviation will not be counted as an error.

Additionally, there are slack variables of $\xi\geq 0$ and $\hat{\xi}\geq 0$. These are zero if a point is inside the indifference margin. If a data point lies above and outside the indifference margin we will have $\xi>0$ and if a data point lies below and outside the indifference margin we will have $\hat{\xi}<0$.

I think you mean the form parameter of your radial basis function when you talk about $\gamma$. If we have

$$\varphi(\boldsymbol{x}_i,\boldsymbol{x}_j|\gamma)=\exp\left[-\gamma||\boldsymbol{x}_i-\boldsymbol{x}_j||^2\right]$$

then $\gamma \in (0,\infty)$ (Note the minus sign in front of $\gamma$). For $\gamma \to 0$ we make the kernel flatter as $\varphi \approx 1$. If $\gamma \to \infty$ we will get a very peaked kernel. Which will be 1 when $\boldsymbol{x}_i\approx \boldsymbol{x}_i$ and almost zero everywhere else.

You should also have a look at the documentation for the implementation of these parameters. It might happen that these parameters are not implemented as you think (see note on $\gamma$).


Neural networks with non-negative weights

One possible method to approach neural networks with non-negative weights is using Feedforward neural network. We can construct the neural network using optimization techniques with the constraint of non-negative weights, as opposed to the normal back propagation method.

A modified Matlab example taken for here is as follows:

load iris_dataset

% Number of neurons
n = 4;

% Number of attributes and number of classifications
[n_attr, ~]  = size(irisInputs);
[n_class, ~] = size(irisTargets);

% Initialize neural network
net = feedforwardnet(n);

% Configure the neural network for this dataset
net = configure(net, irisInputs, irisTargets); %view(net);

fun = @(w) mse_test(w, net, irisInputs, irisTargets);

% Add 'Display' option to display result of iterations
ps_opts = psoptimset ( 'CompletePoll', 'off', 'Display', 'iter', 'MaxIter', 100); %, 'TimeLimit', 120 );

% There is n_attr attributes in dataset, and there are n neurons so there
% are total of n_attr*n input weights (uniform weight)
initial_il_weights = ones(1, n_attr*n)/(n_attr*n);
% There are n bias values, one for each neuron (random)
initial_il_bias    = rand(1, n);
% There is n_class output, so there are total of n_class*n output weights
% (uniform weight)
initial_ol_weights = ones(1, n_class*n)/(n_class*n);
% There are n_class bias values, one for each output neuron (random)
initial_ol_bias    = rand(1, n_class);
% starting values
starting_values = [initial_il_weights, initial_il_bias, ...
                   initial_ol_weights, initial_ol_bias];

% alter the patternsearch function with the appropriate constraints, in your case we would change it so that the lower bounds of the weights are zero
[x, fval, flag, output] = patternsearch(fun, starting_values, [], [],[],[], zeros(size(starting_values)), 1e10, ps_opts);

where the mse_test.m function is as follows:

function mse_calc = mse_test(x, net, inputs, targets)
net = setwb(net, x');
y = net(inputs);
[row col] = size(y);
mse_calc = sum(sum((y - targets).^2))/(row * col);
end


Normalized Euclidean Distance versus cross correlation?

These two metrics are not the same.

The normalized Euclidean distance is the distance between two normalized vectors that have been normalized to length one.  If the vectors are identical then the distance is 0, if the vectors point in opposite directions the distance is 2, and if the vectors are orthogonal (perpendicular) the distance is sqrt(2).  It is a positive definite scalar value between 0 and 2.

The normalized cross-correlation is the dot product between the two normalized vectors.  If the vectors are identical, then the correlation is 1, if the vectors point in opposite directions the correlation is -1, and if the vectors are orthogonal (perpendicular) the correlation is 0.  It is a scalar value between -1 and 1.

This all comes with the understanding that in time-series analysis the cross-correlation is a measure of similarity of two series as a function of the lag of one relative to the other.


Is it bad to have a lot of one class of Data [K-NN classifier]?

Just imagine it practically, If class A data is 90% and class B data is 10% ,then if you just randomly classify the label you prediction as class A then your accuracy will be 90%.
So biased data will lead your model to be biased over the class which has more data as it will give better predictions in your model.


After sorting, try:

df = df.reset_index(drop = True)


target variable prediction among possible answers

I want to have a model that, for example, predicts Joseph but does not consider it wrong because it is one of the two possible answers.

It seems that you are confusing prediction and evaluation: whether an answer is correct or not is a matter of evaluation, and the evaluation method doesn't have to follow the same rules as the predicting system.
For example the system can be designed to always predict a single name, and the evaluation measure can be based on whether the predicted name belongs to a set of correct answers or not. This would technically be simple multiclass classification, since there's a single class predicted. However you would have to implement your own evaluation method, because the standard one will consider correct only if the prediction is exactly the true class.
You could also decide that the system can predict any combination of names, that would be multi-label classification. But here again you could decide any specific evaluation method, for instance count a prediction as correct as long as it contains at least one of the true answers, or contains only names which belong to the true answer, etc.
Note that in theory any multi-label problem can be transformed into a multiclass problem simply by considering any possible set of classes as a single class (e.g. "Mike+Michael" as one class). In general it's a matter of number of values, since if every possible combination can happen in the data there would be $2^N$ classes as a result. I don't think this is relevant in your case, since this would defeat the idea of a flexible evaluation method.


Is there a dataset with news articles and their headlines?

The most widely used ones in text summarization research is the DUC dataset. If you see a paper using dataset "DUC 2015" or "DUC 2016" that's from here.

I have also personally used the Reuters arcihve. You just need to download each article with wget or something similar. See also here.

The CNN / DailyMail dataset is also widely used in summarization especially in recent years, although it labels itself as a Q&A dataset.


Multivariate time series prediction with binary target

I recently trained some multivariate LSTM regressions on a relatively slow, CPU-only laptop, and it worked. Training time didn't take too much, it's definitely feasible.

About your problem, I would definitely use some RNN architecture. LSTM are the most powerful RNN cells up to now, they have more memory and are able to learn longer sequences than GRUs, but they are slower than them. Choose based on you preference/circumstances.

The final layers must be Dense(), and the final output layer should have two nodes with softmax activation, to perform the binary classification. The most appropriate loss would then be some crossentropy measure.


Damerau-Levenshtein Edit Distance in Python

Which one is correct? (Sorry, i don't know how to properly edit the typing with my handphone, my only online device for the time being)

https://www.guyrutenberg.com/2008/12/15/damerau-levenshtein-distance-in-python/

The first one,

"""Compute the Damerau-Levenshtein distance between two given strings (s1 and s2)"""

    def damerau_levenshtein_distance(s1, s2):

        d = {}
        lenstr1 = len(s1)
        lenstr2 = len(s2)
        for i in xrange(-1,lenstr1+1):
            d[(i,-1)] = i+1
        for j in xrange(-1,lenstr2+1):
            d[(-1,j)] = j+1

        for i in xrange(lenstr1):
            for j in xrange(lenstr2):
                if s1[i] == s2[j]:
                    cost = 0
                else:
                    cost = 1
                d[(i,j)] = min(
                               d[(i-1,j)] + 1, # deletion
                               d[(i,j-1)] + 1, # insertion
                               d[(i-1,j-1)] + cost, # substitution
                              )
                if i and j and s1[i]==s2[j-1] and s1[i-1] == s2[j]:
                    d[(i,j)] = min (d[(i,j)], d[i-2,j-2] + cost) # transposition

        return d[lenstr1-1,lenstr2-1]


Grouping high dimension Y-space to lower dimensions

You may find this resource helpful : https://xang1234.github.io/multi-label/

One possibility is to cluster similar labels together so that they are
processed together by the multilabel classification algorithms.
Community detection methods such as the Louvain algorithm allow us to
cluster the label graph. This is implemented in the
NetworkXLabelGraphClusterer with the parameter methods = louvain.

A resource for splitting your data into (hopefully) balanced train,val,test sets (not so easy because you are in a multi label setting) : http://scikit.ml/stratification.html


sk-learn - ValueError: array is too big.

The problem is that you're converting a large sparse dataset into a dense array via toarray(). Don't do that :) Why do you need it in dense array format? Try posting more of your code to see if we can figure out better ways to solve the problem. With any large dataset, you want to stay sparse as long as possible and only work with smaller sets of dense data (if dense is needed at all).


Can features that are the same in every sample contribute to learning?

It has no value if it is same for all the training set.
Let's say, you are using a global health dataset for Life expectancy then country code can be a useful feature. It might contain hidden information.
But if you are doing the same analysis for one country e.g. India, keeping a feature country which has only one value e.g. India, will be of no use.
It will not show any variance, now the variance is shifted to States instead of Country compared to the last example.
In you data, I can see multiple Sensor Id. If very dataset will always have these 4 values and there is no explicit assumption for 3rd column which is dependent on 1st, you can remove the feature


sklearn.naive_bayes VS categorical variables

Hot encode the categorical variables and use Bernoulli naive Bayes. Hot encoding is usually the trick one uses in representing categorical variables.


Unable to understand the meaning of following lines of the research paper for image segmentation

Generally, if you look at image segmentation models, they have two main paths, what the author of your paper calls encoder and decoder paths.

The role of the encoder is to contract the size of the image while extracting meaningful information, while the decoder restores the contracted image to its original dimensions. However, a lot of information is lost during contraction. To overcome this issue the decoder takes some features directly from the contraction path via skip connections.

A certain decoder layer's input is actually the concatenation of two things:

The output of its previous layer
The output of the respective layer (the one with the same dimensions) of the encoder

This becomes more clear if you look at the following image:

The left part of the network is the encoder and the right is the decoder. The skip connections are the grey horizontal lines that go directly from the encoder to the decoder.

These can't appear the way you portray the model because you show only the layers and not any information about how they are connected. I think you assume the network should be sequential, but segmentation models usually aren't.


Deep Learning Binary Text Classification

In a broader sense, what you're referring to here is known as text mining. This is where information is extracted from text according to certain criteria in order to form specific patterns and meanings, etc.

How you would go about implementing this depends on the criteria you have in mind.

For instance, let's suppose you have a blog article in text format (let's call it filepath.txt for this example).

You decide that you wish to sort words into specific categories. For instance, you could categorise the keywords hotel, flights, countries, as travel, i.e. each incidence of the three words would be replaced with data. Let's assume data is a relevant category of interest. Any article that would fall under a travel category would be included in your articles of interest.

Using Python, you can then scan your file and use filedata.replace to replace your keywords with categories:

# Read file
with open('filepath.txt', 'r') as file :
filedata = file.read()

# Replace keyword
filedata = filedata.replace('How ', ' ')
filedata = filedata.replace('Why ', ' ')
filedata = filedata.replace('of ', ' ')
filedata = filedata.replace('to ', ' ')
filedata = filedata.replace('you ', ' ')
filedata = filedata.replace('all ', ' ')
filedata = filedata.replace('and ', ' ')
filedata = filedata.replace('be ', ' ')
filedata = filedata.replace(' a ', ' ')
filedata = filedata.replace(' for ', ' ')
filedata = filedata.replace(' in ', ' ')
filedata = filedata.replace(' is ', ' ')
filedata = filedata.replace(' the ', ' ')
filedata = filedata.replace(' about ', ' ')
filedata = filedata.replace(' an ', ' ')
filedata = filedata.replace('Data', ' data ')
filedata = filedata.replace('Python', ' data ')
filedata = filedata.replace('R', ' data ')
filedata = filedata.replace('machine', ' data ')
filedata = filedata.replace('Linux', ' data ')
filedata = filedata.replace('technology', ' data ')
filedata = filedata.replace('flights', 'travel')
filedata = filedata.replace('countries', 'travel')
filedata = filedata.replace('hotel', 'travel')
filedata = filedata.replace('analytics', 'data')
filedata=  filedata.replace('CNN', 'news')
filedata=  filedata.replace('weather', 'news')
filedata=  filedata.replace('Trump', 'news')
filedata=  filedata.replace('market', 'business')
filedata=  filedata.replace('entrepreneur', 'business')
filedata=  filedata.replace('financial', 'business')

# Write to file
with open('filepath2.txt', 'w') as file:
file.write(filedata)

You could then use a library such as stringr in R to see how frequently the category (which would have the keywords replaced with the category) appears:

require(stringr)
WordList <- str_split(readLines("filepath2.txt"), pattern = " ")
searchqueries<-sort(table(WordList),decreasing=TRUE)[1:100]
searchqueries

e.g. if the category appears X number of times or more, then the article is deemed relevant.

This is just one example of how you could use text mining. Again, much of it hinges on your specific criteria and how you would go about determining relevancy. But that's one way you could approach it.


How would keras model learn at every epoch?

The data points are the same, but the model is different at each epoch. Weights are updated at each iteration so they will be different and in turn, the loss function will be different for the same data point at different epochs. What training process does is to change the weights slowly at each run to minimize the loss for the same data smaller and smaller.

Weights are modified in a way to minimize the error for the data points using an algorithm such as backpropagation in gradient descent. You might also want to check the explanation of backpropagation given by 3B1B in this youtube video.