If we forget about health for a second and we look at position alone, we have 6 players, each of which could be in any of the 100 locations so our state space for position alone would be 100^6.

Yes, that is correct, adding health in, and assuming it was an integer from 1 to 20, then you would have $20^6 \times 100^6$ discrete states.

  My understanding is that you would need to enumerate all possible states for this game in order to successfully implement Q-learning. How would this be done?

For the enumerated case with enough resources, there is a theoretical guarantee that Q learning (and other RL methods) will converge on the optimal result. However, in this case - and many others - there are clearly too many states to calculate or store values for in detail. So the answer is that you don't do that. Instead you need look into ways to calculate approximate calculating action values which you hope will also generalise to previously unseen state/action pairs, at the very least to unseen states and actions that are relevant in real games.

There are many ways to structure the state and action representations to help with approximation in RL. Sutton & Barto's Reinforcement Learning: An Introduction dedicates the whole of the second (of three) sections to approximation in general, and chapter 9 part 5 reviews some popular approaches to representations that work well with simpler linear function approximation.

This kind of approximation is also the main driver behind "Deep" Q-learning (DQN), which is essentially Q-learning with a neural network approximating the action value table. There are some details to add to get this to work well - but plenty of resources available to help with this on the internet: Tutorials, examples and fully coded agents ready to set up and solve problems.

So I would recommend moving to DQN when the state/action space gets too large for a simple tabular approach, and where it is not clear what other simpler approximation could get the job done more efficiently. If you do that, then the state and action representation becomes what will work as inputs to a neural network. There are some rules to follow for that. First, all inputs should be on a suitable scale. Neural networks like to work with inputs with mean 0, standard deviation 1 over the sample population. Often you don't know that for a RL problem, so you can go to the next best thing - scaling to fit in a range such as 0..1 or -1..1

I have also written an answer covering approaches to choices in state representation and feature engineering for another problem  Reinforcement learning: easily learnable state representation which might help.


Using Classification Algorithms to Interpret Clustering Reuslts

There's nothing wrong with this idea and although I don't have literature on hand, I'm fairly confident I've seen this sort of thing done. I disagree that clustering algorithms often don't provide interpretation though. There are definitely plenty that don't, but I'm not sure k-means is one of them. The centroids of your clusters should provide you with the interpretability you're looking for. Passing the results of k-means into a decision tree is probably just going to exchange the centroid for left and right bounds for your features (although it might actually be interesting if the tree ignores particular dimensions in the decision process). Generative models like GMM and LDA also give a lot of useful information.

Regarding literature, although I don't think I've seen this specifically applied to clustering, there's definitely a fair amount of on-going research into techniques for adding interpretability to "black-box models". Consider for example this article: Interpretable & Explorable Approximations of Black Box Models

  We propose Black Box Explanations through Transparent Approximations (BETA), a novel model agnostic framework for explaining the behavior of any black-box classifier by simultaneously optimizing for fidelity to the original model and interpretability of the explanation. To this end, we develop a novel objective function which allows us to learn (with optimality guarantees), a small number of compact decision sets each of which explains the behavior of the black box model in unambiguous, well-defined regions of feature space. Furthermore, our framework also is capable of accepting user input when generating these approximations, thus allowing users to interactively explore how the black-box model behaves in different subspaces that are of interest to the user. To the best of our knowledge, this is the first approach which can produce global explanations of the behavior of any given black box model through joint optimization of unambiguity, fidelity, and interpretability, while also allowing users to explore model behavior based on their preferences. Experimental evaluation with real-world datasets and user studies demonstrates that our approach can generate highly compact, easy-to-understand, yet accurate approximations of various kinds of predictive models compared to state-of-the-art baselines.

Poke around for "black box model interpretability" and you'll find a ton more recent research.


How do feature selection on a sparse matrix?

You can do a dimentionality Reduction as your matrix is Sparse. I would suggest to use PCA. PCA will reduce your 1500 input into k dimensional input of your choice with as much information retained as possible . Here k is a hyperparameter that you need to tune and fine the best one.
Another Approach is LASSO classfier which is a linear model with L1 regularization. This model will perform automatic feature selection and zero out the weight of feature that is not needed. But your input columns must be independent.


What are the input and output channels of a convolution in PyTorch?

in_channels is the number of channels of the input to the convolutional layer. So, for example, in the case of the convolutional layer that applies to the image, in_channels refers to the number of channels of the image. In the case of an RGB image, in_channels == 3 (red, green and blue); in the case of a gray image, in_channels == 1.

out_channels is the number of feature maps, which is often equivalent to the number of kernels that you apply to the input. See e.g. https://stats.stackexchange.com/a/292064/82135 for more info.


What is the best way to propose an item from a set based on previous choices?

There are many great ways to handle this problem. It is a recommendation problem, not a classification problem, as pointed out by others. There are many ways to do recommendation with a data set like this. I'll point out a few methods and you can choose one or try them all.
The first method is called user-based collaborative filtering. The basic idea is to give users recommendations based on the tastes of like-minded users. So, you'd be trying to recommend music based on the listening history of users who have listened to the same songs. Such data can be modeled as a graph or sparse matrix. Then, you choose the exact algorithm depending on how you want to model your data.
The second method is called item-based collaborative filtering. Rather than associating users together, this strategy looks at the set of items a user has 'rated' (the songs a user has listened to) and calculates how similar they are to a specific target item (song), or even to all the songs in your data set. It grabs the set of most-similar items and uses various methods to predict how much a user will like the song.
In this case, you only have binary data (user listened to it or they did not). These calculations tend to work best with actual rating scores (like a 5 star system) because this gives more detailed variation amongst items in the data set.
The third option is to model your data in a graph database like Neo4J and write graph traversal queries in order to find similar items. If you like graph theory, this can be a lot of fun. The sky is the limit in regards to what kinds of traversals will return good results. To get started, think of the users and songs as nodes in the graph, and 'listened' as the edge. $user->listened->$song
Because of ratings and item-based filtering, and because there are probably many songs in your data set, and each user only listens to a very small portion of them, I'd first try a user-based collaborative filtering method which uses sparse matrix operations to calculate recommendations. If your data set is large, these computations scale horizontally so you can leverage parallel processing if you run into performance issues.
You can find more detail about collaborative filtering in this paper.


Counting Number of Parameters in Neural Networks

Actiavation function isnt a parameter.

But here is general formula for counting weghts:

Suppose for neural network with two hidden layers, inputs dimension is "I", Hidden number of neurons in Layer 1 is "H1", Hidden number of neurons in Layer 2 is "H2" And number of outputs is "O"

weights = (I+1)*H1 +(H1+1)*H2 +(H2+1)*O


Why model.fit_generator in keras is taking so much time even before picking the data?

I have noticed this when training on one or more GPUs. I think it is due to Tensorflow having to acquire the resources (in the background it blocks the entire GPU memory). Perhaps some amount of data e.g. the first batch is copied over to the GPU.

I didn't find a way to reduce the waiting time dramatically, but you could try the tensorflow options to allocate less memory and allow memory growth as required. Check out the official docs and this issue for the Keras version.

Here is the latest recommendation form that issue: setting Tensorflow options befor eimporting Keras.

-------------------------- set gpu using tf ---------------------------
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)
-------------------  start importing keras module ---------------------
import keras.backend.tensorflow_backend as K
import keras......


I was able to produce the following with a marker for each point, based on some criteria:

I made a dummy Pandas Dataframe, using the sine and cosine to get lines that cross each other nicely; also making the index itself a column (with reset_index) just to make plotting easier later on.

df = pd.DataFrame.from_dict(dict(sin=np.sin(np.linspace(0, 2*np.pi, 50)),
                             cos=np.cos(np.linspace(0, 2*np.pi, 50))))
df.index.name = "index"
df.reset_index(inplace=True)
df.head(5)

  index sin        cos
0   0   0.000000    1.000000
1   1   0.063424    0.997987
2   2   0.126592    0.991955
3   3   0.189251    0.981929
4   4   0.251148    0.967949

I make a new column which contains the criteria you want, so if one line is above or below another line:

df["criteria"] = df.sin > df.cos    # where sin values are higher than cos values

This will allow us to select which points to plot with which style (e.g. which marker).

The final plot itself is made up of two parts (note that we re-use the same ax object, so all plots appear on the same graph)

The underlying data itself can be added to create the simple lines (these are your moving average columns).

ax = df.sin.plot(c="gray", figsize=(10, 6))    # plots the sine curve in gray
df.cos.plot(c="cyan", ax=ax)                   # re-uses the ax object - important!

The markers, based on our criteria column. We filter the points that satisfy our criteria and plot them with a certain marker, then filter the points that do not have the criteria (using the negation selector: ~). Whereas we used the basic line plot above, we now need to use the scatter kind of plot:

df[df.criteria].plot(kind="scatter", x="index", y="sin",
                     c="green", marker="^", ax=ax, label="higher")

df[~df.criteria].plot(kind="scatter", x="index", y="sin",
                     c="red", marker="v", ax=ax, label="lower");

Note that we continue to use the same ax object.

We can also add the legend, which uses either the Pandas column names by default or the label that was passed in each plot method:

plt.legend()="lower");    # adds legend for all plots

You can see in the matplotlib documentation that there is a marker argument, but it seems you cannot simply specify a column of a Pandas dataframe to give the marker for each individual point.


How to run unmodified Python program on GPU servers with scheduled GPUs?

A popular solution used for job management on GPU environments is SLURM.
SLURM allows specifying the resources needed by a job (e.g. 2 CPUs, 2Gb of RAM, 4 GPUs) and it will be scheduled for execution when the needed resources are available.
A job can be any program or script.


Python Neural Network library/program with simple installation? (Solved?)

If you are trying out neural networks, I would recommend keras, it is a wrapper around TensorFlow. With just a few lines of code, you can build a neural net.

Scikit learn is really good for ml algorithms like logistic or linear regression, naive Bayes, svm and others. I am not sure if it is good enough for neural networks.

I would still think, you would require some blas library.


is there cross validation for xgb classification for multi labels?

Single or multi label doesn't make the difference. Cross validation is only split methodology. It just divides records in your data set to separate train and test splits. Python wrapper implements scikit API, so it'll work with any of the selection methods. Metrics will work too. Just remember to one-hot encode your labels.

  Is using cross validation for xgb or xgb classifier worth? and is there any example I can find?

If you only can afford it, do it for every pipeline you can. You split the data to see how your model generalizes new, unseen records. These splits are randomized, so you preferably would like to repeat it some number of times in a way all of the records has been used some times for training and testing. That's what cross validation does. It's best to obtain distribution of metrics from 50-100 train/test cycles.


Why should I normalize also the output data?

That paper gives a nice answer, where i quoted from. Search for Should I standardize the target variables (column vectors)? in that page.

  Standardizing target variables is typically more a convenience for getting good initial weights than a necessity. However, if you have two or more target variables and your error function is scale-sensitive like the usual least (mean) squares error function, then the variability of each target relative to the others can effect how well the net learns that target. If one target has a range of 0 to 1, while another target has a range of 0 to 1,000,000, the net will expend most of its effort learning the second target to the possible exclusion of the first. So it is essential to rescale the targets so that their variability reflects their importance, or at least is not in inverse relation to their importance. If the targets are of equal importance, they should typically be standardized to the same range or the same standard deviation.


State-of-the-art Python packages that can evaluate language similarity

I suggest you use the Hugging Face implementation which has all the state of the art language models, and fine tune them on your dataset. They have easy to use APIs for finetuning which are same across all the LM models.


How can I improve the accuracy of my confusion matrix to 100%?

Bayes error

To answer your question, first I should explain Bayes error. Assuming we know the exact joint distribution of feature vectors ($\mathbf{x}$) and each class (${C_k}$) as
$P(\mathbf{x},{C_k})$, we build a classifier which assigns label $k$ to each feature vector by this criteria $$\mathop {\arg \max }\limits_k P(\left. {{C_k}} \right|\mathbf{x})$$
It can be shown that this is the best possible classifier by calculating the expected classification error on the whole feature space. This expected classification error is called Bayes error and is the minimum achievable classification error for this feature-label space.

  Training error

If you evaluate your model on the training data and calculate confusion matrix using the training samples you may achieve 100% accuracy because your model may overfit your training data. It means your training error is 0 even the Bayes error may not be.

  Generalization error

If you evaluate your model on the test data and calculate confusion matrix using the test samples you can not achieve 100% accuracy because you are evaluating the generalization capability of your model and its error can not be less than Bayes error.


If the entire production dataset is known is advance, should I use it when fitting normalization parameters?

The point of evaluating your model is to assess how it will perform when it is used on unseen data. Normalizing over the entire dataset causes a data leakage (information coming from the test set). As a result, you will get a biased evaluation of your model.

In any machine learning task, the end goal is to use your model on unseen data to see how much the model has learned and how well it can generalize to unseen data. In that regard, there is no pros for normalizing over the entire dataset, and you should avoid doing so!


Which is first ? Tuning the parameters or selecting the model

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune.

  However, i've also read that model selection shoud be done before tuning the parameters.

Before tuning you need to do some kind of pre-processing before tuning the parameters.
Usually your pipeline will consist of:

Get Data and Clean It.
Do some EDA ( Exploritary Data Analysis) do get better understanding of you data.
Do some feature engineering - roughly, if possible, you would like to transform current data, so it would be more suitable for your ML algo.
Train and evaluate your model on validation data.
Tune your model`s params to get better performance ( fight with overfitting alos included).
Evaluate on Test set

  how are we supposed to do model selection before tuning the parameters

About Model Selection - I think even without having deep inshights about your data, you can already choose some baseline models (I think its a good rule to start with simple models like Linear Regression or K-NN to get the "feel of the performance"), because in most cases you know that you are probably dealing with regression/classification/clustering task, so you can already specify the set on models to try - but before trying the horde of ML models, you should get your data ready.

Some posts like this one can give you the feels of available ml algos, but there are a lot more to try.

In Addition, I think this paper might be very usefull if you are starting to work with ML


Does increasing kernel size in a CNN result in higher accuracy on the training set?

I'd say there is no direct relation between the kernel size and the accuracy.

If you start using larger kernel you may start loosing details in some smaller features (where 3x3 would detect them better) and in other cases, where your dataset has larger features the 5x5 may start detect features that 3x3 misses.

So, I'd say "no".

Anyway, if you add a second convolutional layer on top of your first you'd start having something that's close to larger features. E.g. your small kernels in the first layer would detect small features but the second layer would detect features which are composed of several features from the previous layer.


R recommendations while applying ggplot2 and plotly together to plot a graph

Plotly is R package for creating interactive web-based graphs via plotly's JavaScript graphing library, plotly.js. The plotly R libary contains the ggplotly function , which will convert ggplot2 figures into a Plotly object.

If you are still getting the recommendation even after installing it, it seems it was not properly installed. Please refer the below git hub link and code to do the same installation properly.

Try to via Hadley at https://github.com/hadley/ggplot2

# install devtools
install.packages('devtools')

# install dependency of scales package
install.packages(c("RColorBrewer", "stringr", "dichromat", "munsell", "plyr", "colorspace"))

# Open devtools package
library(devtools)

# move to development mode
# scales and ggplot2 are installed in "~/R-dev" directory, so official version of ggplot2 are not removed.
dev_mode(TRUE)

install_github("hadley/scales", force=TRUE)
# main branch of development
install_github("hadley/ggplot2")
# (Optional) install_github("ggplot2", "kohske", "feature/new-guides-with-gtable")

# Exit dev mode
dev_mode(FALSE)

# load development version of ggplot2
library(ggplot2)


Truncating float/doubles for reproducibility

I'm not expert in this but as far as I know the proper way to test for equality modulo floating point imprecision is to compare the differences of the two values, i.e. instead of:

trunc(a) == trunc(b)

one would do:

abs(a-b) <= epsilon

where epsilon is the constant which represents an acceptable difference, e.g. $10^{-6}$. Of course this requires to read the two values and calculate the difference, instead of simply comparing one against the other.


Why ROC value area under curve of two models is different whereas accuracy, precision, recall, f1-score and confusion matrix is same

Those scorings looks strange for me, but beside that you must remember that F1, accuracy, confusion matrix, etc depends on the chosen threshold, while AUC is threshold-independent (it is an integral over all of the thresholds from 0 to 1).

Your models return some probabilities of being a member of class 1. If you choose to label by '1' only those, which have probability higher than 99% (this is a threshold) then you get a recall at almost 100% and a precision near 0%.


1d time series to time series approximation using deep learning

Yes, you can use one-dimensional convolutions but you have to consider the fact that the size of your inputs should be the same. In Tensorflow you can exploit tf.nn.conv1d for your purpose. RNN networks like LSTMs and GRUs are also other architectures that can be used. If your input has a patter that may happen different times in a single signal try to use conv1d. If you want to use RNNs you have to find which task you have such as a one-to-one or one-to-many-relation. For convolutional architecture, take a look at here and here, and take a look at [here] for RNNs.


Analysis of Alternating Decision Tree on Weka

As per the original paper here on Alternative Tree it says,

"We give a different representation of the same classification rule....
The decision node is identical to what we had before , while the prediction node is associated with a "real valued number"......
The classification that is associated with the path is NOT the label of the leaf, instead that is the sign of the sum of the predictions along the path.for example, the classification of the instance a=b=0.5 is sign(0.5-0.7-0.2)=sign(-0.4)=-1..."

Now there is optimization of it too here which is available here and is implemented by WEKA and JBoost . And of course there is always -Wiki


Yes I suspect you are overfitting. When you build your first stage of models (nearest neighbors, random forest, gradient boosting, etc...) is the process like this?

random_forest.fit(train_data,target)
random_forest_probabilities = random_forest.predict_proba(train_data)
...

If so you are plugging in the same data you used to train the models into your models to get probabilities, which will lead to better probabilities than should be expected in real life and thus much better than should be expected results on your 2nd stage model (i.e.: overfitting). To remedy this you must do everything within a cross validation loop, that looks like this in pseudo code:

for <fold> in <crossvalidation>:
     random_forest.fit(<train_data_fold>,<target_fold>)
     <fold predictions> = random_forest.fit(<test_data_fold>)
     ...

Doing things this way will emulate real life since your predictions will be gathered using models that did not include that data when training them. You can build your final model using all of your prediction variables as input. If you struggle with actually executing on this either edit your question with code or post a new question on how to run everything with a CV loop.


How can I determine best performing groups in a series

Not sure to understand what you are asking. Your purpose is to find the only one best group ?
From the picture you provided, two things are unclear to me:

A and B areas don't have the same width, how are you defining your width ?
A and B are covering areas where the signal looks exactly the same. Is it that for the sake of the example you copy-pasted some fake data or does it have to do with what you are finally trying to do ?

If your width is fixed then you should be able to compute a series of local averages and pick out the best one.


Compare Coefficients of Different Regression Models

Weights across different types of models are not always comparable, so
I think that it would make more sense to do this kind of comparison not across different types of model but within a single type of model varying:

the hyper-parameters (if any),
the set of instances (e.g. selecting different subsets randomly),
the set of features.

I'd recommend in particular varying the instances: if the weight of a feature tends to change a lot depending on the training set it's a sign of overfitting.

If the goal is to compare the importance of some specific features for different models, it's probably more reliable to directly evaluate how the model performs with/without the feature(s).


How to select particular column in Spark(pyspark)?

You could try the following,

testPassengerID = test.select('PassengerID').rdd

this would select the column PassengerID and convert it into a rdd


In R you can run a linear regression. Consider this "academic" minimal example:

df = data.frame(c(3,5,2,7,5,3), c(1,0,1,0,1,0), c(0,1,1,0,1,0))
colnames(df) = c("A", "B", "C")
df

Take this data as an example:

  A B C
1 3 1 0
2 5 0 1
3 2 1 1
4 7 0 0
5 5 1 1
6 3 0 0

Now we can see how B and C describe A in the best way.

reg = lm(A~B+C, data=df)
summary(reg)

Output:

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)    4.917      1.322   3.719   0.0338 *
factor(B)1    -1.750      1.774  -0.987   0.3966
factor(C)1     0.250      1.774   0.141   0.8968
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.048 on 3 degrees of freedom
Multiple R-squared:  0.2525,    Adjusted R-squared:  -0.2459
F-statistic: 0.5066 on 2 and 3 DF,  p-value: 0.6463

This tells us that when B, C is 0, A=4.1917 if B=1 we would have A=4.917-1.750 and if C=1 we would have A=4.917+0.25.

So, we can also make predictions:

predict(reg, newdata=df)

Which would be in this case:

       1        2        3        4        5        6
3.166667 5.166667 3.416667 4.916667 3.416667 4.916667

This is a simple form of ML (linear regression), where the sum of squared residuals is minimized in order to find the coefficients for the intercept as well as B and C which best describe A.

You would write this model like: $A = \beta_0 + \beta_1  B + \beta_2  C + u$, where $u$ is the statistical error term. You would solve this model by minimizing $\sum u^2$ (the sum of squared residuals).

In matrix algebra you could write $y=\beta X + u$, and you would solve this by $(X'X)^{-1}X'y = \hat{\beta}$.

So we do not "maximise" but minimize the statistical error $u$ in order to find the best "fit" for columns B, C given column A.

Have a look at the great book "Introduction to Statistical Learning" to get the main concepts sorted.


The normal distribution is a theoretical model of data. Empirical data can be distributed more similarily or more dissimilarly to a normal distribution.

That empirical data has a couple of notable divergences from a theoretical normal distribution:

Presence of outliers
Not symmetric
Relative "peakedness"

Depending on your goal, you can pick a better model or transform the data to fit the normal distribution model. The power-law distribution might be a better model for the data. If you want to transform the data to better fit to a normal distribution, you can drop the outliers and then apply a log transformation.


What is the actual output of Principal Component Analysis?

Have you tried reading the Intuition section of the PCA page on wiki? Also, the Further Considerations section, I think, explains what the various components represent and addresses your questions.

A short summary of what those sections describe is that the original components (i.e., axes) of the data do not properly represent the relationship inherit in the data. Instead, the components can be combined to yield a new set of components, or axes, that better describe how the data is distributed.

The wiki link states that "PCA can be thought of as fitting an n-dimensional ellipsoid to the data". In that line of thinking, the output of PCA is the eigenvectors and eigenvalues that define the orientation and length of the ellipsoid that best fits the data.

What you do with that ellipsoid is up to you and your application of PCA.


Automatic legal document reviewing advice

I am not a lawyer but I assume a contract is good if it is comprehensive and covers certain aspects extensively. An "easy" solution would be the following:

You could probably train a model which could classify a contract to which area it belongs (aka topic classification). If the probability of belonging with the rest of its the peer contracts is high then it could be a good contract.

Then again I am not sure how effective this would be and what pre-processing steps should be taken and would certainly need a big amount of labelled contracts. I would encourage you to dive into NLP and check the various NLP tasks that may leverage the process with your domain expertise.

Lastly, you could ask for help from a company like Lawgeex to make the NLP (or LLP I think they call it for Law Language abbreviation) stuff for you. But then again, not sure if companies like this could fit the Slovakian market. It would be cool for them to have Language Model that could fine tune in other languages!


I think ALS is very applicable here. I would imagine it would need to be fine tuned quite a bit though. Unless the questions are correlated, it might yield some weird results. Just because two people with PHD's have one child, does not mean the third would only have one child. Basically, the more the data the better. Recommending movies works so well because all the columns are related to movies, but this should work with enough data.


You can use word embedding in order to compare whole phrases. I am aware about two models: Google's word2vec and Stanford's GloVe. Now, word embedding works best with, well - words. However, you could concatenate every word in your phrase and re-train the models. Afterwards, you could calculate their similarity (say, with cosine similarity) and see how similar your whole phrases are semantically.

Hope this helps.


One very obvious way is to use gensim's word2vec most_similar() function to get the most related words to the queried word.

You can check an online version of how it works out here: https://projector.tensorflow.org/

Again it depends on what kind of relations you seek. Other useful methods include:

from nltk.corpus import wordnet as wn

wn.synsets('car')
wn.synsets('car')[0].hyponyms()


Join the List and the Data frame for matching list elements in Python

The question is presented in a somewhat confusing fashion. However, I'm assuming OP has a dataframe setup something like this:

import pandas
data = { 'No': [ 1, 2, 3, 4 ], 'Text': [ 'a', 'ab', 'b', 'bd' ] }
df = pandas.DataFrame( data )

In addition to the df file, OP has a kpi list consisting of 'a' and 'b'. There is also a mask variable, which I am assuming was meant to catch all cases where the df.Text is equal to either 'a' or 'b'. This is where the first mistake is. The way OP wrote that line would match cases where the character a is in any part of the element. In other words 'a' in 'ab' is True. I suspect, OP wanted that statement to be False, which meant it needs to be written as 'a' == 'ab' which would be false. I fixed it in the lines below:

kpi = [ 'a', 'b' ]
mask = df['Text'].apply( lambda x: any( item for item in kpi if item.lower() == x ) )

Basically I changed the item.lower() in x to item.lower() == x. Next, OP indicated that there should be a new column called matchedItem, which duplicates the df.Text when there was a match and '----' when there was not a match. Given the mask variable that OP had written, this can be done simply as such:

df[ 'matchedItem' ] = '----'
df.matchedItem[ mask ] = df.Text[ mask ]

In other words, first I set '----' as the default value for all the elements in column matchedItem. Next, I copy over the elements from column Text only for the rows identified in the mask variable. This results in the following output for print( df ):

   No Text matchedItem
0   1    a           a
1   2   ab        ----
2   3    b           b
3   4   bd        ----

There's a number of things that are unclear on the asking of this question (e.g. did OP really want to set mismatches as '----'? Is No supposed to be an index or a separate column? etc.), but hopefully this answer will guide OP towards the desired result.


How would one use a network trained on CIFAR-10 to find things in a photo?

This is a well defined problem called text spotting. There are numerous avenues to tackle this problem but most of the good ones are based on deep learning. Naively you would use a network like the one you trained on MNIST to slide over your input and see where it fires strongly to build up a string. This approach works reasonably well but to convolute this over your whole input image is extremely computationally expensive. A way that is actually used in practice is a two step process, first there is a network that is trained in localizing areas of interest, which are bounding boxes of parts where there might be texts and then use a more advanced network to grab the text. To my understanding this is also done in one network pass nowadays as opposed to querying on single patches. If you search for text spotting you will find a lot of nice papers and a thesis, mostly from the same guy.

http://www.mathstat.dal.ca/~hgu/Neural%20Comput%20&%20Applic.pdf


Features & Models to compute the probability of certain customer accepting an offer/product from a bank?

I don't think anyone can help you with features without more information about your particular domain, but you could do worse than logistic regression for modeling:

https://en.wikipedia.org/wiki/Discrete_choice

https://en.wikipedia.org/wiki/Logistic_regression


Time series change rate calculations for displaying trend line chart

While it is not very clear to me what specific relationships between your data you want to display, I can give some general advice. I think that this time series visualization calls for so-called diverging color scheme (as opposed to sequential, categorical and other ones). ColorBrewer is nice online tool for selecting an appropriate color scheme and other parameters (note that ColorBrewer's color schemes have built-in support in major data analysis and visualization software, such as R, Python, d3.js, plot.ly and others).

If you would use R environment, it would be quite easy to produce the time series trend line chart that you want by using ggplot2 package and its scale_color_gradient2() function:

... + scale_color_gradient2(midpoint=midValue, low="blue", mid="white", high="orange" )

A quick glance at your profile suggested me that you would prefer a JavaScript solution. In that case, I would advise to use d3.js library or one of multiple other JavaScript visualization libraries or packages. For more details, check this relevant answer of mine.


Optimization motion planning by using Bellman equation

From my understanding that agent is a ball, environment is the plane, action is rolling ball without sliding and the achieved goal is motion planning from 1 point to another point.

That's already a good start. Still, I would recommend to be a bit more concrete to clear everything up. The action is not only the rolling ball, but has to be some kind of instructive control, e.g. move the ball 1cm forward, steer 10° clockwise, accellerate by a - just to name a few possibilities. Given this Action set $A$ with $a\in A$. $P_{ss\prime}^a$ models the state transition probabilities - if you are in a deterministic environment, it is constant 1 for one state and 0 for all others. $R$ is the reward, and here is where it becomes tricky. You want to find the best trajectory to the goal, thus you want to penalize if the agent takes unneccessary actions, leading to the agent reaching the goal as fast as possible. I would recommend a constant -1 for that purpose, if you end the episode when the ball reaches the goal. If this is not the case, it has to be more sophisticated, e.g. the negative distance of the ball towards the goal. This should be everything you need to get your hands dirty.


This uses python and NLTK

import nltk

text = """abcdef
abcd
xabcd
b
bcef
ef
aef
"""

def tokens(lines):
    for line in lines:
        for n in range(1, len(line)):
            ts = nltk.ngrams(line, n)
            for t in ts: yield str.join('', t)
        yield line

ts = tokens(text.splitlines())
freq = nltk.FreqDist(ts)
print(freq.most_common())
# [('b', 5), ('c', 4), ('ef', 4), ('bc', 4), ('e', 4)...

There are other solutions, of course, but I think this meets your criteria. You may need to optimize if your lines are huge as this will create every possible n-gram.


Why does my cost either increase or go down very slowly

I would refer you to this website that provides a very extensive and detailed walk through the math behind the logistic regression and it's cost function.
https://www.internalpointers.com/post/cost-function-logistic-regression

And here is a complete tutorial on how to manually code the whole thing in python:

https://towardsdatascience.com/building-a-logistic-regression-in-python-301d27367c24

Finally, I don't know the reason you need to implement this manually, there are pretty good ready-to-use functions that have been used and tested and that would save you a lot of time.

Good luck !


Should inputs be shuffled for training a model with expected time dependency?

You don't shuffle the actual inputs and outputs in your prediction model. Nevertheless, depending on your prediction model you might have to model the sample distribution including mean and variance. To model sample statistics you can use methods like bootstraping. However, this method assumes your model and inputs to be stationary (none changing distribution over time) and therefore neglects timely-variations, like structural breaks. But there have been modifications, e.g. bloc bootstrapping, to overcome this. You can read more about sampling in time-series here


how to design sigma(std) vector to have values above > 0 in VAE autoencoder

Yeah, almost, except that Relu might cause problems ones it goes to zero (having zero variance is a little odd and the gradient goes to zero as well). Usually you'd go with

$$ \sigma = \exp(\rho) $$

where $\rho$ is coming from the encoder.


In CNN (Convolutional Neural Network), does the combination of previous layer's filters make next layer's filters?

I guess you are making mistake about the filters. After applying filters to the input of each layer, the output will be used as the input of the next layer. The first layer's filters try to find the edges in the image and their output show whether those edges exist in a specified position or not. Next layer filters try to find patterns in the outputs of the previous layer which shows the existence of edges. Due to the point that each filter is a window and specifies a receptive field on the input, it finds patterns in the input which are more abstract and more complicated than the previous layers' activations.


When to normalize or regularize features in Data Science

Step one would be understanding how the algorithms you are using work. Certain algorithms work better (distance based usually) when scaled, others don’t (like random forest). Knowing how an algorithm works will go along way to understanding when (and why) you should scale your data.


To datawarehouse or not to data warehouse?

The main purpose of a datawarehouse is the ability to aggregate different types of data, and columns in a rapid way ( near realtime ) . Storage capability isn't the problem Datawarehousing is trying to solve. I can't really answer your question since i'm not quite aware of the volume of analytics you want to perform, but if it's for training a model over and over ( Online Learning ) , just set up a whole pipeline to apply the transformations you want ( the ETL part ) , model, train, and run your predictive method as many times as you want.


Sampling in Text Classification: can the results be considered 'reliable'?

There seems to be a mistake in your method:

I read about the use of downsampling and upsampling, so I applied them before training and testing the dataset.

It's incorrect to change the distribution of the test set. When resampling, the resampling should be applied only on the training set. The goal is to force the model to take into account the two classes, because in case of imbalance the model tends to focus on the majority class. However the true proportion of the class in the "real dataset" is still the same, and the test set should follow this true proportion. Otherwise the performance looks artificially good on the test set, even though the classifier will make more mistakes with real data since it doesn't have the same distribution.
So the performance values that you obtain on a resampled dataset are meaningless, I'm afraid.

I am considering a publication (not only to include similar analysis), so I would like to check if such results can be considered reliable, despite of the imbalance.

If you are considering a peer-reviewed publication, you must also make sure that your contribution is original (i.e. new) and brings some advantage over the existing methods. This means that you need to know the state of the art in spam classification (there are a lot of papers already published about this task) and show what your method improves something compared to the existing methods. Ideally this is done by proving that your new method obtains better performance than the state of the arts methods using a benchmark dataset. But it's usually hard to beat state of the art performance on a well-known problem.


How to recognize a two part term when the space is removed? ("bigdata" and "big data")

There is a nice implementation of this in gensim: http://radimrehurek.com/gensim/models/phrases.html

Basically, it uses a data-driven approach to detect phrases, ie. common collocations. So if you feed the Phrase class a bunch of sentences, and the phrase "big data" comes up a lot, then the class will learn to combine "big data" into a single token "big_data". There is a more complete tutorial-style blog post about it here: http://www.markhneedham.com/blog/2015/02/12/pythongensim-creating-bigrams-over-how-i-met-your-mother-transcripts/


Loss for CNN decreases and settles but training accuracy does not improve

I guess I was able to figure out the answer to this when I looked carefully at the feature space and printed out the outputs from different convolutional layers. It appears that this issue appears when you have a lot of redundant features in your feature space. Since, this was a binary classification problem with FALSE and TRUE, each image by default had a lot of features that would correspond to a FALSE and the TRUEs only had a small percentage of the feature space being different. The trainer easily learnt the large percentage of the FALSE feature space and hence the loss easily went down while the small TRUE feature space was missed. This is the reason why the loss was going down but accuracy not improving. The trick to avoiding this issue was to downsample the initial image in the beginning before going to the convolutional layer. One could do it with some pyramidal pooling if on Tensorflow, or simply with imresize in matlab.


What is best practice to feature engineer from prior event counts?

You may want to try both options out and see which is better. Feature engineering I think is more like a trial and error (iterative) process.


How can I override the values of 1 column in a big dataframe using a small second dataframe?

Try this:

for recipient_id, office in override_df.values:
    donations_df.loc[donations_df["recipient_id"] == recipient_id, "office"] = office

However, you should use a reference table to avoid this problem!


Decision Stumps with same value leaf nodes

What is the overall response rate? If it's low (even 15-20%) it may be difficult to find decision stumps that contain one leaf with > 50% response!

You could consider oversampling or changing cutoff probability, but I think if your using only 2 leaf trees, your model is bound to struggle.


A convolution employs a weight sharing principle which will complicate the mathematics significantly but let's try to get through the weeds. I am drawing most of my explanation from this source.

Forward pass
As you observed the forward pass of the convolutional layer can be expressed as
$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$
where in our case $k_1$ and $k_2$ is the size of the kernel, in our case $k_1=k_2=2$. So this says for an output $x_{0,0} = 0.25$ like you found. $m$ and $n$ iterate across the dimensions of the kernel.
Backpropagation
Assuming you are using the mean squared error (MSE) defined as
$E = \frac{1}{2}\sum_p (t_p - y_p)^2$,
we want to determine
$\frac{\partial E}{\partial w^l_{m', n'}}$ in order to update the weights. $m'$ and $n'$ are the indices in the kernel matrix not be confused with its iterators. For example $w^1_{0,0} = -0.13$ in our example. We can also see that for an input image $H$x$K$ the output dimension after the convolutional layer will be
$(H-k_1+1)$x$(W-k_2+1)$.
In our case that would be $4$x$4$ as you showed. Let's calculate the error term. Each term found in the output space has been influenced by the kernel weights. The kernel weight $w^1_{0,0} = -0.13$ contributed to the output $x^1_{0,0} = 0.25$ and every single other output. Thus we express its contribution to the total error as
$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \frac{\partial E}{\partial x^l_{i, j}} \frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}}$.
This iterates across the entire output space, determines the error that output is contributing and then determines the contribution factor of the kernel weight with respect to that output.
Let us call the contribution to the error from the output space delta for simplicity and to keep track of the backpropagated error,
$\frac{\partial E}{\partial x^l_{i, j}} = \delta^l_{i,j}$.
The contribution from the weights
The convolution is defined as
$x_{i, j}^l = \sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l$,
thus,
$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = \frac{\partial}{\partial w^l_{m', n'}} (\sum_m \sum_n w_{m,n}^l o_{i+m, j+n}^{l-1} + b_{i, j}^l)$.
By expanding the summation we end up observing that the derivative will only be non-zero when $m=m'$ and $n=n'$. We then get
$\frac{\partial x^l_{i, j}}{\partial w^l_{m', n'}} = o^{l-1}_{i+m', j+n'}$.
Then back in our error term
$\frac{\partial E}{\partial w^l_{m', n'}} = \sum_{i=0}^{H-k_1} \sum_{j=0}^{W-k_2} \delta_{i,j}^l o^{l-1}_{i+m', j+n'}$.
Stochastic gradient descent
$w^{l(t+1)}_{m', n'} = w^{l(t)}_{m', n'} - \eta \frac{\partial E}{\partial w^l_{m', n'}}$
Let's calculate some of them
import numpy as np
from scipy import signal
o = np.array([(0.51, 0.9, 0.88, 0.84, 0.05),
              (0.4, 0.62, 0.22, 0.59, 0.1),
              (0.11, 0.2, 0.74, 0.33, 0.14),
              (0.47, 0.01, 0.85, 0.7, 0.09),
              (0.76, 0.19, 0.72, 0.17, 0.57)])
d = np.array([(0, 0, 0.0686, 0),
              (0, 0.0364, 0, 0),
              (0, 0.0467, 0, 0),
              (0, 0, 0, -0.0681)])

gradient = signal.convolve2d(np.rot90(np.rot90(d)), o, 'valid')

array([[ 0.044606,  0.094061], [ 0.011262,  0.068288]])

Now you can put that into the SGD equation in place of $\frac{\partial E}{\partial w}$.

Please let me know if theres errors in the derivation.

Update: Corrected code


How to estimate the accuracy on a large dataset?

Accuracy has a specific meaning classification -  the data points with predicted labels must exactly match actual labels over the total number of data points.

In order to calculate accuracy, you need the actual labels for each data point. If you do not have actual labels for a data point, those data points can not be used in the analysis.


Pros and cons of using the zscore of a dataset before normalizing it during feature engineering?

Normalizing an already normalized dataset should not change anything unless for some reason a different normalization scheme is used.


Which Machine Learning methods would be good for a beginner-level Data Science project for described dataset?

That doesn't look at all like a simple dataset for a beginner's project:

First if you want to predict future trends you need to look at time series, which is not the easiest type of method. Additionally our data covers only 2 years, it might be too short for good time predictions.
Your dataset is made of various types of attributes so it will probably require some serious preprocessing work (e.g. description string) in order to make the data usable for a particular task.
Start with plotting your data: volume of sales by country, number of customer by country, most common products, this kind of thing.


Theoretically, you do indeed have a linear model, yes.

You have a linear relationship between the dependent variable and your parameters.
You would still be going through the process of performing linear regression, fitting a line through points via a linear combination of some regressors. However you have a non-linear equation due to the higher-order regressors you have manually inserted (x1x2 and x2^3).

This is enough to still call it a linear model generally speaking - see some useful answers in this thread.

In general I wouldn't say you have a linear model in the strictest sense at the end, as you are modelling linear relationship between your dependent variable y and non-linear combinations of your regressors: x, x2, and x3, but perhaps there could be an underlying feature to be observed which is exactly equal to x1x2, call it x4, and then you would have removed one of the non-linear covariates. Tyhe same goes for the final non-linear term.


Validation accuracy is always close to training accuracy

The fact that the training accuracy and the validation accuracy are close it is nothing to be concerned about. As you mention, it means your model are generalizing well.

The thing to be worried about is the low training accuracy. It seems that your model is under-fitting the data but with low variance. This is the typical scenario of a model with high bias and low variance.

The bias is the error (whatever you measure this error) in your training data. It tells you about how well the model is forecasting with seen data. High bias means poor performance (under-fitting). The variance is the difference in performance between the training data and the validation data. If the differences are high (far better performance in training than in validation) the model has high variance (over-fitting).

Typically, decreasing the bias implies increasing the variance (bias-variance trade-off) then, probably, as you get better results in training, the validationa ccuracy may decrease. But, if the model is well designed, you may end up with low bias and low variance.


Parsing and storing a large amount of HTML data

The terms "structured data" or "unstructured data" are not defined in such a way that a given dataset is always either one or the other. There are gray areas and I think this is one example. Since you cannot rely on the structure in your data, I would categorize this as unstructured.

To understand if it's a good idea to store the whole HTML in the DB (and same question for the PNGs), you need to weigh the pro's and con's. Pro storing everything in the DB is the simplicity: You don't have separate places where data is stored, so if you take a snapshot from the DB at some point in time and restore it, you restore the entire state as it was at that time. You do not need to worry about your disk storage separately, to restore that to a given state. Against storing everything in the DB is the amount of data. Can the DB handle it, or does performance suffer too much? Think about retrieving the data, searching/querying it, storing data, making back-ups. This will depend on your choice of database.

The same question for ELK and MySQL: What are the pros and cons of each? MySQL is simpler to install, that's always good. MySQL gives you a relational datamodel (tables can be related using foreign keys). Is that an advantage? MySQL gives you transactions. Is that helpful? ELK mainly gives you scalability, meaning that probably it would allow everything to be stored in the DB and still meet your performance needs.

If you can't store everything in MySQL (HTML and PNGs), then before choosing to store part of the data somewhere else, my first option would be to change DB technology to something that can store everything, rather than to start storing things in different places. So in that case ELK might be a good option, but store the PNGs there, too.


Why Decision Tree boundary forms a square shape and SVM a circular/oval one?

Shape of the SVM decision boundary depends on the kernel (similarity function) used. The "standard" version of SVM has linear decision boundary. The one displayed could be using Gaussian kernel.

Decision boundary of a decision tree is determined by overlapping orthogonal half-planes (representing the result of each subsequent decision) and can end up as displayed on the pictures.

See more here:

https://shapeofdata.wordpress.com/2013/07/02/decision-trees/

https://www.quora.com/What-are-Kernels-in-Machine-Learning-and-SVM


The update rules are ;
$a := a - \alpha.\frac{\partial L}{\partial a}$
$b := b - \alpha.\frac{\partial L}{\partial b}$
Here L which is the Loss [Squared Error in this case ] is given by;
$L  = \frac{1}{2}.(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})})^{2}$
So ;
$\frac{\partial L}{\partial a} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.(-x).e^{-ax+b}  $
$\frac{\partial L}{\partial b} = \frac{1}{2}.2(\hat{y} - \frac{1.0}{(1.0 + exp^{-ax + b})}).\frac{-1.0}{(1.0 + exp^{-ax + b})^{2}}.e^{-ax+b} $
So now guess an intial value for a and b.
You can plug in the an x and y pair for and update the weights simultaneously. The thing to be careful is that when updating for a and b in an iteration use the previous a and b values. That is if you have updated your a prior to b dont use the updated value of a to update b.


algorithm to predict numeric values with sklearn

You should try linear regression with sci-kit learn. You can use both year and mileage as predictors for price.


How to get Keras accuracy for each step in an epoch like in Tensorflow?

Create a custom CallBack:

from tensorflow.keras.callbacks import Callback

class NBatchLogger(Callback):
    "A Logger that log average performance per `display` steps."

    def __init__(self, display):
        self.step = 0
        self.display = display
        self.metric_cache = {}

    def on_batch_end(self, batch, logs={}):
        self.step += 1
        for k in self.params['metrics']:
            if k in logs:
                self.metric_cache[k] = self.metric_cache.get(k, 0) + logs[k]
        if self.step % self.display == 0:
            metrics_log = ''
            for (k, v) in self.metric_cache.items():
                val = v / self.display
                if abs(val) > 1e-3:
                    metrics_log += ' - %s: %.4f' % (k, val)
                else:
                    metrics_log += ' - %s: %.4e' % (k, val)
            print(f'step: {self.step}/{self.params['steps']} ... {metrics_log}')
            self.metric_cache.clear()

The inspiration for the code is here.


Generating data that look alike my original data

What you want is a generative model.

A simple generative model from the deep learning family are autoencoders: neural networks that receive as input your data and are trained to output that very same data.

There are different types of autoencoders. One of the most simple are contractive autoencoders, that have a bottle neck layer, that is, a layer with very few units. For your case, with only 4 input features, you may have a 2 unit (or even 1 unit, you may try to tune this hyperparameter) hidden layer as bottleneck.

Once fully trained, you only take the part of the autoencoder from the bottleneck to the output, and feed it with random numbers as input, and expect to get as output data that follows the same distribution as the original inputs.

The idea is that the training has allowed the net to learn representations of the input data distributions in the form of latent variables.

Depending on the distribution of your input data, a simple contractive autoencoder may not be able to properly learn good representations. More advanced variants include denoising autoencoders, sparse autoencoders and variational autoencoders.

Other generative models that are currently very trendy are Generative Adversarial Networks.


Does regularisation make the loss noisy?

I think your best bet to answer that question is to continue to test it on other datasets to confirm that your observation holds true in multiple situations.

Here is my intuition (although I'm by no means an expert).

The regular loss function is (wx + b - y)^2
The loss function for L2 Regularization is L2 = (wx + b -y)^2 + lamdba * w^2

In the first equation, you only have to worry about w as one term. As a simple example, If you update it by .1 and it improves, you'd be safe to update it again in the next round by .1

In the L2 equation, you have an additional instance of w to deal with and it's squared and multiplied by another number. It would be easy to imagine that the same circumstance may lead you to jump the gap and skip over where the gradient is equal to zero.


First of all, be assured that the kernel matrix (the 2x2 matrix in the figure) in CNN is not constrained to have its diagonal filled with a unique value, and is thus not a Toeplitz matrix.

In www.deeplearningbook.org,  we can read:

  "Discrete convolution can be viewed as multiplication by a matrix, but the matrix has several entries constrained to be equal to other entries"

This means that the global operation of passing a kernel on the input data of a CNN could be expressed as the multiplication of this input data by a matrix. i.e. by a large and sparse Toeplitz matrix.

Authors mention the illustrative simple case of univariate discrete convolution. In univariate discrete convolution, we would be applying a 1-D kernel matrix of length $m$ on 1-D input data of length $n$. Let us take n=10 and m=3.
Doing the convolution could be done by multiplying the input vector data by a n*n matrix, whose diagonal would be composed of the 3 weights of the kernel, shifted of one column to the right at each new row.


Does Gradient Boosting detect non-linear relationships?

GB method works by minimizing a loss function and by splitting each node in a fashion that produces high pure leaves. there is no population formula being estimated and therefore you can estimate all types of relations between the target and the features.
However I wouldn't put in the model correlated variables as:

  For gradient boosted trees, there's generally no strong need to check
  for multicollinearity because of its robustness. But practically
  speaking, you still should do some basic checks. For example, if you
  discover that two variables are 100% the same, then of course there's
  no point in keeping both. Even if it's 98% correlated, it's usually
  okay to drop one variable without degrading the overall model.
  Source: Quora


Predict time to reaction (chemical engineering)?

Is the term you're looking for simply the rate law of the reaction?
Every chemical reaction has an associated kinetic rate law, consisting of its rate constant, and its reaction orders.  Consider the reaction:
$A + B  -> C$
Due to the conservation of mass, we can effectively model the rate of formation of the product C, as the rate of consumption of either reactants A or B:
$d[C]/dt = -d[A]/dt = -d[B]/dt$
The brackets in this equation represents the concentration of each respective chemical in the solution.  That is, the how much of each molecule of chemical is in a particular volume of solution.  This rate of formation is effectively the rate of the reaction itself:
$v = d[C]/dt$
with v being the reaction velocity.  As we can see, the reaction velocity is itself a function of the rate of product formation, which is itself directly related to reactant consumption.  This, in turn, also means that an instantaneous reaction velocity is also a function of the concentrations of reactant in that instant.  Typically, to determine the rate law, the initial rate is measured from initial concentrations, as they are the easiest to measure.
But how do we know HOW they are related?
In short: if we don't do an experiment, we don't.
Without doing an experiment, we can only very generally describe the rate law of a reaction.  One of the most common forms of rate laws is the following:
$v = k[A]^x[B]^y$
where k is our rate constant, and x and y are our reaction orders.  The mystery lies with these variables, and they are determined experimentally as they vary with every reaction.
I am far from an expert in kinetics, but your problem must consider the kinetics of the particular reaction being done.  Another quirk of your problem is that the rate constant k itself depends on temperature by a rather involved equation for someone who doesn't know of, or particularly care for the intricacies of chemical kinetics.
Additionally, if I'm understanding your model correctly, you'll also need to consider how the flow of the system itself may or may not impact the concentration of each chemical species.  As far as I understand, the initial addition of reactants will generate a gradient of concentration that, as a result, will impact the kinetics of the reaction as a function of its rate of diffusion through the volume of the reaction. One can reasonably conceive that the flow you're describing can affect this rate of diffusion. This is something that is far beyond my realm of expertise, and is more in the realm of fluid mechanics.  Of course, for the purpose of this problem, you can intelligently cheese these things, especially with the reaction rate issues by looking up relevant values from well-documented processes.
If you want to do some further reading on rudimentary kinetics, consider the following:
https://www.chemguide.co.uk/physical/basicratesmenu.html#top
https://opentextbc.ca/chemistry/chapter/12-3-rate-laws/


How to select multiple columns in a RDD with Spark (pySpark)?

I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer spark 2.3.2)

First convert rdd to DataFrame:

df = rdd.toDF(["M","Tu","W","Th","F","Sa","Su"])

Then select days you want to work with:

df.select("M","W","F").show(3)

Or directly use map with lambda:

rdd.map(lambda x: [x[i] for i in [0,2,4])

Hope it helps!


Can GAN generate very happy facial expression

Here's a paper by Tom White and its repository studying the specifics of sampling from generative models. Section 3 in the paper, Attribute Vectors, might be of interest for your problem. I think you'll likely have to rely on another dataset to learn the latent code for a smile to use on your own images, but given the results of the author's own work it should be fairly good.

Hope this helps!


The issue with building a regression model on all 3 of these is that you are potentially introducing multicollinearity into the model. Although log(input) and sqrt(input) are not linear functions of the input a quick test (using Matlab) shows they are still highly correlated (depending on the range)

input=rand(1,100);
input_log=log(input);
input_rt=sqrt(input);
corrcoef(input, input_log) %->0.8549
corrcoef(input, input_rt) %->0.9779
corrcoef(input_sq, input_rt) %0>0.9407

Multicollinearity will not reduce the predictive power of the model, but it will make the regression coefficients of these variables difficult to interpret since small adjustments in the data may cause the regression model to switch which of the 3 input variations it finds significant. It also adds unnecessary complexity to the model. I would train the models with the 3 separately if you want to observe the effects.

Additionally, applying things like sqrt and log to your output can be useful if the data currently can't be described by a linear relationship.


Why my Decision Tree Model is not further bifurcated in to Age

You have your Decision Tree set to a default expansion of 3 levels. The Decison Tree works with the most significant features first, and it has not deemed age as important. Increase the default expansion to more levels to see the bifurcation of age.

A word of caution: deep decision trees are prone to overfitting.


Hindsight Experience Replay (HER) results obtained 50 times faster than original paper?

I am reading the results incorrectly.

1 epoch = 50 episodes = 2500 timesteps
50 epochs = 2500 episodse = 125,000 timesteps
125,000*16 CPU cores = 2 million time steps.

Will give a more detailed answer later.


ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0,

It has to do with your coding style. Your code is too similar to mathematical notion. Issues include doing too much on single line and operator precedence in Python

Your code:

vnew = dt * Minv @ (f*src[it]  -  (Kv @ v) -  (Ku @ u)) + v

Breaking into simpler statements would allow for easier debugging.

term_1 = (f*src[it])  -  (Kv @ v) -  (Ku @ u)
term_2 = term_1 + v
term_3 =  Minv @ term_2
vnew = dt * term_3


The validation set is there to stop you from using the test set until you are done tuning your model. When you are done tuning, you would like to have a realistic view of how the model will perform on unseen data, which is where the test set comes into play.

But tuning the model is not only hyperparameters. It involves things like feature selection, feature engineering and aslo the choice of algorithm. Even though it seems like you are already decided on a model, you should consider alternatives as it might mot be the optimal choice.


Predicting a random variable Y will increase within n minutes if random variable X increases

You need to look for lag effects - the easiest thing is to try a number of lag time windows and look at how they correlate with target. If you think there is a distinctive lag impact, then at some point the correlation strength should increase.


You can use the --timeout flag to increase the worker timeout.

Run gunicorn --help for more information about the available flags


Keras: Softmax output into embedding layer

You are describing a variation of Generative adversarial networks (GANs). The Generator, the part of the model that creates new examples, needs to output complete sentences. The Discriminator, the classifier, takes those complete sentences as input.

In particular, you are describing a DiscoGAN which discovers cross-domain relations given unpaired data. DiscoGANs have been applied to the style of images.


Output a word instead of a vector after word embedding?

It depends on the architecture of the model.

A lot of work uses the text data to classify some specific category (e.g., the IMDB movie reviews or sentiment analysis on Tweets).

Other work can actually take the words of a sentence and predict the last word. Or break it into each word predicting the subsequent word, which is really what the RNN/embedding dimension is doing.

Other's have made these even at the character level. Ultimately, it depends on how you process the data and specify your outcome. Below are some useful tutorials that should add a little more clarity.

http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
http://deeplearning.net/tutorial/lstm.html
http://machinelearningmastery.com/sequence-classification-lstm-recurrent-neural-networks-python-keras/
https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py


With GlobalAveragePooling2D, only one Feature per Feature map is selected by averaging every elements of the Feature Map.
e.g. if your global average pooling layer input is 220 x 220 x 30 you will find 1x1x30 output.
It means that you are finding a global representative feature from every slice. That is Global Average Pooling.
No, this isn't specific to transfer learning. It is used over feature maps in the classification layer, that is easier to interpret and less prone to overfitting than a normal fully connected layer.
On the other hand,
Flattening is simply converting a multi-dimensional feature map to a single dimension without any kinds of feature selection.


How do we solve this logistic regression question?

It can be found, assuming a proper learning rate, a suitable threshold, and binary cross-entropy cost, since it translates this into a convex problem, in which we have one global optimum. We don't have closed form solution for logistic regression, but through gradient descent we can get to this optimum arbitrarily close. I'd suggest running a logreg from scikit-learn or a familiar library.


difference betwen predicting seen and unseen data

What you want to do in building predictive models , in general , is closing the gap between the loss on seen data ( training data ) and unseen data ( test data ). Increasing the numbers of features will naturally make the hypothesis ( the function that is meant to learn from your training and will receive unseen data ( X ) to predict your target value ( Y ) ) more complex and can fit your training data perfectly at a certain point, BUT there is a thresh-hold ( which you went above according to your results by increasing the number of features) where your model starts giving very good results on your training set and much poorer results on the test set, and that is called overfitting which is a problem caused by high variance in your model which basically means you are allowing your function to be complex enough to fit your training data perfectly, and be less able to predict unseen data ( your test data ).

Go with the feature selection technique if it reduces the gap between your training loss and test loss!

Notice here in this example that increasing the number of features gives lesser test loss but, at a certain point, the loss on test starts going up again.


Here is an example on the iris dataset

t(sapply(by(iris$Sepal.Length,iris$Species,function(x){x[1:2]}),as.numeric))

Where species is your customer, and Sepal.Length is your fruit.


I can tell from your screenshot that you are plotting the validation accuracy. When you overfit your training accuracy should be very high, but your validation accuracy should get lower and lower. Or if you think in terms of error rather than accuracy you should see the following plot in case of overfitting. In the figure below the x-axis contains the training progress, i.e. the number of training iterations. The training error (blue) keeps decreasing, while the validation error (red) starts increasing at the point where you start overfitting.

This picture is from the wikipedia article on overfitting by the way: https://en.wikipedia.org/wiki/Overfitting Have a look.

So to answer your question: No, I don't think you are overfitting. If increasing the number of features would make the overfitting more and more significant the validation accuracy should be falling, not stay constant. In your case it seems that more features are simply no longer adding additional benefit for the classification.


What are the basic approaches for balancing a dataset for machine learning?

Well, you just train it on unbalanced dataset, it is not a problem. I don't think you need to apply some special techniques.

The only case where you may want to do something special is if your classes are skewed by their nature (if the skewness is the property of class itself, not of just your specific dataset). For example, if you build a classifier which will tell if someone has a malignant tumor in X-ray image or not. By nature of the task absolute majority of patients will not have a malignant tumor. In such case you may want to tune your classifier algorithm a bit, for example introduce a weighting into your SVM or smth like it.

A special case is if you have EXTREMELY unbalanced classes. For example 100000 of positive examples and 20 negatives. In such case you will want to go away from classification task to the approach called 'anomaly detection'.


Oversampling before Cross-Validation, is it a problem?

Oversampling the training data may help the classifier to better predict on the originally less represented class. This does not mean that it should be applied to performance metrics, as it changes the original target distribution and thus creates bias in the results.

Imagine the problem of cancer detection, where your original dataset is unbalanced: 10% of the patients have cancer y=1 and the remaining 90% don't y=0. If you train a classifier which is prone to error on unbalanced datasets (such as an Artificial Neural Network), you may end up predicting always the majority class: y=0.

If you oversample to a new distribution, let's say 50/50,  your classifier is expected to increase the performance, specially on the positive class. Nonetheless, to measure the performance on real data, which is by itself skewed, measure on oversampled may not be the best choice.

Thus, if you are optimizing the hyperparameters or choosing from a set of classifiers, cross-validating with oversampled data may provide you with a different perspective on the classifier's ability to predict on both classes with equal importance. Nonetheless, if you are estimating the real-life prediction capability, I would not advise you to oversample such validation data!


Hyperparameter optimization performance comparison

Just measure the out-of-fold-value of your choice, you can add time complexity to the table and than just extract percantage difference.


Why would batch normalization allows us to use higher learning rate in the neural network?

Section 3.3 of the original batch normalization paper has a good explanation of why this is the case.

Problem with a higher learning rate

First you need to understand the problem with higher learning rate. Higher learning rate causes exploding or vanishing gradients. In other words, gradients get multiplied by each other, so lower layers experience a compound effect of the gradients that are in higher levels.

How does batch normalization help?

Batch normalization is all about keeping the activations of all layers normalized, preventing them from becoming too large or small. So this directly helps to prevent exploding/vanishing gradient. Due to this reason, batch normalization allows higher learning rates.


Choosing best predictors neural networks

A very quick way is to run some Tree-based ML model on your data, such as Random Forest or XGBoost. Tree-based models can return importance coefficients, estimating the relative explanatory power of each variable. You can implement a very large and deep ensemble of trees (we don't really care about overfitting at this point) so they return you the three strongest predictors. You can then take them and feed them into a Neural Network.

Another, more time consuming method is to run the model multiple timesa and substitute each variable, in alternation, with random noise with the same mean and standard deviation of the original variable. This perturbation method will tell you how much performance decreases when one variable is replaces by noise. This is accurate but very time consuming.


How many features does (Random Forest) need for the trees?

The (maximum) number of features is a hyperparameter, i.e. it depends on what you set it to or what the default of the implementation you are using is. In "Introduction to Machine Learning with Python" by Mueller and Guido the authors recommend the following:

  As described earlier, max_features determines how random each tree is,
  and a smaller max_features reduces overfitting. In general, it’s a
  good rule of thumb to use the default values:
  max_features=sqrt(n_features) for classification and max_features=log2(n_features) for regression.

Which is, for example, in line with the default in scikit learn. This what the parameter looks like for classification in scikit learn:

  max_features: int, float, string or None, optional (default=”auto”)

  The number of features to consider when looking for the best split:

  If int, then consider max_features features at each split.
  If float, then max_features is a fraction and int(max_features *
  n_features) features are considered at each split.
  If “auto”, then max_features=sqrt(n_features).
  If “sqrt”, then max_features=sqrt(n_features) (same as “auto”).
  If “log2”, then max_features=log2(n_features).
  If None, then max_features=n_features.

  Note: the search for a split does not stop until at least one valid
  partition of the node samples is found, even if it requires to
  effectively inspect more than max_features features.

Source: scikit learn documentation


How to force pytorch model to predict only positive values

If you know that your output are positive, I think it makes more sense to enforce the positivity in your neural network by applying relu function or softplus $\ln(1. + \exp(x))$. You could also have a look at Generalized models which extend linear regresssion to cases where the variable to predict is only positive (Gamma regression) or between 0 and 1 (logistic regression). If you are predicting a categorical variable, you could also perform one hot encoding and transform your regression problem in a classification. Last but not least, as suggested in the last question it might be interesting to normalise your output between 0 and 1 and have a logistic regression in the last layer. Hope this help


Working Behavior of BERT vs Transformers vs Self-Attention+LSTM vs Attention+LSTM on the scientific STEM data classification task?

A1. BERT by itself may not be useful for scientific terms. You have 2 choices, either find a pre-trained embedding specific to the scientific text database or use transfer learning and build upon BERT.
A2. Fastext or Glove will have the same issue as BERT
A3. You have mentioned that BERT works well at a high level. So I would not advice you to create embeddings from scratch and instead use transfer learning to enhance the embeddings


MLP classifier Gridsearch CV parameters to tune?

Usually learning rate would be between 0.0 to 1.0 , please go ahead by specifying values in those range.
This thread would be definitely helpful for your further research and reference
Thanks,
Durga


Let's say we have a set of 40 numbers from 1 to 40.
We have to pick 4 subsets of 10 numbers.
Case 1 - Bagging -
We will pick the first number, put it back, and then pick the next. This makes all the draw independent and consequently have very little correlation.
So, if you make a Tree on the first 10 samples and another Tree on the next, both the trees will have little correlation and high variance(among them) (more independent splits).
At the same time, because of selection with replacement the data points will be repeated [~63% unique] [Ref], which will increase the bias of individual Trees.

In case of bagging, the sample size is equal to the size of the dataset but we just created this scenario to compare it with Pasting.

Same logic goes for splitting with Random Features subset i.e. RandomForest.
It might be possible that a Split on a particular Feature may result in the correlated next split(always). So if we randomly pick a subset of features before each split, then this will further reduce the Correlation. But again, we will have increased Bias.
Case 2 - Pasting -
Here, because of selection without replacement, the data points in each sample will be unique which will result in lesser bias of individual Trees.


How to explain decision tree algortihm in layman's terms?

I learn well by examples, so perhaps this will be helpful:

Lets say you are a resort considering your next advertizing campaign. You will be sending out 1,000 vouchers that give two free days at the resort, with the hope that the customer will decide to extend it to a full week (who want's a two day vacation?!).

You have a rather extensive database of customer data, including where they are from, income range, duration of stay(s) and money spent at each stay, number of children, marital status, and employment status (working, unemployed, retired). You would like to create a decision tree to help select the customers to send the vouchers to (specifically, those who will spend at least $200 per day and stay at least one week.

To create the tree, a computer will perform the following steps:

From the above list of customer data, find the property that best separates the customers into two groups.
Repeat step one for each of the above groups using the remaining properties.
In the end, you will have a tree where at each point, you can make one of two decisions. Following a path leads to a decision. The split points will be chosen to maximize the probability of a correct classification.


What model should I choose for Jobs recommender system?

One method is using locality-sensitive hashing (LSH), an approximate nearest neighbor search method.

Each document, both resumes and job descriptions, is hashed into the same space. There are several ways to perform the hashing. An older method is shingling. The entire process is outlined in Chapter 3: Finding Similar Items in Mining of Massive Datasets. A newer method is doc2vec.

Once all the documents are in the same space, LSH maps similar items to the same “buckets” with high probability. It is a type of hashing where collisions are features, not bugs. The collisions are a variation of clustering. Given a new document, all similar documents are retrieved also. Given a resume, retrieve similar job descriptions. Given a job description, retrieve similar resumes. (After retrieval, filter out documents in the same document class)

Since LSH is an approximate algorithm, it scales well to millions of documents and handles noisy data. Resumes and job descriptions are noisy descriptions so it is appropriate to have an algorithm that can handle noise.


Identifying common keyphrase frequency in large dataset

I would like to attempt to identify frequently used words or groups of words

The difficulty here would be to capture multiword terms, as opposed to single words. This implies using n-grams for various values of $n$, and that can cause a bias when comparing the frequency of two terms of different length (number of words).

  I would also need some way of filtering out invalid words like ('I', 'and' etc)

These are called stop words (sometimes function words or grammatical words). They are characterized by the fact that they appear very frequently even though they consist in a quite small subset of the vocabulary (fyi this is related to Zipf's law for natural language). These two properties make them easy enough to list in a predefined list so that they can be excluded, there are many lists available (e.g. here or there).

Since you don't have any predefined list of terms, a baseline approach could go along these lines:

For every value of $n$ to consider, collect all the $n$-grams
Remove any $n$-gram which contains only stop words (or which contains mostly stop-words) (note: it might be better to do this step first, but only if it's safe to assume that multiword terms don't contain stop words)
Calculate the document frequency for every candidate term (the same DF as in TF-IDF weights)
Filter out the terms which have a very low document frequency (experiment with different values for the threshold). This step should eliminate a lot of noise, but probably not all of it.
You will probably still need a bit of manual filtering here if your goal is to obtain a clean list of actual terms. Normally there should be few long n-grams left and the ones left should be good for the most part, however there might still be a lot of false positive unigrams and bigrams.

This approach is very basic but it's easily adjustable, you can adapt it to your data, possibly add steps etc. Otherwise there are probably specialized tools for terminology extraction, but I'm not familiar with any.


SKLEARN GridSearchCV hinting higher accuracy than Pipeline but with same parameters as Pipeline estimators

The GridsearchCV object in sklearn does a cross-validation on the data you feed it during your fit. In your case you have specified cv=5: this means GridSearchCV splits your data into train/test splits 5 times and reports on the mean performance over those 5 trials to be 0.868.
You asked why GridSearchCV knows the best parameters without feeding it testing data - this is because it's taking your training data & splitting that into a smaller subset of train/test splits (5 of them to be exact).
You are then evaluating the best model on the testing set and getting accuracy of 0.805. This can happen if the single train/test split you performed split the data in a way that causes some of the more difficult samples to predict to be put in the testing set. If you were to perform the same cross-validation your GridSearchCV performed (say with cv=5) you might find the average performance to be a little higher and closer to the 0.868 it cited.


Currently I am fiddling around with Ax/BoTorch to optimize a ResNet50 for Object classification (and later on detection via SSD). I found it quite hard to apply the samples from the docs to my own model, so this is a very first prototype. Just have a look at this GIST...

https://gist.github.com/pinkerltm/74f98485d1408849476fb02686d0c228

So far I can tell, that it is somehow optimizing the learning rate into the right direction, but I think I have to wire some more Hyperparameters properly and perhaps optimize longer to get a result which is compareable to simply defining the best parameters manually so far, as Accuracy with the optimized Learning rate only gets around 67% after several 100 epochs while at least 71% should be easily possible with this model.

I am also very interested in sharing experiences and findings on this. I will edit my answer here accordingly so that it should provide at least an simple example of optimizing a CNN Classification Network with Ax/BoTorch.


Similarity matching between two distinct datasets (marketing case study)

If I understand your question correctly, you have two groups of people: Group A, each of whom has purchased the product, say yogurt; and Group B, each of whom has not purchased yogurt. Your problem at hand is to find all people in Group B who will be likely to purchase your yogurt, if they have a similar profile as people in Group A.
This seems to be a very common problem in causal inference, where you need to match the treated people with the control group, but since one person could not be both treated and untreated, we need to find "similar" people on both sides such that they are comparable (in terms of their characteristics, or variables) so that we can make a causal inference from there.
Now, returning to your problem, I don't think it is necessary to do the clustering for matching. Instead, you could consider the "matching" approach commonly used for causal inference. Here is an r packages that comes to my mind: MatchIt.
In essence, what you need to do is to regard Group A as treatment and Group B as control. I believe they provide many different ways of matching algorithm and you can certainly play around to see which one works for you the best.


I don't understand why you would like to fill values with zeros ! This would basically mean, "this guy, who is 170 cm tall, weights 0 kg" and would fool your network. In my opinion, you have two options:

discard missing values (the entire row): you end up with less but more consistent training data
if you really need these rows, then fill missing values with some heuristics: for example, give them the mean of their column, or apply a simple linear regression. Beware that this will add a bias in the learning process, but it would be definitely better than giving random values.

At very least, if you have a lot of missing values, then maybe you should think of selecting a specialized model for partial training data. You actually don't tell us what network you are using, but you might modify it to handle missing values more intelligently than by filling missing values.

Finally, the need of normalizing once again depend on your model (which you don't describe). But this would definitely come after the processing part.


How to set class weights for imbalanced classes in Keras?

You could simply implement the class_weight from sklearn:

Let's import the module first

from sklearn.utils import class_weight

In order to calculate the class weight do the following

class_weights = class_weight.compute_class_weight('balanced',
                                                 np.unique(y_train),
                                                 y_train)

Thirdly and lastly add it to the model fitting

model.fit(X_train, y_train, class_weight=class_weights)

Attention: I edited this post and changed the variable name from class_weight to class_weights in order to not to overwrite the imported module. Adjust accordingly when copying code from the comments.


Choosing class labels from annotated data

That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric:
\begin{eqnarray}
\kappa &=& \frac{p_{o}-p_{e}}{1-p_{e}}\\
       &=& 1 - \frac{1-p_{o}}{1-p_{e}},
\end{eqnarray}
where $p_{o}$ and $p_{e}$ are the amount of agreement you observe and that due to chance, respectively. The reason this is important is that the amount of agreement your human annotators achieve is a theoretical upper-bound on the performance of your machine learning solution—it provides context for interpreting the performance of your algorithmic approach.


Keras LSTM Input Shape - Batch Size and Time Step

I don't see much point in doing sliding windows with LSTM. The whole idea of this layer is to capture long-term dependencies and you're just throwing it away with the 5 step window limit.

The easiest for this case would be using a Conv1D with kernel_size=5 instead. Then the windows would go automatically with the kernel size. (But only one layer, if you add more, you will capture more steps). Other possibilities are 2 layers with kernel size = 3, or maybe 3 layers with sizes 3, 2, 2.

If you do want to use windows with LSTM, you will have to organize the data manually. This means you will loop your data and get segments of length 5 and treat each segment as an individual sequence.

In this case your input shape will be (5,1) and you will have far more than 82 samples. On the other hand, if all your sets are longer than length 5, you will need no padding at all.

Example loop:

originalData = load_a_list_of_samples()
windowData = []
for sample in originalData:
    L = len(sample) #number of time steps
    for segment in range(L - 5 + 1):
        windowData.append(sample[segment:segment+5])

windowData = np.array(windowData)

My suggestion though, unless you have this requirement for some special reason, is don't use windows at all. Let the LSTM do their job and capture long term dependencies.

Pad the data as intended, use the Masking layer and place the LSTM with return_sequences=True.

The results will be like (showing only the length dimension)

inputs = [step1 , step2 , step3 , step4 , step5 , step6 , step7 ]
outpus = [class1, class2, class3, class4, class5, class6, class7]

Make sure you mask your loss function as well, as I'm not sure Keras will do the masking job correctly there.


Does this encoder-decoder LSTM make sense for time series sequence to sequence?

Yes, it makes sense. Seq2seq models represent, in the RNN family, the best for multistep predictions. More classical RNNs, on the other side, are not that good for predicting long sequences.

If you need to implement a seq2seq model in TensorFlow 2.0 / Keras, each model follows the following structure:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, RepeatVector, Dense, TimeDistributed
from tensorflow.keras.activations import elu

# these are just made up hyperparameters, change them as you wish
hidden_size = 50

seq2seq = Sequential([

    LSTM(hidden_size, input_shape = (input_sequence_length, no_vars)),

    RepeatVector(prediction_length),

    LSTM(hidden_size, return_sequences = True),

    Dense(hidden_size, activation = elu)

    TimeDistributed(Dense(1, activation = elu))

])

and then train it as usual with seq2seq.compile() and seq2seq.fit().

If you want to stack more LSTM() layers on top of each other, simply add them to the model I depicted above. Please keep in mind that LSTM models are very computationally expensice; without a good GPU even this "basic" model could be painfully long to train.

One modification I'd suggest, looking at your image, is to make the LSTM-encoder and -decoder parts of equal size and depth.

Alternatively, you can implement a more classical "Autoencoder-like" architecture, with LSTM() layers for encoding and decoding, and Dense() layers in the middle. However, seq2seq models are the most powerful at the moment.

To my knowledge, the only models more state-of-the-art than this are attention models. The problem is that they are so much state-of-the-art that TensorFlow/Keras doesn't have built-in layers for them, and you'd have to create your own custom layers (it's a pain). The only extensive implementation of attention models I found is from this blog post, but things are going to be very very complicated here. I didn't try to implement one yet.


What's the difference between sequence preprocessing and text preprocessing in Keras?

In tf.keras.preprocessing.text (docs) you have utilities to process discrete token sequences, normally used to represent text.
In tf.keras.preprocessing.sequence (docs) you have utilities to process both continuous value sequences (normally used to represent time series) like TimeSeriesGenerator, and discrete token sequences (i.e. text), like the skipgrams function.


What is the impact of changing image sources on an image recognition?

In general the accuracy will change (even dramatically as a google research team found out) if only with slight changes in the input image quality.
References:

Understanding How Image Quality Affects Deep Neural Networks

Image quality is an important practical challenge that is often
overlooked in the design of machine vision systems. Commonly, machine
vision systems are trained and tested on high quality image datasets,
yet in practical applications the input images can not be assumed to
be of high quality. Recently, deep neural networks have obtained
state-of-the-art performance on many machine vision tasks. In this
paper we provide an evaluation of 4 state-of-the-art deep neural
network models for image classification under quality distortions. We
consider five types of quality distortions: blur, noise, contrast,
JPEG, and JPEG2000 compression. We show that the existing networks are
susceptible to these quality distortions, particularly to blur and
noise. These results enable future work in developing deep neural
networks that are more invariant to quality distortions.

Does Deep Learning Have Deep Flaws?

A recent study of neural networks found that for every correctly
classified image, one can generate an "adversarial", visually
indistinguishable image that will be misclassified. This suggests
potential deep flaws in all neural networks, including possibly a
human brain.


Multivariate VAR model: ValueError: x already contains a constant

It seems you may have things working, but maybe I can still help

Catching those pesky non-varying features

Assuming a is your dataframe, with only numerical types and columns as features, you can try the following, which will return rows filled with NaN values for features of zero variance i.e. the mean value is equal to the minimum value, so the value cannot vary.

a.T.where(a.T['mean'] == a.T['min'])

They check is slightly differently in the source code of the VAR model which raises your error:

result = (np.ptp(s) == 0.0 and np.any(s != 0.0))

So they check that the value doesn't have any peaks (ptp = "peak-to-peak"), so it doesn't vary - and also that the value is not equal to zero.

In any case, there is little tolerance for it. You have done right by trying to prune out features with low (near zero) variance.

Another possiblity which I didn't notice you having already tried, would be to utilise the other arguments to the fit() method of the VAR model class. It has the following signature:

VAR.fit(maxlags=None, method='ols', ic=None, trend='c', verbose=False)

If you were to change the trend argument to nc, meaning no constant - you may also get around your error. Check out the documentation.

Source code analysis

Looking through the contents of the file that raised your error, it seems it occurs while processing the "original input data" - as defined by the function add_trend (where your error was raised).

This suggest that your reasoning about the error being found on the data after lagged features are produced would be incorrect in this case (although it's a clever idea!)

Clearing my name...

You said:

  ... So the original answer suggested in the previous question was not entirely sufficient.

I would like to point out that my answer to your linked question was indeed sufficient, because I had written that there are constants - not that the values must all be equal to zero!  :-P