I do not program on python. Nevertheless, I would say the key relies in the number of samples (629,145). They are many. The SVM has to test them to pick which are good support vectors for data partition/regression and given the size of the dataset there are a lot of alternatives. That issue plus the number of different C, gamma/sigma and, perhaps, epsilon (not sure you are classifying or regressing) tested during the optimisation of the SVM hinders convergence.
There are people who used clusters instead of the original dataset to train the SVM. For instance:
Barros de Almeida, M., de Padua Braga, A., Braga, J.P., 2000. SVM-KM: speeding SVMs learning with a priori cluster selection and k-means, in: Proceedings. Vol.1. Sixth Brazilian Symposium on Neural Networks. Rio de Janeiro, (Brazil), pp. 162–167.
can be a good starting reference but, I am aware there a others. There is at least one R package based on that idea (LinearizedSVR).
In addition, there are analytical methods to infer the values of the parameters C, gamma/sigma and epsilon (for regression) based on the characteristics of the training dataset. That means no optimisation is necessary, although there is a friend that still tunes gamma/sigma but fixes C fallowing these approaches. I think there is available code (at least for R) somewhere in the net. The refs are:
Cherkassky, V., Ma, Y., 2004. Practical selection of SVM parameters and noise estimation for SVM regression. Neural Networks 17 (1), 113–126. 10.1016/S0893-6080(03)00169-2
for regression and,
Keerthi, S.S., Lin, C.-J., 2003. Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel. Neural Comput. 15 (7), 1667–1689. 10.1162/089976603321891855
for classification SVMs.


Books on time series and sequence classification

There are numerous topics that you've mentioned but I will suggest those which I've read and are helpful.

For hidden markov models and markov processes I suggest reading Pattern Classification by Richard O. Duda. You can also take a look at Pattern Recognition and Machine Learning by Christopher Bishop. For better understanding Markov processes and their behaviour, you can also take the course stochastic process which is also known as random process. I suggest taking that course before reading any book, because you may need some help if you're not very familiar to the concepts.

For LSTMs, I highly suggest taking a look at the fifth course of deep learning on coursera by pr. Andrew Ng. If you do their homework you can realise how the inner operations exactly work and you'll have a very deep understanding of time series and what their nature is. After that, you can take a look at the Deep Learning book by Ian Goodfellow.


Understanding experiments in Continual Learning

In the figure, the class-accuracy for each of the two classes per task is shown (y-axis) as the number of different tasks learned is increased (x-axis).
So in the first graph, we see that as the model learns subsequent tasks (task 1: distinguishing 0-1, task 2: distinguishing 2-3, etc), the classification accuracy of the model on '0's remains near 100%, whereas the classification accuracy of the model on '1's decreases.


Is there any domain where Bayesian Networks outperform neural networks?

One of the areas where Bayesian approaches are often used, is where one needs interpretability of the prediction system. You don't want to give doctors a Neural net and say that it's 95% accurate. You rather want to explain the assumptions your method makes, as well as the decision process the method uses.

Similar area is when you have a strong prior domain knowledge and want to use it in the system.


Name of this type of cross variable interaction Plot


Differentiable loss function for ranking problem in regression model

One option is to use standard framing as learning to rank the relative positions, thus pairwise loss function.


Is it possible to change misclassification cost in caret?

I would have to read more carefully: train has a weights parameter that do this.


You don't need to load the whole dataset into memory at once. The only data you need in memory are the samples in a single training batch. Use the fit_generator method rather than fit to pass in an iterator that feeds samples to your model from disk rather than loading all of that data at once. Here's a tutorial that discusses this more.


How can I create a table from an existing table in SQL but using cells from the old table as columns in the new table?

I tried this once, but due to some database settings I was not permitted to create a table with an SQL-command. Maybe something similar is happening to you? I had to copy the DDL code (Data Definition Language) for the table and create it, and then add the rows.
Maybe you could first copy and run the DDL. Should be available in whatever program you are using for the database.
https://www.educba.com/sql-ddl-commands/
Once the table is created, it will be empty. Then you populate it. Here is some example code.
INSERT INTO table2 (column1, column2, column3, ...)
SELECT column1, column2, column3, ...
FROM table1
WHERE condition1 AND condition2 AND condition3;


How to investigate what has been learnt in CNNs?

There is a famous paper about tracking what a CNN network has learnt. You can visualise to see which parts are more engaged in the classification using DeConvNet. In this paper, it was officially observed that first layers attempt to find simple lines and edges while deeper layers try to put the previous things together to make abstract concepts, like mouth, eye and such meaningful things. As an example take a look at the following image:

I guess there are implementations of this paper that you can replace your pre-trained model with the one already exists and see what exactly is learnt by your network.

  Isn't this image a different vector?

They are different vectors. ML and DL models are for generalisation which means they should be good at test time.


How will Occam's Razor principle work in Machine learning

Occam’s razor principle:

  Having two hypotheses (here, decision boundaries) that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

In your example, both A and B have zero training error, thus B (shorter explanation) is preferred.

What if training error is not the same?

If boundary A had a smaller training error than B, selecting becomes tricky. We need to quantify "explanation size" the same as "empirical risk" and combine the two in one scoring function, then proceed to compare A and B. An example would be Akaike Information Criterion (AIC) that combines empirical risk (measured with negative log-likelihood) and explanation size (measured with the number of parameters) in one score.

As a side note, AIC cannot be used for all models, there are many alternatives to AIC too.

Relation to validation set

In many practical cases, when model progresses toward more complexity (larger explanation) to reach a lower training error, AIC and the like can be replaced with a validation set (a set on which the model is not trained). We stop the progress when validation error (error of model on validation set) starts to increase. This way, we strike a balance between low training error and short explanation.


How do I properly write scipy.stats.binom.cdf() details

>>> from scipy.stats import binom
>>> binom.cdf(0, 10000,10/19)
0.0

Binomial random variable are nonnegative. Hence you want to compute the probability that it is equal to zero which is

$$\left(\frac{10}{19} \right)^{10000}\approx 10^{-2787.54}\approx 0$$


Following your code in your other thread:

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')
t0 = time()
tf = tf_vectorizer.fit_transform(data_samples)
print("done in %0.3fs." % (time() - t0))

print("Topic modelling with LDA...")
lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=5,
                                learning_method='online',
                                learning_offset=50.,
                                random_state=0)

lda_x = lda.fit_transform(tf)
# so lda_x is your doc-topic distribution that you can use for feature vector to your SVM model.
# lda.components_ is your topic-word distribution.

Hope this helps!


How to set class-weight for imbalanced classes in KerasClassifier while it is used inside the GridSearchCV?

grid_result = grid.fit(X_train, y_train, clf__class_weight={0:0.95, 1:0.05})

FYI, per the docs fit_params should no longer be passed to the GridSearchCV constructor as a dict, but should be passed directly to fit as above.

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html


How does "sameModel" mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

It appears that "sameModel" is just an example of loading the model back without any influence on "model" from the docs.


What are the disadvantages of using deep neural networks compared to a linear model?

First of all, we should say that a single affine layer of a neural network without any non-linearities/activations is practically the same as a linear model. Here we are referring to deep neural networks that have multiple layers and activation functions (non-linearities as relu, elu, tanh, sigmoid etc.)

Second of all nonlinearities and multiple layers introduce a nonconvex and usually rather complex error space which means that we have many local minimums that the training of the deep neural network can converge to. This means that a lot of hyperparameters have to be tuned in order to get to a place in the error space where the error is small enough so that the model will be useful.

A lot of hyper parameters which could start from 10 and reach up to 40 or 50 are dealt with bayesian optimization using Gaussian processes to optimize them which still does not guarantee good performance.

Their training is very slow and adding the tuning of the hyperparameters into that makes it even slower where in comparison the linear model would be much faster to be trained. This introduces a serious cost-benefit tradeoff.

A trained linear model has weights which are interpretable and give useful information to the data scientist onto how various features play a role for the task at hand.


What is the minimum/suggested sequence length for training an LSTM?

LSTMs would run into problems beyond 500 time steps.
original paper by Hochreiter and Schmidhuber: "LSTM can learn to bridge minimal time lags in excess of 1000 discrete time steps."

Example: Your video is 60 seconds long. I would start by encoding 5 frames per second using a CNN (use resnet (SEnets have better performance but i havent read the paper) till the layer just before final affine layer which feeds into the softmax encode your images. Use these encodings as inputs in a GRU (Hence a 300 time steps for this GRU) and use its final state to make the class prediction (use a separate loss for each label?)


Backpropagation with log likelihood cost function and softmax activation

No you actually didn't really understand how softmax functions it outputs a probability distribution hence if there are 10 output neurons you will have 10 probabilities for the 10 respective classes i.e. the neuron with the highest probability will be more activated that is none of the output neurons will give 0 as output that is what softmax is it takes exponential average of every class to produce a probability distribution over k different classes here k=10.Now as you said suppose you have 10 output neurons then while back propagating the error, only the gradient w.r.t. yth output neuron is non-zero and all others are zero,this is wrong if you go and give it a read the error or cost function is calculated as follows when there are multiple neurons:

Now as you can see the cost is calculated over all the output neurons for all the training examples in the batch that is of size 'n' here hence when you do backpropagation that is when you calculate"dc/dw" and "dc/db" it includes output from all the output neurons and your statement is wrong because if gradients of other output neurons are 0 then how will your backpropagation update there weight matrices. I know its confusing but if you read his ch-2 you should be able to understand it.i have used cross entropy to explain you this but the same method will work for any cost function that you take .

As far as the derivation is considered it is quite easy just go back and study his chapter two all four equation BP1,BP2,BP3,BP4 and understand there derivation it will take some time but it is easy once you understand the composite function nature of neural networks and how to differentiate composite functions using the chain rule.


How to create a Document Categorization Classifier for different contexts of Documents

This depends on whether you have enough training data for all languages. If yes, doing language ID and language-specific models might be a good choice, especially if there is a BERT-like model available for each language.
An alternative would be, do the language ID and than machine-translate the input into English and only train an English classifier. You can use e..g, high-quality pre-trained Marian models recently published by the University of Helsinki.
Otherwise, I would use pre-trained multilingual representations (probably XLM-R that is much better than Multilingual BERT) to get representation and train a single classifier for all languages. The multilingual representations seem to have even some zero-shot abilities, i.e., the classifiers seem to generalize even for languages that are no in the training data.


Is feature scaling at all needed for a feature set with a single feature?

I understand that feature scaling is required to bring features in
different magnitudes on a common scale so the model is not biased
towards features with higher magnitudes. But if there is only a single
feature in a feature set. For example, I have a quantity (events over
time) on a time series and I'm forecasting for the future. In this
case, do I need feature scaling (normalization or standardization?) or
will it be unnecessary?

Usually we would try to do feature scalling although the idea is to remove the dependence of absolute magnitiude in affecting  convergence. it will help if the distribution is made gaussian with zero mean and unit variance. Since the gradient will be lower the stochastic descent will be a lot smoother.


Do I need an Artificial Intelligence API?

One needs to use an artificial intelligence (AI) API, if there is a need to add AI functionality to a software application - this is pretty obvious. Traditionally, my advice on machine learning (ML) software includes the following two excellent curated lists of resources: this one and this one.

However, keep in mind that ML is just a subset of AI domain, so if your tasks involve AI areas beyond ML, you need more AI-focused tools or platforms. For example, you can take a look at ai-one's AI platforms and APIs as well as interesting general AI open source project OpenCog.

In addition to the above-mentioned AI-focused platforms, IBM's Watson AI system deserves a separate mention, as quite cool and promising. It offers its own ecosystem for developers, called IBM Watson Developer Cloud, based on IBM's BlueMix cloud computing platform-as-a-service (PaaS). However, at the present time, I find this offering to be quite expensive as well as limiting, especially for individual developers, small startups and other small businesses, due to its tight integration with and reliance only on a single PaaS (Blue Mix). It will be interesting to watch this space as competition in AI domain and marketplace IMHO will surely intensify in the future.


How to shift rows values as columns in pandas?

You want pandas.get_dummies.

If you call get_dummies on a categorical column, it will output the binary dummy variables you're looking for. You should then be able to merge this with your original DataFrame on the index, or construct a new DataFrame using only the columns you want.


how to create multiple plot from a panda Dataframe

If I correctly got what you meant: you want to plot the first row of all the columns?
I guess this could work.
x= range(0,24)
y = df.iloc[0,:].values
xticks = df.columns.tolist()
I'm assuming you want a line plot

plt.plot(x,y)
plt.xticks(xticks) # to change the x ticks of the graph.
plt.show()

The '0' signifies the first row and ':' for all the columns.


T-SNE is another dimensionality reduction algorithm not mentioned in the article in the other answer. Used for VERY high dimensional data, if you have trained some embeddings for your dataset. Reference Here . Python standard library here.

cheers


Sklearn: How to adjust data set proportion during training, but not testing

Try to use predictor option class_weight='balanced' or auto. It worked really well for me for SGDClassifier in a similar situation.


How to deal with Missing Not at Random Data for k-means clustering?

I think there are two approaches you can take:

1) Square the 'homevalue'

Assuming that homevalue is a numeric you can square this value.

Then, you will increase the difference between $0$-valued homes and low-valued homes. Further, it will make the difference between high-valued homes to low-valued homes also larger.

2) Split into two features

You can also split this information into two different features:

Home value $v$, where $v \in \mathbb{R}^+$
Owns a home $h$, where $h \in \{true, false\}$ and $h = \substack{false, \quad if \, v \, equals \, 0\\true, \quad \ otherwise}$

This way, your model has a way to explicitly distinguish between people that own or not a home.

Then hopefully you have another feature indicating the lifestyle or earnings of a person that doesn't own a home.


Python : data type handling by sklearn and impact on memory usage and performance

Overall, scikit-learn is designed to be not inefficient but it's goal is not hyperefficiency. If your goal is hyperefficiency then you should switch to a different machine learning package.
The goal of scikit-learn is to provide a high level interface with machine learning and handle the implementation details "under the hood". That is why there is little documentation about how different datatypes are handled. If you are interested in how different algorithms handle different datatypes, you have to look directly at the source code. All of the scikit-learn source code is on GitHub.


NER: Extracting entities from an article

CRF is a standard for such case but bi-LSTM + CRF are said to be even better (e.g. https://arxiv.org/pdf/1508.01991.pdf). Not sure if you need POS as this is usually solved using the same techniques - may not help in the main task of entity extraction.

Depending how different are articles in specific categories and how much data you have for each of them, you may need to train each category separately and have some classifier in the beginning to decide category - like in your point 1. but word2vec-like classifier may be more robust here.


Google data studio: How to order a dimension?

Sort like this:

Hope this works, and I'm adding extra words to fulfill minimum character count.


Clarification wanted for make_step function of Google's deep dream script

What is this "something"? Is it dst.data? I stepped through a debugger and found that dst.data was just a matrix of zeros right after the assignment and then filled with values after the backward pass.

Yes. dst.data is the working contents of the layer inside the CNN that you are trying to maximise. The idea is that you want to generate an image that has a high neuron activation in this layer by making changes to the input. If I understand this correctly though, it should be populated immediately after the forward pass here: net.forward(end=end)

  Anyways, assuming it finds how "off" the result of the forward pass is, why does it try to do a backwards propagation? I thought the point of deep dream wasn't to further train the model but "morph" the input image into whatever the original model's layer represents.

It is not training. However, like with training we cannot directly measure how "off" the source that we want to change is from an ideal value, so instead we calculate how to move toward a better value by taking gradients. Back propagation is the usual method for figuring out gradients to parameters in the CNN. There are some main differences with training:

Instead of trying to minimise a cost, we want to increase a metric which summarises how excited the target layer is - we are not trying to find any stationary point (e.g. the maximum possible value), and instead Deep Dream usually just stops arbitrarily after a fixed number of iterations.
We back propagate further than usual, all the way to the input layer. That's not something you do normally during training.
We don't use any of the gradients to the weights. The weights in the neural network are never changed. We are only interested in the gradients at the input - but to get them we need to calculate all the others first.

  What exactly does src.data[:] += step_size/np.abs(g).mean() * g do? It seems like applying whatever calculation was done above to the original image. Is this line what actually "morphs" the image?

It takes a single step in the image data along the gradients that we have calculated will trigger more activity in the target layer. Yes this alters the input image. It should be repeated a few times, according to how extreme you want the Deep Dream effect to be.


Good practices for manual modifications of data

1.  Do not modify the original data

Having the original data source intact is important.  You may find that updates you make to the data are not valid.  You may also find a more efficient way to make updates and you will want to regression test those updates.

Always work with a copy of the data, and add columns/properties/metadata that includes any processed corrections.

Example, if your data is a .csv file that includes a city name that contains several misspellings:

1. Copy the .csv to a new file
2. Add a column for containing "adjusted_city_name"
3. Copy data from the city_name column to the
   adjusted_city_name column and make corrections in that column.

2. Document proposed changes

Any changes you want to make to data should be documented so that they can be replicated moving forward.

Version control and timestamp the document every time you change it.  That will help in troubleshooting at a later date.

Be explicit.  Do not simply state "correct capitalization problems", state "ensure that the first letter of each city name begins with a capital letter and the remaining letters are lower-case."

Update the document with references to any automation routines that have been built to manage data cleansing.

3. Decide on a standard data cleansing technology

Whether you use perl, python, java, a particular utility, a manual process or something else is not the issue.  The issue is that in the future you want to hand the data cleansing process to somebody else.  If they have to know 12 different data cleansing technologies, delegating the cleansing procedure will be very difficult.

4. Standardize the workflow

There should be a standard way to handle new data.  Ideally, it will be as simple as dropping a file in a specific location and a predictable automated process cleanses it and hands off a cleansed set of data to the next processing step.

5. Make as few changes as is absolutely necessary

It's always better to have a fault tolerant analysis than one that makes assumptions about the data.

6. Avoid manual updates

It's always tempting, but people are error-prone and again it makes delegation difficult.

Notes on manual processing

To more completely address the original question as to whether there's a "good" way to do manual processing, I would say no, there is not.  My answer is based on experience and is not one that I make arbitrarily.

I have had more than one project lose days of time due to a client insisting that a manual data cleansing process was just fine and could be handled internally.  You do not want your projects to be dependent on a single individual accomplishing a judgement based task of varying scale.

It's much better to have that individual build and document a rule set based on what they would do than to have them manually cleanse data.  (And then automating that rule set)

If automation fails you in the end or is simply not possible, the ability to delegate that rule set to others without domain specific knowledge is vital.

In the end, routines to do something like correct city names can be applied to other data sets.


Multi-label text classification. 3-4 labels per text, 100 labels overall

I think it's always a good idea to start with a basic option. Once you have a decent basic model, you can try to improve performance with something more advanced and you would have a baseline so you know if the performance of the advanced model is really good or just ok.
In this case the basic option would be to train 100 binary classifiers, one for every label. The text could be represented as a TFIDF vector. I would suggest removing the rare words, it can significantly improve performance. Then a traditional model like Naive Bayes, Decision Trees or SVM can be used.
The number of labels by itself is not an issue, however their distribution matters, especially if some of them have very few instances (expect bad performance on these).


How to select between models when AUC scores are similar?

Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).

Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.


Does the "laplacian" kernel used in the SVM context come from a Hilbert space inner product?

Yes, this is guaranteed by the Moore–Aronszajn theorem.

$K(\mathbf{x},\mathbf{y}) = e^{-\|\mathbf{x} - \mathbf{y}\|}$ is a positive definite kernel.  This means it is a symmetric function satisfying $$\sum_{i,j=1}^{n} c_i c_j K(\mathbf{x_i},\mathbf{x_j}) \ge 0$$ for all $n \in \mathbb{N}$, all $\mathbf{x_1},\dotsb ,\mathbf{x_n} \in \mathbb{R}^n$, and all $c_1, \dotsb , c_n \in \mathbb{R}$.

The Moore–Aronszajn theorem says that for each such function there exists a unique reproducing kernel Hilbert space with reproducing kernel $K(\mathbf{x},\mathbf{y})$.

That is, there must be a unique Hilbert space $H$ of functions $f: \mathbb{R}^n \to \mathbb{R}$ with a mapping $\phi: \mathbb{R}^n \to H$ such that $K(\mathbf{x},\mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ for all $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$, and such that $K(\mathbf{x},\mathbf{y})$ satisfies the reproducing property: $$\langle f, K(\cdot, \mathbf{x}) \rangle = f(\mathbf{x}) \qquad \forall \mathbf{x} \in \mathbb{R}^n, \enspace \forall f \in H$$

While the reproducing kernel Hilbert space is unique for a given positive definite kernel $K(\mathbf{x}, \mathbf{y})$, there can be other mappings $\phi$ to other Hilbert spaces which also satisfy $K(\mathbf{x},\mathbf{y}) = \langle \phi(\mathbf{x}), \phi(\mathbf{y}) \rangle$ but without the reproducing property.


Variation in test data accuracy is perfectly normal. Your network is fitting for your train data. If you see (big) variation in train accuracy, then you should be worried about learning rate tuning, regularization etc.
But here, everything seems to be going fine. Remember your network only fits on your train data, so it modifies its weights according to that data. Since your test data -which the network never uses for weight update- is not exactly your train data (even though it should come from the same distribution ... so test accuracy should globally improve over iterations if the network is learning), test accuracy is bound not to improve exactly at each iteration.


Tensorflow Model fitting ValueError: Layer sequential expects 1 inputs, but it received 520 input tensors

First of all look at the shape of tensors that your tf.data.Dataset returns then try to set the input_shape of the first Dense layer like:
model = keras.Sequential([
    layers.Dense(520, activation='relu', input_shape=(1, 519)),
    layers.Dense(520, activation='relu'),
    layers.Dense(520, activation='relu'),
    layers.Dense(1)
  ])

or explicitly add the Input layer
or set the number of neurons of the first Dense layer corresponding to the number of features (519)
Also read the docs they are really great:
https://www.tensorflow.org/api_docs/python/tf/keras/Sequential


I solved the problem, I added an additional conv layer with 64 feature maps and increased size of dense layer from 512 to 1024 also I trained the model with the increased number of epochs(from 50 to 150). The result is impressive! Everything as I wanted.


Think about how you would verify the model on your own. You could train it on data in the past, stopping before the present, and then ask the model to predict for a period that you already had data for. That way you can check how accurate it is (either by eye or with error metrics such as MAE, MAPE, RMSE, etc), and adjust accordingly.

cross_validation just automates this process. The first parameter you give is your trained model m (not the data). You then also give the prediction horizon - how frequently you want to predict (in your case '15min', assuming Python). You may then give an initial (how long to train before starting the tests) and a period (how frequently to stop and do a prediction). If you don't give them, Prophet will assign defaults of initial = 3 * horizon, and cutoffs every half a horizon. You then have a long running series of validations, each time predicting forward and calculating the error (you can use the other fbprophet.diagnostics tool, performance_metrics for this). This is somewhat akin to k-fold cross-validation in non-time-series machine learning.

The documentation on this is actually pretty good. See also this blog post for an example in Python.


How to deal with memory exceptions while processing large amount of data?

I had also faced similar issues when my python script was processing my 12GB and 35GB files. It used take weeks for processing and many times failed.

You need to check your mathematical operation, if it can work on subset of data. In my case, I finally was able to split the file in multiple pieces and run the mathematical operation on individual pieces. It helped improving the speed as well as better exceptional handling.

If you cant split your file, then check if your code can work in batches. Then you can read your file in chunks (in pd.read_csv you can supply chunk size) and then feed to your formula piece by piece.


You basically answered your question: if y1, y2, y3 are independent, there is no point to use a single model. If they are dependent (say, y1=1 implies that y2 is not 2), then the single model helps to exclude the non-existent case.

However, if y1 is a function of x1 and x2, and y2 is a function of x3 and x4, and x1,x2,x3, and x4 are independent, y1 and y2 are also independent.

I would start from whatever seems simpler and more logical. Based on your description of weak dependence between variables, multiple models seem more promising to me.


What is the difference between multiply and dot functions that is used to merge layer in Keras?

The multiply() function performs element-wise multiplication. For example, let us consider 1D CNN for simplicity and you pass two inputs of batch size b with a tensor length of 5, the output will be (b,5) as it's element-wise multiplication.
Let us assume two tensors of length 5 as follows: [1,2,3,4,5] and [6,7,8,9,10], the result shall be [6,14,24,36,50] as it's just element-wise multiplication.

In the case of dot(), it takes the dot product, and the dot product for 1D is mathematically defined as: a.b = sum(a_i * b_i), where i ranges from 0 to n-1; where n is the number of elements in vector a and b. (For 2-D , you can consider it as matrix multiplication). So the result shall be of length (b,1) where b is the  batch size. In case of out example, the results of dot product of [1,2,3,4,5] and [6,7,8,9,10] shall be [130]


How to implement a Fourier Convolution layer in keras?

An FFT-based convolution can be broken up into 3 parts: an FFT of the input images and the filters, a bunch of element-wise products followed by a sum across input channels, and then an IFFT of the outputs (Source).

Or as it is written in the paper:

So, for a Fourier Convolution Layer you need to:

Take the input layer and transform it to the Fourier domain:

input_fft = tf.spectral.rfft2d(input)

Take each kernel and transform it to the Fourier domain:

weights_fft = tf.spectral.rfft2d(layer.get_weights())

Note: The Fourier domain "images" for the input and the kernels need to be of the same size.
Perform element-wise multiplication between the input's Fourier transform and Fourier transform of each of the kernels:

conv_fft = keras.layers.Multiply(input_fft, weights_fft)

Perform Inverse Fourier transformation to go back to the spatial domain.

layer_output = tf.spectral.irfft2d(conv_fft)

Note: I used pseudo-code, it will probably needs some tuning for it to actually to work.


How to find impactful words affecting classification?

First of all, let's just clarify that proving causation is quite difficult. And therefore, you will mostly need to show correlation. Also, for text classification, the impact of each word is not linear. Each word impacts the meaning of the sentence relative to the other words and vice versa, therefore, you need to keep that in mind.

With that said, there are a few ways you can look at the impact of words on a classification:

Frequencies

If you have multiple classes, you could look at how frequently certain words appear in each class. If the word "good" only appears in positive sentences, it is safe to assume that it has a high impact.

This method has the benefit of being model-independent.

Feature importance

If you are using a bag-of-word representation as your input, you could look at how much the presence/absence of each feature affects the results. For instance, let's say that you can classify positive sentences with 80% accuracy with all words as features. Now, try to classify the sentences again by removing certain words. If you remove the word "good" from sentences, you can look at how the performance varies.

Or, you could do the opposite and try to simply classify each word. If you classify the word "good" as positive, it means that it is probably a word that impacts sentences into being positive. If your model provides confidence, you could use that to know how much information the word brings to the sentence.

Attention mechanism

The attention mechanism will be able to tell you which part of the input influences the results the most. This works better if you use a sequential model


Conjugated gradient method. What is an A-matrix in case of neural networks

The traditional conjugate gradient descent is an increment on the gradient descent that just takes a direction that is fully orthogonal to the previous descent direction. There is no $A$ matrix in that case.

There are different rules (you can check some in my old optimization toolbox at https://github.com/mbrucher/scikit-optimization/blob/master/scikits/optimization/step/conjugate_gradient_step.py). If I remember properly FR combined with strong Wolfe-Powell line search rule give one of the best answer. The issue is that it requires more computation, which is why line search is never used in neural networks optimization.


Evaluate Keras model with scikit-learn metrics

You could use class KerasClassifier from keras.wrappers.scikit_learn, which wraps a Keras model in a scikit-learn interface, so that it can be used like other scikit-learn models and then you could evaluate it with scikit-learn's scoring functions, e.g.:

from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.metrics import roc_curve, auc

keras_model = ...
classifier = KerasClassifier(keras_model, batch_size=32)
y_pred_keras = classifier.predict(X_test).ravel()
fpr, tpr, thresholds = roc_curve(y_test, y_pred_keras)
auc_score = auc(fpr, tpr)

You can find more info on the Keras docs, or googling around (e.g. this and this).


What kind of data visualization should I use?

First, I think you'll need to measure when you've made a typing mistake. For example, you might log each key press and then in an analysis after, look at when you press the backspace key. If you press it only once, you might consider the key you pressed to be incorrect and the one you type after to be the correct key.

This supplies you with a truth value. It would be difficult to measure anything if you don't know what would ideally happen.

In terms of visualizing this, I would opt for a confusion matrix. There are some nice visuals provided by Seaborn, but it might look like what's in this SO answer. As you can see, each letter has a high value for itself, and maybe a couple mistakes for other letters. Looking at this plot, you might say "F" is often typed when "E" is desired. The y-axis would be the letter you intended to type, the x-axis might be the letter you actually typed. This could help you see which letters are frequently mistyped. Additionally, it would be intuitive to compute ratios off of this.

If you're not interested in which keys are mistyped as other keys, you could easily do a bar chart of key frequencies. Or a bar chart where each x-tick is a letter with proportion typed (in)correctly.


Which algorithm to use for transactional data

On this data, you can perform a lot of supervised learning. If you know, supervised learning is when the machine learns with data which has labels. In supervised learning, there are two subsets. Those are regression and classification. Classification is when you predict on something which is discrete, such as male or female, or survived or not survived. On the basis of regression, you can predict non-discrete things, such as the price of a house, or GDP of a country.
Based on your dataset, I think you can do a lot of EDA(exploratory data analysis) with classification. Maybe you can predict which gender buys more. There are many things you can do with the dataset, but here are the algorithms you can use.
If you have a small dataset, Logistic Regression and Naive Bayes are the best algorithms. For starters, the k-NN(k-nearest neighbors) algorithm is the best though. If you start getting into complex data, then Decision Tree is the best algorithm.
Now after all of these, there is the most complex algorithm(which is basically a bunch of decision trees mixed together) which is Random Forest. This algorithm is for if you have a really huge dataset with many labels.
Hope this helps!


Image Classification with two very similar and random-ish classes

This is likely a combination of 1, 2, and 3. Without any data to back this claim up, 800 data points seems woefully small for the scale of problem you are trying to solve.

Assuming the problem is solvable, why not try to build a model directly onto the time series of the underlying data? What extra information are you getting from the image that you can't get from the time series? I would recommend reading up on some time series models.

Keep in mind that people using way more sophisticated methods are likely trying to build similar models, and likely are not succeeding.


Is there a way to get the sample weights that were used to fit a scikit-learn estimator?

A quick search of the source code, e.g. RandomForestClassifier's, doesn't find anywhere that sample_weight gets saved as a class attribute. I suspect that was a conscious decision: the sample weights are directly tied to the dataset, which also doesn't get saved for later use; that's why sample_weight appears in the fit method rather than the class instantiation.


In PCA, every principal component a eigen vector?

Yes, a principal component is defined as the eigenvector of the covariance matrix of the data (zero-centered).

The eigenvector with the largest eigenvalue points in the direction of the greatest variance in the data. The smaller the eigenvalue, the less dispersed data points are in the direction of that eigenvector.

I'd recommend reading on the intuition behind PCA on Wikipedia.


Build text complexity model based on complex examples

There have been many ways to measure text complexity proposed in the literature, I don't have any particular survey to recommend but google is your friend.

Many of these measures are heuristics, i.e. they work in an unsupervised way. I don't remember the details but I've seen some works using a combination of several of these measures to obtain more accurate results.

A basic way would be to be build a language model on the complex text, measure the complexity against this model for any new text and assume that if it's not similar then it's not complex, but as you rightly noticed it's not a very safe assumption.

At the most basic level, you can use the type token ratio (TTR): divide the number of types (unique tokens) by the total number of tokens. The TTR is a quite good indicator of lexical diversity, so complex text is likely to give a high value. It's a very crude measure but it's useful as a baseline: whatever system you try, if it doesn't give better results than a threshold on the TTR then it's not a good system :)


Compare job ads with a given set of categories (which each consists of terms)

In terms of data understanding, I would recommend an unsupervised approach first. For example you could ...

.. perform tf/idf vectoring and build a simple tag cloud
.. perform Latent Dirichlet Allocation to get an overview of latent topics in your data (like clustering and interpret the categories)
.. train a word embedding (e.g. word2vec) on top of your data. Then perform a dimension reduction (PCA) to visually explore your data. (You could find semantically and syntactically clusters)

After the unsupervides data exploration you could perform a text classification based on classical approaches (e.g. bag of word models) or with neural nets (=supervised).


When, if ever, is nearest neighbor classification the best choice?

One situation where NN would be ideal is if the data are sample points of a piecewise constant function. In this case, the true function is composed of a tesselation of its domain, with a constant value for all points within each division.


Does a variable with different range for rows affect the classification model

The first solution is valid. You need to keep the same unit for all rows per a given feature. But for currencies, you should be careful about the rate of conversion which might be dependent on the date of the data collection.

For the second solution, I am not quite sure about your end application but you may use it if you are trying to incorporate the effect of currency type in your model prediction. But if you are proposing it as alternative to the first solution, your model will less likely to infer the conversion rate among the various currency types.


Data Engineering Stack - collect, transform and visualize geospatial data

You can do this, waaay to easier to what you are currently doing.

For the data scraping, use whatever makes you happy. In my case I will use Uipath or just python, depending on the complexity. But this is up to you, you just want some dataset in a format that suits you.

Once you have your data, you want to visualize that. This is a classical data science task. I am from the python army so I will suggest going to python. If you are good with Java the transition will be like slicing hot butter.

There are some good libraries that will help you greatly.
Here I suggest some packages that I have used in the past and they will help you greatly.

Folium
Geoplot
Intro to folium (again)

In my case, I will probably stick to Foliumm since there is plenty of code to re-use on the internet and it is a piece of cake.

This is my personal opinion on what I will do. There might be other tools and languages...


I don't know that I'd call this a machine learning problem per se...though others may disagree, this sounds like an optimization problem.

Your description is vague, but let's pretend it's a guitar that you're trying to tune and you have 6 strings. The trick is to define an objective function that quantifies, with a single number, how 'good' the tuning is. (The program can independently manipulate each string's tension). Perhaps it's some deviance in power from an ideal FFT, and you sum the deviance from each string.

Optimization routines such as simplex, sub plex, Nelder-Mead, GA, etc., are ways of manipulating the tension in 'smart' ways that obviate the need for a brute force (or 'grid search') solution. So long as you can define an objective function, you can start trying it out, though the local max/min problem will need to be dealt with.

I'd look into some tutorials on minimization algorithms


(RL Curiosity) - "Exploration by Random Network Distillation" - what's the benefit?

Both papers are attempting to incentivize exploration for reinforcement learning agents.

Curiosity-driven Exploration by Self-supervised Prediction does that by adding an inverse model, increases the complexity of the loss function of the overall model.

Exploration by Random Network Distillation does that having two separate models. One of the model tracks and learns the value of exploration.

There are pros and cons to both approaches. It is easier to train a single loss function/model. However, two models that learn to interact are more flexible.


Why deep feedforward neural networks are restricted to solve classification problems?

There is regression by neural networks. For example, have a look at this analysis (Deep Regression), RBF neural networks, and General regression neural networks. Also, linear regression can be implemented using a neural network.
If using neural networks for interpolation, this may work very good. However, if regression is used to extrapolate, results may be unsatisfactory.
Note also that the universal approximation theorems say only that a contineous function can be approximated within a compact set (thus in particular bounded set!).


Preserving explained variance while reducing dimensionality

$f$ is not defined for $\mathbb{R}^M$, so as well as reducing $x \in \mathbb{R}^N$ to $y \in \mathbb{R}^M$ you are creating a function approximator $g(y) \approx f(x)$.

I suggest using a neural network autoencoder with N dimension inputs and M dimension "bottleneck" layer. You will need to scale inputs and outputs. Using a standard auto-encoder, you will have to measure the fraction of preserved variance in $f(x)$ afterwards. The auto-encoder is not directly going to do that for you, instead it will, like PCA, attempt to use y to encode x (unlike PCA, it may do so nonlinearly).

You could take this further if $f(x)$ is differentiable. Instead of usual mean-squared error as the loss function, you could use $\frac{1}{k}\sum_{i=1}^{k} (f(x_{i}) - f(\hat{x_i}))^2$ - that will encourage the NN to preserve variance in $x$ that matters to $f(x)$. You will have to figure out the gradient function for this analytically. Also bear in mind I have not tried this, just attempted to match your requirement to some theory.

In addition, you will now have $g(y)$ as it will be $f(\hat{x})$. You can generate any $y$ from $x$ by running the first half of the autoencoder, and can generate any $\hat{x}$ from $y$ by running second half of the autoencoder.

You might also be able to adapt t-SNE by using your function $f(x)$ to generate distances that need to be preserved when reducing dimensions.


Image Recognition Model with Adaptive Learning

Suppose during testing it suggested with accuracy 40% for an image containing person of say class A. I would like to add this image back to the models training dataset and train the model adaptively, so that it improves the accuracy for this person. That would mean transferring an image from the test set to the training set.

Whilst adding data that appears important to your task to the training set will often improve performance, you have a major problem here. You must base your measure of performance on the test set. Altering the test set can invalidate your test results so far. And of course if you take a problem item out of the test set, your performance on the test set will go up even if you don't use the it to train with, because it was a result that was bringing down your average. So, you might improve your classifier, but at the expense of not knowing whether you have improved the classifier.

A good default position here is that the test set should not be altered. Once set aside, that data is for measuring performance of your tuned model, and should not be used for anything else, to avoid this measurement problem.

However, there are a couple of things you could do:

Error analysis

Look at the failing cases in the test set and try to figure out why they are being misclassified. For an image classification task, you might be able to figure out by viewing the image directly. Is there unusual lighting, pose, any different foreground or background objects? In the case of face recognition, has the subject got different hairstyle, makeup, hat, glasses or clothing?

If you spot something that's a one-off difference in your problem image, then that may be a sign that you need some more variety in your training data and should collect more. You should not collect it from the test data though, because then you will not be able to tell how well your classifier is doing at the task any more.

It is possible that investigating a failed item will show up something that means you want to alter or remove the test data item. This would happen for example if you decide it is not representative of how you want your model to be used. For instance, all the other images in the data set are from CCTV - which is how you intend it to be used - and the bad example is from a studio photo shoot. In that case, it may be worth the effort to clean up your test data and re-evaluate your best models so far.

K-fold cross validation

You can use k-fold cross validation to gain more accurate assessments of perfomance when tuning hyperparamaters (number of neurons, layer architecture). This doesn't technically replace a test set for measuring final performance, but it's not too bad a fallback position. An extreme version of this would be leave-onme-out cross validation.

In theory you could select your best hyperparameters using k-fold cross validation, have a good idea that this would be the best you could get, and a rough idea of what the generalisation performance was. Then train a final model using the same hyperparameters and all of your data.

You may be able to get away with this if you are happy to not know (or not need to report) an unbiased performance metric, but be reasonably certain you have close to the top performance out of the models that you have investigated.

Keep the test set as-is, do both the above, maybe collect more training data

I would recommend instead this as the best approach if you can afford the time and effort. For better or worse, you have set your goals for the first attempt at your project, when you selected the test set.

You can search harder for hyperparameters that generalise the best, using a small amount of data, by using k-fold cross validation. You can try to figure out why your accuracy is low by looking at the errors from the model and finding ways to fix them without using the test data in the training set.

Eventually you will have your best model (according to k-fold CV on the training data), and a measure of how it performs on the test set. The project is then done. Any further work to get even better accuracy would require some kind of re-thinking or start of a new project.


Cluster analysis is not supposed to produce paritions of equal size. It is meant to discover structure in the data.

If the majority of objects is highly similar, then this majority is supposed to be in the majority cluster.

Consider all your data is identical. Any clustering algorithm producing more than one cluster has failed, in my opinion...

So you may be using the wrong class of algorithms for your problem.


Value Updation Dynamic Programming Reinforcement learning

The probabilities you describe refer only to the go-north action. It means that if you want to go north, you have 80% chance of actually going north and 20% of going left or right, making the problem more difficult (non-deterministic). This rule applies to every direction. Also, the formula does not tell which action to chose, just how to update the values. In order to select an action, assuming a greedy-policy, you'd select the one with the highest expected value ($V(s')$).

The formula says to sum the values for all possible outcomes from the best action. So, supposing go-north is indeed the best action, you have:

$$.8 * (-.1 + 0) + .1 * (-.1 + 0) + .1 * (-.1 + 0) = -.1$$

But let us suppose that you still don't know which is the best action and want to select one greedily. Then you must compute the sum for each possible action (north, south, east, west). Your example has all values set to 0 and the same reward and so is not very interesting. Let's say you have a +1 reward to east (-0.1 for the remaining directions) and that south already has V(s) = 0.5 (0 for the remaining states). Then you compute the value for each action (let $\gamma = 1$, since it is a user-adjusted parameter):

North: $.8 * (-.1 + 0) + .1 * (-.1 + 0) + .1 * (1 + 0) = -.08 - .01 + .1 = .01$
South: $.8 * (-.1 + .5) + .1 * (-.1 + 0) + .1 * (1 + 0) = 0.32 - .01 + .1 = .41$
East: $.8 * (1 + 0) + .1 * (-.1 + 0) + .1 * (-.1 + .5) = .8 - .01 + .04 = .83$
West: $.8 * (-.1 + 0) + .1 * (-.1 + 0) + .1 * (-.1 + .5) = -.08 - .01 + .04 = -.05$

So you would update your policy to go East from the current state, and update the current state value to 0.83.


A CNN will learn to recognize patterns across space.  So, as you say, a CNN will learn to recognize components of an image (e.g., lines, curves, etc.) and then learn to combine these components to recognize larger structures (e.g., faces, objects, etc.).

You could say, in a very general way, that a RNN will similarly learn to recognize patterns across time.  So a RNN that is trained to translate text might learn that "dog" should be translated differently if preceded by the word "hot".

The mechanism by which the two kinds of NNs represent these patterns is different, however.  In the case of a CNN, you are looking for the same patterns on all the different subfields of the image.  In the case of a RNN you are (in the simplest case) feeding the hidden layers from the previous step as an additional input into the next step.  While the RNN builds up memory in this process, it is not looking for the same patterns over different slices of time in the same way that a CNN is looking for the same patterns over different regions of space.

I should also note that when I say "time" and "space" here, it shouldn't be taken too literally.  You could run a RNN on a single image for image captioning, for instance, and the meaning of "time" would simply be the order in which different parts of the image are processed.  So objects initially processed will inform the captioning of later objects processed.


Estimating class prevalence in unlabelled data after predicting labels with a binary classifier

There's a domain named quantification that deals with this kind of problem. It aims to create "quantifiers" (instead of classifiers) that will focus more on estimating the prevalence of a class in a population rather than on individual classifications.
An easy approach is "adjusted count" (AC), but there are other (potentially better) approaches. You can find more in this paper or this one.

Basically, the idea of AC is:

1.1) Learn a binary classifier from the train dataset

1.2) Estimate the False Positive Rate (fpr) and True Positive Rate (tpr) from the training set, using cross-validation

2) Estimate prevalence of the test set based on the observed prevalence in the test set, corrected by estimated fpr and tpr (I guess the ideal is to have different test sets with different prevalences)

That way, you can estimate the prevalence of your sample based on the fraction of predicted positives that are actually positive, and the fraction of predicted negatives that are actually positive (and I guess you can easily compute confidence intervals).

The good thing with this is that your model will be way more robust to a change of prevalence in a population, rather than if you just count the positive and negative instances. (All this is explained better in the link papers)


How to predict on part of image after training on other part of image?

As you seem to plan to build your own character classifier, I assume that you are happy to do some programming but just want to avoid heavy graphics stuff. A practical approach would be to use tesseract's bounding boxes just as a guide. Look to the left and right for more characters. Knowing where the rows are and what height and width the characters are is a good basis. An extension of rows gives you the upper and lower boundaries of the new candidate boxes. To find the left and right boundaries, you can project the pixels to the baseline of the row and look for positions with a low number of black pixels. A second factor for the boundary decision can be the expected width, which should be somewhere between the width for "I" and "W". A third factor can be the letter/number that your classifier recognises and how well this fits into the context (e.g. use character language model).


A simple solution is to do the following:

temp = tf.get_collection("my_important_ops")
print(temp.get_shape().as_list())

Then you can evaluate the outputs.


Multiple-input multiple-output CNN with custom loss function

Looks correct. When u use same branches in NN architecture - this name siamese-type nn. U may see there how to implement it in PyPorch. In my opinion, PyTorch must easiest framework for this task. Also implementation of encoder-decoder network u can find there.

This is frequently using technique when u need to compute, as example, distance between few inputs. But i'm had doubts about concatenation part. Witch dimension u need for outputs? If same as input, then all looks fine. If different dim, then just add fully connected layers after last convolution.


It dates back to 1998, so most likely has been abandoned, or "acquired" by Microsoft as the creator currently works there and has done since publishing that research.

See https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/ists97.pdf
and http://research.microsoft.com/en-us/people/cyl for the author. Maybe you could try to contact him.


The way you have set your DQN up, it is designed to solve just one maze at a time. It has not (and cannot) learn to solve mazes in general, because it has no access to data about the layout of the maze, and a basic DQN agent has no capability to memorise layout seen so far.
You could view the training process as general algorithm for "solving the maze". That is, given a new maze, you can take your written agent, run it for a while in training mode and it will produce a policy that solves that maze. You may be able to tune or adjust the algorithm so that it does it the most efficiently in terms of time or number of steps. It will not be able to do so as well as a hard-coded maze-solving algorithm, because DQN and Q-learning is a very generic learning algorithm that takes no advantage of consisent logic seen in mazes.
If you want to train a more generic maze-solving agent, this becomes a harder problem to solve using reinforcement learning, but it is achievable. There are two key - and linked - assumptions of the theory behind Q learning that you would need to address before you could make a general maze solver that coped with variations such s moving the walls, but without needing to be re-trained:

The Markov property is assumed for the state variables, which means that the state description contains all the relevant information about future state transitions and rewards. Without some way to know where the walls are, or at least note the position of walls it has seen before, the agent does not have access to a state representation with the Markov property.

There is at least one deterministic optimal policy. If you are intending to withold information from the agent (and not give it capabilities such as memory in order to construct missing information), then there may not be a deterministic optimal policy.

So it is possible to use reinforcement learning to teach an agent about solving mazes in general, and gain the ability to attempt to navigate out of a new unseen before maze. If that is of interest to you, you need to first decide what capabilities and knowledge about the environment the agent is allowed to observe. E.g. what specifically interests you about writing a maze solver?
Which approaches you could use to make a more general maze solver depends critically on how you want to frame the problem. For instance it is entirely valid to give the solver an image of the maze as input if you want the agent to learn to solve mazes in a similar manner to a human solving a printed puzzle.
There is one common factor that you will need for training a more generic maze solver - you will want to train on many example mazes taken from the population of all possible mazes. Training on just one maze as an example will typically be about as successful as training a supervised classifier or regression task on one example. Adding many more mazes, or better a maze generator, and training on many of them is likely to make training time much longer than before.


with ML/DL model Is possible predict numbers of items required?

The first thing to do is to reorganize the data in a way which suits the problem that you're trying to solve: since the goal is to predict the number of each type of vehicle by month, your data should contain a column for this number.
Here you need to count the number of rows for each month and each type of vehicle so that you obtain something like this:
v_type   district    month number
Advanced District 1  jan   1
General  District 1  jan   2
Advanced District 2  jan   0
General  District 2  jan   3
Advanced District 1  feb   2
General  District 1  feb   3
...

After that you will probably need to represent the month (and year) in a way that a regression algorithm can use, typically an integer starting at month 1.


Predict which user will buy with an offer - discount

You can use Decision trees for a single model prediction for both the set of users.

A good start would be to first read up on Decision Trees and their applications.

You can include the offer as a decision(as a boolean in this case).

buying with offer and buying without offer can be the decision criterion.

You can, in fact go ahead and put in the offer values also. For example,

offer>10% and offer <10%


See, your conclusions appear to be correct, your statistical results explain your conclusions , i would say statistically speaking even with a P-value of 0.055 you still have an error rate of 15 to 25% that you may be accepting a wrong null hypothesis and maybe your alternate hypothesis is right , so combining knowledge of statistics with data science i will suggest you to use catboost(if you have some categorical features in your dataset it will give you a strong baseline prediction before you start tuning it) otherwise random forest is good any day and if you have a small dataset (e.g. less than 1500 instances in your training data) i would suggest use random forest any day over MLP because MLP based classifiers need more data to classify better with less data they tend to become bias towards your training data.But if you have a large enough dataset you can actually use MLP based classifier try some basuc feature selection techniques such as RFECV or feature_importanece_ function you can use contained in every model after you train it . And if at the end of the day you are still confused to which model to use i would suggest use all of them stack them and you will have something better.Making these decisions solely on the basis of statistics is tough because statistics puts everything into an abstract form that is a whole dataset being represented by 1 number thus it is always prone to certain errors thus i always prefer to combine my data science experiments with my statistical results and then only conclude about my results.Also while using statistical analysis a lot of things depend on your dataset i.e. is it big enough to draw meaningful conclusions out of it what might be the sampling error and many more things .Hope this helps.


Are there any statistical/otherwise tests that I can run to understand the quality of the labeling I get from my code.

Yes. You can treat your automated labelling as a model in its own right (it essentially is an "expert" model, that takes in additional features). Collect ground truth data with known accurate labels, and use a metric such as accuracy, AUROC, F1 score to determine how well your expert model works. Usually you should pick one metric that makes sense to you based on the problem you are trying to solve. Most statistics libraries will allow you to assess these metrics on arbitrary data, so it should be possible to find the relevant part of the API and feed it your synthetic labels to compare with ground truth data you have collected.

But wait! That is exactly what you would need to do in order to train and assess an ML model. About the best you can hope is that you will need less ground truth data to assess your expert than you would to train the ML version from scratch. For complex ML models such as deep neural networks, this could well be true.

Note that you may hit a limitation from this approach. Say that you assess from ground truth data that your expert is 85% accurate. You will not be able to use it to train an ML solution that is better than 85% accurate, because the ML will learn to copy the expert's predictions as best it can.

One thing to watch out for - if you analyse why the expert got some values wrong and then adjust it to give correct answers for those labels, you will need to remove the labels you used to do this from your test data - and ideally collect more, or take more from some unused pool, to replace them. Otherwise you will fit your expert to the test data and get incorrect reporting of how well the expert generalises to unseen data. Same reasoning means you should not use tweets and labels that you analysed when developing your expert model in order to test it.


Document Layout Analysis - state-of-the-art?

After some research, I came across ICDAR (International Conference on Document Analysis and Recognition), which is taking place biannually and seems to be the most complete and up-to-date source for state-of-the-art document analysis algorithms.

ICDAR is hosting a competition in different subfields each year. At the time of this writing, the latest results from ICDAR 2015 have been published as Competition on Recognition of Documents with Complex Layouts –
RDCL2015. (Search online for a PDF version.) This paper describes state-of-the-art document layout analysis algorithms, and compares their performance to both open-source and commercial solutions.


does xgb multi-class require one-hot encoding?

As Majid stated in the comment, using AUC is causing this error as normally ROC curves are calculated for binary classification. Try removing the eval_metric line and your code will run properly. That or removing the watchlist and early stopping options.

The eval metrics you need to use for multiclass are either merror or mlogloss which are the only ones specific for multiclass in the xgboost documentation.


What is the Best NoSQL backend for a mobile game

Some factors you might consider:

Developer familiarity: go with whatever you or your developers are familiar with.  Mongo, Couch, Riak, DynamoDB etc all have their strengths but all should do ok here, so rather than going for an unfamiliar solution that might be slightly better go for familiar and save a bunch of development time.

Ease of cloud deployment:  for example, if you are using Amazon AWS, then DynamoDB is likely an excellent choice.  Sure, you could use Mongo on AWS, but why bother?  Other cloud providers have their own preferred db, for example if you are using Google AppEngine, it makes sense to use BigTable or Cloud Datastore.

Your use case seems both well suited to NoSQL and not very challenging since your data has a natural partition by user.  I think you'd be technically ok with anything, which is why I'm mainly covering other factors.


Dealing with events that have not yet happened when building a model

This setting is common in reliability, health care, and mortality. The statistical analysis method is called Survival Analysis. All users are coded according to their start date (or week or month).  You use the empirical data to estimate the survival function, which is the probability that the time of defection is later than some specified time t.

Your baseline model will estimate survival function for all users.  Then you can do more sophisticated modeling to estimate what factors or behaviors might predict defection (churn), given your baseline survival function.  Basically, any model that is predictive will yield a survival probability that is significantly lower than the baseline.

There's another approach which involves attempting to identify precursor events patterns or user behavior pattern that foreshadow defection. Any given event/behavior pattern might occur for users that defect, or for users that stay. For this analysis, you may need to censor your data to only include users that have been members for some minimum period of time. The minimum time period can be estimated using your estimate of survival function, or even simple histogram analysis of the distribution of membership period for users who have defected.


Feature Selection (FS) methods are focused on specializing the data as much as possible to find accurate models for your problem. Some of the main issues that drive the need for FS are:

Curse of dimensionality: Most algorithms suffer to grasp relevant chacteristics of data for a specific prediction task, when the number of dimensions (features) of the data is high, and  the number of examples is not sufficiently big. Check here some more detailed explanation
Correlation between variables: Typically, the presence of highly correlated pair of variables can cause ML algorithms to pay to much attention to a particular effect that is "over-represented". For this reason many FS methods address the reduction of this correlation. Reducing the number of correlated variables oftenly increases the model predictive power.
Latent features: Although specific variables might be highly expressive for your problem, a lot of power can be achieved when finding "latent features", such as linear and non-linear combinations of the original variables. Here there are hundreds of approaches, from PCA to Neural Networks. Independently from the approach (and statistical assumptions) the idea is to create new features that condense the information of a bigger set of features into a smaller one. Hopefuly the new set of features is more representative, and being smaller could be more easily learnt.

Feature selection does not necessarily improve the predictive quality of the model. Reducing or transforming the features might lead you to loss of information and then a less accurate model. Is an open and complex field of research. However in many cases it becomes quite useful. It will depend on how good and different are your original features for describing the target variable.
If you look into bionformatics, you'll see people dealing with thousands or even millions of features, while having only hundred of examples. Here feature selection becomes increasingly relevant.

PS: Most commonly I've seen the term "feature selection" used for the creation of compound features, as most examples I've mentioned, while Feature Extraction term is used for actually removing specific features from the dataset without taking into account is relation with the target variable.


Encode the days of week as numeric variable

The answer is as follows:
If you are using a Linear Regression Model - then it will assume that Sunday (6) is bigger (or carries more weight) than all other days in the week. If that assumption makes sense for your problem domain - you are fine.
If you are using a model such as Random Forests - then it doesn't really matter (meaning you are good). Random Forests are typically scale invariant - They do not assume higher weightage for large values - They will figure out how best to split up the numeric values (0 - 6) to meet your objective.


So, the question is how to model the following problem: input a sub section of a sequence, then output the rest of the sequence. This is a sequence-to-sequence problem.
For this, as Derek O has eluded to, I would suggest a encoder-decoder architecture. In the encoder, you would encode your input sequence (using RNN/LSTM), which will give you a "hidden representation", then pass your hidden representation through a decoder (RNN/LSTM) to then decode the hidden representation into the resining sequence of integers.
Here is an article that provides more detail around encoder-decoder models: https://towardsdatascience.com/understanding-encoder-decoder-sequence-to-sequence-model-679e04af4346


Transformation of categorical variables (binary vs numerical)

While using one-hot (binary) encoding certainly takes more space, it also implies an independence assumption among the data. On the other hand, using integers such as 1, 2 and 3 implies some kind of a relationship between them.

The problem that you mention of linear increase in size with one-hot encoding is common and can be treated by using something such as an embedding. An embedding also helps define a sense of distance among different datapoints.

https://en.wikipedia.org/wiki/Word_embedding


I work in a restricted environment and I've heard of others that are worse than mine, but no pandas??? Geez, that's rough!

You could do this by reading through the file, line-by-line, column by column and then applying a series of your own functions to test/fix the data in each cell. But you would have to have all of the defined needs well ahead of time and then write the methods to get each data point in a methodical fashion.

So it can be done but you're doing a massive end-run around pandas to get there.


How to arrange the image dataset in CNN?

Directory structure like in dogscats/.(atleast I kept it this way)

dogscats

    |-- train

          |-- cats

                |-- catpic0, catpic1, …

          |-- dogs/

                |-- dogpic0, dogpic1, …

    |-- valid

          |-- cats

                |-- catpic0+x, catpic1+x, …

          |-- dogs

                |-- dogpic0+x, dogpic1+x, …

    |-- test

           |-- catpic0+x+y, catpic1+x+y, dogpic0+x+y, dogpic1+x+y

Becareful With the naming of the files also..

Also note that you will need a mapping of image names and classes as well like in a CSV or something...

Also there is no globally accepted directory structure, it completely depends on the API you will use...

What is a good train/validation/test split?(depends on your dataset size)

can do $80/20$ (train/validation)

if you have or are creating a 'test' split, use for (train/validation/test):

can do $80/15/5$
can do $70/20/10$
can do $60/20/20$

Remembering that sole aim is to generalize eventually on test sets...


How to create a Time Series Training Dataset with variable sequence length

I solved it the following way:

zero padding all time series that are smaller than the longest one
adding a Masking()layer with masking_value=0. which ignores all zero values before feeding the network.

The part of the model with the Masking layer looks like the following:

model = keras.Sequential()
model.add(layers.Masking(mask_value=0., input_shape=(None, 1)))
model.add(layers.LSTM(100))

Additional Info: Implementing the Masking Layer for a Model with two separate Inputs was kind of unconvinient with a the Keras Functional API, therefore i implemented that part of the model with the Keras Sequential API and connected it to the rest of the model which is implemented with the Functional API.


How to choose the features for a neural network?

A very strong correlation between the new feature and an existing feature is a fairly good sign that the new feature provides little new information.  A low correlation between the new feature and existing features is likely preferable.

A strong linear correlation between the new feature and the predicted variable is an good sign that a new feature will be valuable, but the absence of a high correlation is not necessary a sign of a poor feature, because neural networks are not restricted to linear combinations of variables.

If the new feature was manually constructed from a combination of existing features, consider leaving it out.  The beauty of neural networks is that little feature engineering and preprocessing is required -- features are instead learned by intermediate layers.  Whenever possible, prefer learning features to engineering them.


If I understood correctly you have images similar to a heat-map like Fig 6:

Well, non-natural images related applications can benefit from DNN as much as natural images, but you are unlikely to be able to use any transfer learning technique.

If you have this heat-map-like images (please consider posting some samples) you can try a simple CNN structure. Since you are a starter you could benefit from using this modeler called Ennui and outputting the source-code in Python or Julia, just remember to take a quick class on CNN to understand your options. For a complete course see Stanford's YouTube playlist on CNNs.

Some papers related to it:

Inverse Design of Airfoil Using a Deep Convolutional Neural Network
Application of Convolutional Neural Network to Predict Airfoil Lift Coefficient
Deep Learning Methods for Reynolds-Averaged Navier-Stokes Simulations of Airfoil Flows

You can inspire your models architecture by theirs

If you want a book to cover NN for deep learning try Goodfellow's.


SVM behavior when regularization parameter equals 0

First, the remark from the wikipedia article deals with small (positive) values of $\lambda$, not $\lambda=0$.  Indeed, if $\lambda=0$, then every separating hyperplane achieves the minimum score of 0.

If the data is linearly separable, then taking $\lambda$ small enough that the first term dominates ensures that minimizing the loss will require taking a separating hyperplane to zero out the first term, and subject to that minimizing the second term is equivalent to the original hard SVM.


Is always converting a input vector into matrix and apply cnn's good idea?

This is a job for dimensionality reduction techniques. PCA is a simple and intuitive method that you should probably try first. If you find it is insufficient or ineffective, you could try using an autoencoder.
As you said, using a CNN would imply a locational relationship between variables. If such a relationship does not exist, the CNN could have difficulty finding suitable weights. It is possible that forcing the CNN to link seemingly unrelated variables could help the model generalize better or reveal underlying structure that you cannot perceive, but there are other problems that you will run into. For example, CNNs generally use pooling layers as a form of nonlinear dimensionality reduction. However, if your variables are completely independent and unrelated, you would simply be throwing away possibly critical information using this down-sampling technique.
So while it might be feasible to use a CNN, you should probably use the right tool for the job.


How can I get students to stick with data science?

I have not tried this yet, but I have decided to try it on a friend who is a software engineer. He is so used to picking up skills on the fly and I am concerned he will lose interest since data science takes a bit longer to learn.

I suspect a top-down-approach, like Jeremy Howard's fast.ai, will have an easier time to keep hobby students interested. I think people that express interest often come for the fancy deep stuff and getting them hands on with that will probably make it feel more worthwhile for them. Then when they already have toys to play with you can slowly progress into the fundamentals.

I don't know if it's the "best" way to learn data science, but learning the wrong way is better than losing interest and quitting.


When a dataset is huge, what do you do to train with all the images on i t?

A couple of options:

Rent a bigger computer on a cloud service.
Move to a distributed computing framework (e.g., Spark).
Use a data loading function that only loads the needed data.

Option #3 is the simplest. Most training does not need all of the data in memory at the same time.
Dask can be used for this type of image loading .


Naming the Axes of my Graph on iPython Using Matplotlib.pyplot

When you use matplotlib's plot function, it holds an object behind the scenes for you. You can change this object with more calls to plt and then only once everything has been done should you plt.show() the graph.

Here is a simply example that does what you want:

In [1]: import matplotlib.pyplot as plt
In [2]: import numpy as np

In [3]: x = np.linspace(0, 10, 100)
In [4]: y = x ** 2

The following lines change the plot object, adding the axes labels - but we don't show it until all are complete...

In [5]: plt.plot(x, y)
Out[5]: [<matplotlib.lines.Line2D at 0x7f3c896bb9b0>]

In [6]: plt.xlabel("Time")
Out[6]: Text(0.5, 0, 'Time')

In [7]: plt.ylabel("Speed")
Out[7]: Text(0, 0.5, 'Speed')

Now we are done, so show it:

In [8]: plt.show()

Have a look here for a more thorough demo, which also shows the object explicitly and lets you better understand what is going on.


What is the best practice for tuning hyperparameters using validation data?

If you're seeing performance that is much better on the validation than the unseen test data, then that is suggestive of some sort of overfitting or, if not, that the data do not come from the same distribution. That could mean that your test images are very different from the training and validation data, for example.
First, I'd double check the data to make sure the train, validation and test sets are definitely distinct, and that all three sets have roughly the same number of positive and negative examples. If this isn't obviously the problem, then most likely I'd guess that you are overfitting the hyperparameters to the validation dataset.
Using cross-validation would most likely help you to see whether this is the case, as you'd see more dramatic variation between each fold if the parameters were overfit towards one specific validation fold.
If you repeat the process on the test dataset, then that data is no longer "unseen" and is part of the model, even if you just use it to tune hyperparameters. You should keep the test dataset completely separate from the model building process so that when you come to measuring model performance, you get a true estimation of what would happen on unseen data.


Both Spark and Flink are designed to process data in batch or stream over distributed environment.

Flink primarily being defined as its ability to process streaming data in real time and being considered as good option for processing low data latency data and high fault tolerance on distributed systems on a large scale.
However, Spark is utilized predominantly in the case of batch processing over distributed environment.
Spark do have streaming API on the other hand but Flink is getting more traction for executing this need.
Flink can be utilized on local JVM,  standalone, Yarn and cloud similarly Spark can be deployed over local, standalone, Yarn, Mesos.

There are many more difference between these two. I found some really interesting answers for your question on Quora, have a look-

Differences between Apache Spark and Apache Flink - 1
Difference between Apache Flink and Apache Spark - 2
Difference between Hadoop, spark, flink

Hope it helps!


How to determine x and y in 2 dimensional K-means clustering?

I see no problem with the example of clustering on two numerical attributes like height and weight like in the example.

The only thing I can think of is somewhere lost in translation, your professor was trying to explain the concept of reducing many dimensions (attributes) to two and then clustering on those derived dimensions. That is a common technique for trying to visually spot clusters in a complex data set.


suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector

Why isn't it very very simple?

The problem description is simple, but not in a way that is accessible to the standard RL algorithms. First of all, the analogy of switches and "corresponding" has no meaning to the RL. Secondly, the low complexity is not detectable and is outside of the assumptions of RL agents in general.

Reinforcement Learning is based on an internal model of the problem as a generic Markov Decision Process (MDP). The RL agent has to discover, by trial and error, what the MDP does, typically starting with no knowledge or assumptions other than the set of states and allowed actions. The agent needs to discover enough about the MDP to make optimal actions, whilst the only assumption is that the environment is a MDP.

The simple switch problem can definitely be modelled as a MDP. The agent then has to discover the simple configuration in the toy problem. What makes it harder than the problem description:

State/action space. There are 1024 states and 10 actions. There is no a priori reason to link any action with any outcome, so to be thorough the agent has to try them all. A minimal search would be across 10,240 state/action pairs.
State transitions. There are 1024 states where an action can be taken, and a terminal state. So there are 1024 * 1025 = 1,049,600 possible state transitions for every action. In your case, each state/action pair has one outcome, and there are roughly 5,000 different transitions (because half of them are to the terminal state).
The agent does not know that the state transitions are deterministic and sparse. It can only discover that by trial and error. For instance, perhaps in a particular configuration, taking one particular action would cause some bits to re-set. Maybe that only happens one time in four attempts. Once you add the possibility of arbitrary probability distributions here, then the number of potential MDPs is infinite. RL algorithms are designed to cope with this, still learning a good approximation of an optimal policy eventually. There is no simple way to make them dumber and faster against a simpler environment without encoding some of your knowledge about that environment.
The agent does not know that the rewards are deterministic and simple. Again it can only discover this by trial and error. To the agent the different state vectors and actions are neither similar to each other (all switches and bits) nor have any association (action 0 is not associated in any way with bit 0). Even after the associations are observed, they are not encoded in a way that the agent can reason by analogy. Again, once you add the possibility of arbitrary probability distributions for reward, then the number of potential MDPs is infinite.

  Suggest a reinforcement learning agent that will learn to efficiently switch on a bit vector

You have made things a lot harder for yourself by trying a policy gradient method with a neural network function approximator. Although this probably can be made to solve your problem, it is an advanced technique, and also known to be unstable and hard to train. It might be a good choice when the relationship between the state, transitions and rewards was very complex, and there were more states possible with meaningful similarity between them (in a numerical/statistical sense, not in terms of analogy, the neural network still won't find the analogy with switches and numbered bits). Even then, it will learn slowly and cautiously, needing to repeat state/action pairs multiple times in order to learn that the transitions and reward are reliable.

Unless there is some good reason for you to stick with a policy gradient method, I suggest using a tabular algorithm (i.e. no function approximation, just a table of action value estimates) and something like single step Q-Learning. That has the advantage that because you know the algorithm is deterministic, you can set a high learning rate and it will remain stable. In fact Q-learning will probably learn an optimal policy in much less than 10,000 episodes. However, initially it will learn just one of the many possible optimal policies and tend to stick with it  - i.e. some permutation of taking each action 0 to 9, once each, but always in the same order because it has learned that is safe.


Try the following:

data sample;
Input ID name $ Dob;
informat DOB date9. ;
format DOB mmddyy10. ;
datalines;
1 abc 22jan1996
2 xyz 25aug1996
;
run;
Proc print data = sample;
run;

I have added the informat line. The problem with the original code is that you have not told SAS what is the format that DOB will come in. In this case: 22jan1996 is of date9. format, so I added the informat telling SAS that the data will come in this way.

The format DOB line tells SAS to display the data as mmddyy10. which makes 22jan1996 look like 01/22/1996.

Lastly, the . before the ; in the format/informat lines specifies that it is a format type. There are character formats: $8. or numeric formats Best12. etc...


Does PMML support probability calibration?

I'm not too familiar with PMML, but probability calibration (or at least, the most well-known methods, Platt scaling and isotonic regression) can be viewed as a stacked ensemble, with the output of your model being fed into a univariate regression model.  PMML appears to support ensembling; see the fourth example at http://dmg.org/pmml/v4-3/MultipleModels.html


NoSQL Comparison - Is this part of my Job?

I am not very familiar with Redis, but as far as I understand the two are based on different principles and used for different things. Redis is in-memory while HBase is not. They are solutions to different problems. Maybe you should start by describing what kind of problems they are suitable for.


It might be not the best approach, but without knowing the data its hard to be precise.
If you have no ideas where to start, I can suggest trying this things first.

If you want to get the "impact" of the holiday, try to one-hot encode "holiday" variable, and use some simple model that is interpretable. You need to undestand how each holiday-feature impacts the revenue, maybe linear regressiong is a good way to start and move on to more advanced algo`s.
You can try to play with Facebook`s Prophet - its an package for timeseries predictions, wich shows the daily impacts by default. Here is the link: https://research.fb.com/prophet-forecasting-at-scale/ . And link to python package: https://github.com/facebook/prophet

Additional post about Propet and transactios forecasting: https://www.kaggle.com/armamut/predicting-transactions-fb-prophet-tutorial


Dimension of output in Dense layer Keras


MobileNet and MobileNetV2: Bad Inference Results

I found the solution: The source of the error was the wrong preprocessing. I used

from keras.applications.imagenet_utils import preprocess_input

instead of

from keras.applications.mobilenet_v2 import preprocess_input

This wrong function for preprocessing caused the images being misclassified. The latter is the correct preprocessing function.


what is the difference in terms namely Correlation, correlated and collinearity?

Pearson correlation is the usual correlation when nothing further is specified and specifically refers to linear association.
$$\rho_{XY}=\dfrac{cov(X,Y)}{\sigma_X\sigma_Y}$$
In the world, people use “correlation” to mean any kind of association, but this is wrong from the standpoint of statistics. Arrange points symmetrically on a parabola and run them through that equation; you’ll get zero correlation, despite the obvious relationship.
There also is Spearman correlation, which does Pearson correlation on the ranks of the values.
If the points are $(0,1)$, $(2,4)$, $(3,3)$, the Spearman correlation is calculated by converting the $x$-values to their ranks and the $y$-values to their ranks: $(1,1)$, $(2,3)$, $(3,2)$. Then run the transformed points through the usual equation for (Pearson) correction.
To separate “correlation” and “correlated”, the former is a noun while the latter is an adjective. If there is “correlation” between two variables, they are “correlated”.
Collinear seems to come up in the context of regression and refers to predictor variables that are correlated. The related “multicollinear” means multiple regression predictors that have a linear relationship with another predictor, as if you could regress one predictor on some of the others and get decent accuracy. “Multicollinearity” seems to be the more common term to use when we talk about related predictors, as “collinear” variables strikes me as perfectly related with a correlation of $1$ (think of measurements in both meters in kilometers), while multicollinearity, to me, does not mean a perfect predictive ability unless “perfect” multicollinearity is specified.
“Collinear” and “multicollinear” are adjectives; “collinearity” and “multicollinearity” are the nouns.


Does policy optimization learn policies to make better actions with higher probability?

The image in your question looks to me like a loose hierarchy explaining how various Reinforcement Learning methods relate to each other. At the top are broad categories of algorithm based on whether they are value-based or policy-based, towards the bottom are more specific methods.

There is more than one way to categorise and split RL algorithms, and it would be messy to try and include all the ways that they relate to each other. Just bear in mind that it is a very rough guide.

The main difference between value-based and policy-based methods is:

Value-based methods learn a value function (by interacting with the environment or a model of it), and consider the optimal policy to be one that takes actions with maximum value.
Policy-based methods learn a policy function directly, and many can do so without considering the estimated value of a state or action (although they may still need to calculate individual values or returns within episodes).

Note Actor-Critic is linked to both headings, as it learns both a policy function (the "actor") and a value function (the "critic").

  I would like to know is it correct to say: Policy Optimization learns policies to make better actions with higher probability?

Yes that is broadly correct, although you don't define "better". A policy function typically returns some probability distribution over possible actions for any given state. When learning, it will tend to increase probabilities of actions that resulted in better returns (discounted sums of rewards) and reduce probabilities of actions that did not. This may be a very random, high variance process though, depending on the environment.

There are exceptions. Some policy-based methods learn a deterministic policy $a=\pi(s, \theta)$, and adjust the value based on adding some noise to this function to explore and adjusting the function based on the results of the different action. These don't behave like your statement (because there is no probability to make higher).

  Also, what is the location of Proximal Policy Optimization in the picture?

Proximal Policy Optimization is definitely a policy-based method, and it does use an estimate of a value function too for updates (in this case a value called advantage which you also see used in Advantage Actor-Critic).

In the diagram I would probably place it under Actor-Critic Methods in a new row as a specific example of one. However, it does have some significant variations from "vanilla" Actor-Critic, based on how it restricts large changes to the policy function.


How to understand clearly the feature importance computing in random forest model

Let's start with this formula for the Variable Importance Measure (VIM) based in Gini Impurity (GI)*:

$$VIM_{jm}^{(Gini)} = GI_m - GI_l - GL_r$$

Here, $m$ is a given node in a single decision tree which performs a split on feature $j$. For this node you can compute the Gini Impurity $GI_m$ (see * note at the very bottom for the formula). Now this node has two childs: a left node $l$ and a right node $r$. For these the Gini Impurities are $GI_l$ and $GL_r$ respectively. If you substract the Gini impurities of the child nodes from the Gini Impurity of node $m$, then you know by how much the split performed by this node has decreased the overall Gini Impurity. Because before the split it was $GI_m$ and after the split the remaining impurity is the sum of $GL_l$ of $GL_R$. Therefore, the difference ($GI_m - (GI_l + GL_r)
= GI_m - GI_l - GL_r$) is the corresponding impurity reduction.

(as a side node: this is the unweighted gini importance calculation since $GI_l$ and $GI_r$ are considered with equal weights. There are also different weighted versions. For details and a comparison please refer to "The effect of splitting on random forests", Hemant Ishwaran, Mach Learn (2015) 99:75–118)

For feature importance however, you're interested in the overall importance of each feature and not a single node. And a given feature can be present in different branches of a tree.

Here is an example:

(Source: "Artificial intelligence - a modern approach"; Norvig, Russell; 2010; 3rd Ed)

As you can see "Alternate?" is being used at multiple nodes. That is why they use set $M$ in the blog you posted to derive the overall importance of a feature $j$ in a tree $i$:

$$VIM_{ij}^{(Gini)} = \sum_{m \in M}VIM_{jm}^{(Gini)}$$

As a side node: in some papers and books the authors do not calculate the impurity indexed by features $j$ but indexed by a split value of a given node, e.g. $s$, (which of course operates on features). Don't get confused by that.

*
The Gini Impurity of a given node $m$ with $k \in K$ being the classes is defined as follows:
$$GI_m = \sum_{k \in K} \hat{p}_k(m)(1- \hat{p}_k(m)).$$

Where $\hat{p}_k(m)$ is the class frequency of class $k$ in node $m$.