Extracting sub features from inside a df cell?

You were not specific where you wanted to end up with the data in this frame, so I will simply show how to break out the features and sub features into a format that can be pivoted into a form as needed:

Code:

Most important element is taking a feature column and breaking out the sub features.  That can be done like:

def get_sub_features(feature_col):
    # split on commas and then colons
    feature_df = feature_col.str.split(',').apply(
        lambda feature: pd.Series(
            dict([sub_feature.split(':') for sub_feature in feature]),
            name=feature_col.name), 1)

    # add a feature name column to use as an index
    feature_df['feature'] = feature_col.name

    # name the columns as sub-feature for later stacking
    feature_df.columns.names = ['sub-feature']

    # return dataframe with id/feature_name index
    new_index = [feature_df.index.name, 'feature']
    return feature_df.reset_index().set_index(new_index)

Test Code:

df = pd.read_fwf(StringIO(u"""
    Id  Acol                    Bcol    Ccol            Dcol
    1   X:0232,Y:10332,Z:23891  E:1222  F:12912,G:1292  V:1281
    2   X:432,W:2932            R:2392  T:292,U:29203   Q:2392
    3   Y:29320,W:2392          R:2932  G:239,T:2392    Q:2391"""
                          ), header=1).set_index(['Id'])
print(df)

feature_cols = ['Acol', 'Bcol', 'Ccol', 'Dcol']
stacked = pd.concat(get_sub_features(df[f]).stack() for f in feature_cols)
print(stacked)

Results:

                      Acol    Bcol            Ccol    Dcol
Id
1   X:0232,Y:10332,Z:23891  E:1222  F:12912,G:1292  V:1281
2             X:432,W:2932  R:2392   T:292,U:29203  Q:2392
3           Y:29320,W:2392  R:2932    G:239,T:2392  Q:2391

Id  feature  sub-feature
1   Acol     X               0232
             Y              10332
             Z              23891
2   Acol     W               2932
             X                432
3   Acol     W               2392
             Y              29320
1   Bcol     E               1222
2   Bcol     R               2392
3   Bcol     R               2932
1   Ccol     F              12912
             G               1292
2   Ccol     T                292
             U              29203
3   Ccol     G                239
             T               2392
1   Dcol     V               1281
2   Dcol     Q               2392
3   Dcol     Q               2391
dtype: object

Accessing Data:

Some examples:

print(stacked.xs('T', level=2))
print(stacked.iloc[stacked.index.get_level_values('sub-feature') == 'T'])

Results:

               0
Id feature
2  Ccol      292
3  Ccol     2392

                           0
Id feature sub-feature
2  Ccol    T             292
3  Ccol    T            2392


You could use the ratios of bugs (or any other variable) divided by lines of code (LOC). Since ratios may vary a lot for instance bugs vs. vulnerabilities the resulting plot oftentimes doesn't look as good so you could use a normalization procedure, in fact, is the recommended procedure.

In your case Bugs and a few vulnerabilities divided by LOC doesn't look too bad so I included the plot of raw ratios too in this example. I'm using a count overlapping points plot which is a variant of a barplot that doesn't fill bars and only cares about the value as a point. You could use any normalization that you feel gives you the best result. Here I'm only centering the data. In both plots the highest value means higher complexity according to the ratio metric.

Raw Ratios of variables divided by LOC

Normalized Ratios of variables divided by LOC

Code in R needed to replicate these plots

require("reshape2")
require("ggplot2")

df1 <- data.frame(App = c("SweetApp", "SourApp", "SaltyApp"),
                  LOC = c(10000, 5660000, 1500),
                  Bugs = c(10, 120, 55),
                  CodeSmells = c(50, 30, 20),
                  Vulnerabilities = c(2, 3, 10))

#Define ratios
df1$RatiosBugs <- df1$Bugs / df1$LOC
df1$RatiosCodeSmells <- df1$CodeSmells / df1$LOC
df1$RatiosVulnerabilities <- df1$Vulnerabilities / df1$LOC

#Normalize Ratios
df1$NormRatiosBugs <- scale(df1$RatiosBugs, scale = FALSE)
df1$NormRatiosCodeSmells <- scale(df1$RatiosCodeSmells, scale = FALSE)
df1$NormRatiosVulnerabilities <- scale(df1$RatiosVulnerabilities, scale = FALSE)

dfRaw <- df1[, c("App", "RatiosBugs", "RatiosCodeSmells", "RatiosVulnerabilities")]
dfNorm <- df1[, c("App", "NormRatiosBugs",
              "RatiosCodeSmells", "RatiosVulnerabilities")]
dfRaw.m <- melt(dfRaw, id.vars = c("App"))
dfNorm.m <- melt(dfNorm, id.vars = c("App"))

#Plot Raw Ratios
ggplot(dfRaw.m, aes(App, value)) +
  geom_count(aes(color = variable), position = position_dodge(width =  0.4), stat = "identity")

#Plot Normalized Ratios
ggplot(dfNorm.m, aes(App, value)) +
  geom_count(aes(color = variable), position = position_dodge(width = 0.4), stat = "identity")


How to find a confidence level given the z value

OK, for a 95% confidence interval, you want to know how many standard deviations away from the mean your point estimate is (the "z-score").  To get that, you take off the 5% "tails".  Working in percentile form you have 100-95 which yields a value of 5, or 0.05 in decimal form.

Divide that in half to get 0.025 and then, in R, use the qnorm function to get the z-star ("critical value").  Since you only care about one "side" of the curve (the values on either side are mirror images of each other) and you want a positive number, pass the argument lower.tail=FALSE.

So, in the end, it would look like this:

qnorm(.025,lower.tail=FALSE)

yielding a value of 1.959964

You then plug that value into the equation for the margin of error to finish things up.

If you want to go the other direction, from a "critical value" to a probability, use the pnorm function.  Something like:

pnorm(1.959964,lower.tail=FALSE)

which will give you back 0.025


Detecting original vs. edited (reposted / recompressed) image

My question is if there is a better way to represent ordering in this case other than labeling pairs 0 and 1.

I solved this by creating an artificial comparator layer, which takes two inputs, and outputs 0 if the first is smaller or 1 if the first is greater. This allows backpropagation to correctly push the two away from each other within the same batch.

Also, if there are other obvious improvements in the whole process.

See my other question: Producing a confidence output to use in a weighted average layer


How we can have RF-QLearning or SVR-QLearning (Combine these algorithm with a Q-Learning )

You would need to train the RF-QN or SVR-QN on a very large batch/sample generated in the same way as a mini-batch in the DQN version. The input data would be the states and actions visited whilst simulating or running the environment for the batch, and the label would be the TD Target $r + \gamma \text{max}_{a'}\hat{q}(s', a')$ using an old copy of the RF or SVR model to calculate $\hat{q}$ with a loss function of MSE.

The reason you would need a very large batch (and why you generally don't see this done, because it will be far slower than NN) is because neither RF nor SVR in their basic form can be made to run online. However, if you manage to find online versions of the algorithm (I know there are some for RF), then you can use them almost identically to a DNN.

Wherever you see $\hat{q}$ in the pseudo-code, you know you need to run the model forward to estimate Q values. Typically you need to do this to make action selections (by finding best Q value for possible actions) and to generate TD Targets for further training.

Here's how you would need to change a DQN implementation in general terms:

You need two copies of your SVR or RF model: a "target model" and a "learning model". You would start with a basic target model that can predict either random values or fixed values for $Q(s,a)$ - those will clearly be wrong, but it should not matter. Each model will train using output from the previous one, plus the real data about transitions and reward. On each training session, the model becomes more accurate, and allows for better selection of actions.
Wherever you see a call to predict or train the DNN, replace with same call to predict or train the SVR or RF. When you see the current network being cloned to the "target network" do the same with the SVR or RF model. Pay attention to when the code uses the learning network or the target network, and use the appropriate SVR or RF model.
If you are using the default full batch training of RF or SVR, then you will need to generate a large dataset to train at once, and you should clone the resulting newly trained SVR or RF model direct to the target model immediately. Also, as the learning model in SVR or RF does not work online, you will need to use the target network to select actions in your case - that should be fine, although it may slow down learning further. Depending on your RL problem, without online versions of the algorithms, your dataset could easily need to be 100,000 or more records per training and cloning step.


How to Ascertain Sine Wave and Harmonics

You want to look at the spectral density to get an idea of the distribution of the frequency components. Check out the 'psd' package in r or the signal processing toolbox if you have access to Matlab.


Your data looks logarithmic. Try using the scipy.optimize curve_fit() function to find the approximate log coefficients. I tried several built-in python functions, but couldn't get a good fit on any of them, but they can be used as a starting point.
Finally, I ran the curve_fit() function on the data you posted, and after making a few adjustments, I was able to produce the green line on this chart. The blue line is your original data. They are so close together that it is difficult to see the difference. The yellow line is the one that the curve_fit gave me for a best fit.
I also included the extrapolated points (green line again) and the values for the next 15 points below. As you can see, the line is asymptotic to the x axis.

FYI, My adjustments included adding a np.log() to the denominator to convert the base of the log. Also, I changed the intercept to 1 as that is where your data starts.  If you want to automate this, you could write some functions to iterate over different values of the log base and the intercept. I included comments in the code for more details.
import matplotlib.pyplot as plt  # Matlab-style plotting
import numpy as np
import scipy
from scipy.optimize import curve_fit
x = [1.0,2.0,3.0,4.0,5.0,6.0]
x2 = np.arange(0, 22.0, 1.0)
y = [1.0, 0.488, 0.300, 0.213, 0.163, 0.127]
x = np.array(x)
y = np.array(y)

plt.plot(x,y)
# get the approximate values for the log function
data = scipy.optimize.curve_fit(lambda t,a,b: a+b*np.log(t),  x,  y, p0=(1, 1))
# plot those values, the fit is not very good
y1 = data[0][0] + data[0][1]*np.log(x) # yellow
plt.plot(x,y1)
# original coefficients were .91 and -.48
# adjust the 2 coefficents until you get a better fit
# first coefficient is like the intercept but for the line of x of 1, not the y axis
# I also changed the log base value by including the np.log() on the denominator,
# this gives the line a larger curve than the original
y2 = 1.0 - np.log(x2)/np.log(x2+1.9) # green
plt.plot(x2,y2)
plt.show

data

(array([ 0.91083753, -0.48242954]),
 array([[ 0.00543936, -0.00380311],
        [-0.00380311,  0.00346828]]))

y2

array([       inf, 1.        , 0.49069866, 0.30871637, 0.21896813,
       0.16675119, 0.13310187, 0.10985155, 0.09295092, 0.08018302,
       0.0702405 , 0.06230656, 0.05584666, 0.05049745, 0.04600401,
       0.04218259, 0.03889766, 0.03604721, 0.03355312, 0.03135462,
       0.02940378, 0.02766231])


Use of decision trees for classifying images

Yes, you can use decision trees or random forests for this task and this was even the state-of-the-art approach until the early/mid 2010's. For example, here is a paper from 2010 in which exactly this was done. Nowadays, we use convolutional neural networks (CNN's) as they usually yield a higher accuracy and are more convenient to train, as long as you have a lot of images and decent computational ressources.

The most convenient thing about CNN's is that they implicitly create the features by themselves during the training process, while other machine learning techniques, e.g. decision trees, require you to create a set of features beforehand.

For example, in the referenced paper, the authors had to generate the local binary patterns of the images and then used them to train their decision tree. With a CNN they would have simply used the original images and would most probably got a higher accuracy. However, note that they would have needed more images to train a CNN than to train a decision tree.


Yes it is possible. Once you actually return the results from the Tensorflow model, they will (by default) be returned as NumPy arrays. You can then use them as input e.g. to a SciKit Learn model.

Have a look at this thread, which shows some nice examples the types returned by TF models.

One last consideration: The model must return results in such a way that they are in the normal memory, reachable for the CPU - not somehow left in GPU memory.


NLP: Mapping Penn treebank and Brown corpus, to Universal PoS Tags

NLTK contains options for retrieving brown, treebank corpora with universal tags, instead of their own tagging schemes.

nltk.corpus.treebank.tagged_words(tagset='universal')

instead of:

nltk.corpus.treebank.tagged_words()

Similarly:

nltk.corpus.brown.tagged_words(tagset='universal')
nltk.corpus.nps_chat.tagged_words(tagset='universal')
nltk.corpus.conll2000.tagged_words(tagset='universal')

As far as I am aware, none of the other tagged corpora support the universal tagset option.


correct value of output_dim or units parameter of dense

There is no known way to determine a good network structure evaluating the number of inputs or outputs. It relies on the number of training examples, batch size, number of epochs, basically, in every significant parameter of the network.

Moreover, a high number of units can introduce problems like overfitting and exploding gradient problems. On the other side, a lower number of units can cause a model to have high bias and low accuracy values. Once again, it depends on the size of data used for training.

Sadly it is trying some different values that give you the best adjustments. You may choose the combination that gives you the lowest loss and validation loss values, as well as the best accuracy for your dataset.

You could do some proportion on your number of units values, something like:

# Build the model
model = Sequential()
model.add(Dense(num_classes * 8, input_shape=(shape_value,),  activation = 'relu' ))
model.add(Dropout(0.5))

model.add(Dense(num_classes * 4, activation = 'relu'))
model.add(Dropout(0.2))

model.add(Dense(num_classes * 2, activation = 'relu'))
model.add(Dropout(0.2))

#Output layer
model.add(Dense(num_classes, activation = 'softmax'))

The model above shows an example of a categorisation AI system. The num_classes are the number of different categories the system has to choose. For instance, in the iris dataset from Keras, we have:

Iris Setosa
Iris Versicolour
Iris Virginica

num_classes = 3

However, this could lead to worse results than with other random values. We need to tune the parameters to the training dataset by trial and error.


How to normalize data of a different nature?

You would usually just scale all of them to be within the same range.

You can do this by using something like the Scikit-Learn MinMaxScaler, or just a simple function like this:

def scale(data, new_min=-1, new_max=1)
    """Scale values of data to be within the range [new_min, new_max]
    data must be a numpy array or a Pandas Dataframe/Series"""

    return (data - data.min()) / (data.max() - data.min()) * (new_max - new_min) + new_min

Between -1 and +1 is just nice, as the data is centered around zero. You could play with those values.

You can think about and perhaps experiment to see if you should scale all variables together (meaning one global min and max in the dataset), or whether to scale the individual columns/features of your dataset, so each one lies in the given range.

A tip for financial data is to use the log returns - that means to take your raw prices, compute the logarithm of those values, then take the difference between the closing prices of each day.

The reason for this is to because the resulting values are normally distributed, which is an underlying assumption of many models you will subsequently use (Boosting, ARIMA, GARCH for volatilities etc.). There are also other reasons of convenience - check out this article


Spark Mllib - FPG-Growth - Machine Learning

FP-growth is a frequent pattern association rule learning algorithm. Thus it's a rule based machine learning algorithm.

When you call the following :

val model = fpg.run(transactions)

You are actually creating and Frequent Pattern model without generating candidates. So actually you'll need to generate afterwards if you need to use them with :

model.generateAssociationRules(minConfidence)

Now, concerning the usual flow of building and validating such models, your code doesn't deal with that. It's usually done through quality measures variations.

It can also be done through Feature extraction techniques.

With such techniques, you should consider the following. For each association rule, you'll need to measure the improvements in accuracy that a commonly used predictor can obtain from an additional feature, constructed according to the exceptions to the rule. In other terms, you'll have to select a reference set of rules that should help your model perform better. I strongly advice you to read this paper about the topic.

So now what does that mean ? This means that you'll need to implement that pipeline yourself because it's not implemented in Spark yet.

I hope that this answers your question.


unsupported operand type(s) for -: 'list' and 'list' using python

You are facing the error on the line x.append((a[i] - b[i] + c[i])) due to expression a[i] - b[i] + c[i]. Let us see the how it is being calculated. Suppose I have the following data:

array([[1, 2, 3],
       [4, 5, 6],
       [7, 8, 9]])

Then the values of a, b and c are:

a
Out[20]: [[1], [4], [7]]

b
Out[21]: [[2], [5], [8]]

c
Out[22]: [[3], [6], [9]]

Let us take i=0 and print a[0]:

Out[23]: [1]

Hence what you are calculating is [1] - [4] + [7] instead you should be calculating 1 - 4 + 7. To do this change the line with error to:

x.append((a[i][0] - b[i][0] + c[i][0]))


Cannot get the prediction right using Stochastic Gradient Descent: Always predicts 1

My network does always predict the same class. What is the problem?

I had this a couple of times. Although I'm currently too lazy to go through your code, I think I can give some general hints which might also help others who have the same symptom but probably different underlying problems.

Debugging Neural Networks

Fitting one item datasets

For every class i the network should be able to predict, try the following:

Create a dataset of only one data point of class i.
Fit the network to this dataset.
Does the network learn to predict "class i"?

If this doesn't work, there are four possible error sources:

Buggy training algorithm: Try a smaller model, print a lot of values which are calculated in between and see if those match your expectation.

Dividing by 0: Add a small number to the denominator
Logarithm of 0 / negativ number: Like dividing by 0

Data: It is possible that your data has the wrong type. For example, it might be necessary that your data is of type float32 but actually is an integer.
Model: It is also possible that you just created a model which cannot possibly predict what you want. This should be revealed when you try simpler models.
Initialization / Optimization: Depending on the model, your initialization and your optimization algorithm might play a crucial role. For beginners who use standard stochastic gradient descent, I would say it is mainly important to initialize the weights randomly (each weight a different value). - see also: this question / answer

Learning Curve

See sklearn for details.

The idea is to start with a tiny training dataset (probably only one item). Then the model should be able to fit the data perfectly. If this works, you make a slightly larger dataset. Your training error should slightly go up at some point. This reveals your models capacity to model the data.

Data analysis

Check how often the other class(es) appear. If one class dominates the others (e.g. one class is 99.9% of the data), this is a problem. Look for "outlier detection" techniques.

More

Learning rate: If your network doesn't improve and get only slightly better than random chance, try reducing the learning rate. For computer vision, a learning rate of 0.001 is often used / working. This is also relevant if you use Adam as an optimizer.
Preprocessing: Make sure you use the same preprocessing for training and testing. You might see differences in the confusion matrix (see this question)

Common Mistakes

This is inspired by reddit:

You forgot to apply preprocessing
Dying ReLU
To small / to big learning rate
Wrong activation function in final layer:

Your targets are not in sum one? -> Don't use softmax
Single elements of your targets are negative -> Don't use Softmax, ReLU, Sigmoid. tanh might be an option

Too deep network: You fail to train. Try a simpler neural network first.


Tensor Mean of values greater than a threshold


What does the descision boundary of a relu look like?

What would be the shape of this separator (decision boundary) in case we also take a relu on the output and only then threshold?

For just a single neuron, indeed the decision boundary will just be a plane.

I keep thinking it has to be non linear, otherwise NNs wouldn't work...

Single-layer neural networks don't demonstrate the same nonlinearity as multi-layer networks, only as much as an activation function.  (With sigmoid activation, you're just doing logistic regression.)

As a follow up, I would like to understand multi level descision boundaries...

With multiple layers, you gain the ability to combine boundary planes into polytopal regions.  A ReLU-only neural network represents a piecewise-linear function, with finitely many polytopal (i.e., flat-edged) pieces.  That's clear for every neuron in the first layer. In each subsequent layer, you take a linear combination of such functions, which is again such a function; applying a ReLU to that then can cut each region in two, but the result is still such a function.  (You may also be interested in this paper, and this expository blog post.)
You can see some of this in the tensorflow playground.  In a version of that link, I've grabbed the weights and put the resulting function into Wolfram|Alpha.  You can see how the boundary lines of each first-layer neuron shows up as a kink, and each of the resulting regions supports a linear function.  Then slicing at a fixed height gives you the hexagonal region that works well enough to classify the circle dataset.  Increasing the number of layers or neurons per layer allows you to cut the space into more regions.
Then, even if the output neuron applies a sigmoid as in your tensorflow playground, the decision boundary is just a level set of a piecewise-linear function with all linear boundaries, which is itself linear.


Set seed for a Class that calls Keras Models

No you should declare it ahead of the class, just after importing the packages. You have declare the seed for numpy and tensorflow separately. For further details read this blog -
https://machinelearningmastery.com/reproducible-results-neural-networks-keras/


Predict Customer Next Purchase with Sequence

For sequence problems, I generally used Recurrent Neural Network. It has a property to learn its values, which is based upon the previous state and current input.
Since in your case sequence is important, you can use RNN. You can also try LSTM(type of RNN) cell.

  And Additional Note: For product suggestion(where sequence is not
  important) you can also use Apriori algorithm. This algorithm try to
  build association between the products of a single order.


In multiclass classification problems, the number of output units (i.e. neurons) is equal to the number of classes. Only in binary classification, you have 2 classes but just 1 output unit.

The mapping between output units and classes is decided by you. You assign each class an index from $0$ to $N-1$, where $N$ is the number of classes. This values (or one-hot versions of them) are what you supply as expected output to the loss function. Output units form a vector of size $N$, with each unit being at a specific position in that vector. The unit at position $i$ represents the class with index $i$.

The output of the softmax is also a vector of $N$ probability values. The probability at position $i$ represents the probability of the input data belonging to the class with index $i$.


Rule of thumb for good number of features when dealing with grouped data

This a really hard question to answer. I recommend you do some reading to get a feeling on what can be done and particularly, what can be done for your particular task.

This paper is a must. But if you prefer a more practical approach have a look at these two interesting sources:

a) ML Mastery which also provides additional further readings

b) Kaggle

Good luck!


Maximum Likelihood with Gradient Descent or Coordinate Descent blows up

You correctly computed the gradients, but the optimization step is incorrect. Your gradient is computed at point $(\mu,\sigma)$, and, therefore, it should be used to update $\mu$ and $\sigma$ simultaneously. Correct update rule is
$$
\begin{align}
\mu_{t+1} &\longleftarrow \mu_t + \gamma \left(\frac{\overline{x}}{\sigma^2_t} - \frac{\mu_t}{\sigma^2_t}\right) \\
\sigma_{t+1} &\longleftarrow \sigma_t + \gamma\left(\frac{1}{n\sigma^3_t}\sum^n_{i=1}(x^{(i)} - \mu_{t})^2 - \frac{1}{\sigma_t}\right)
\end{align}
$$
The formula that you have in the question implements the alternating optimization which may or may not converge (it is guaranteed to converge if the gradient norm along the optimization path is at most one).
Therefore, I corrected your code as follows:
# Loop through and update mu and sigma
mus = [mu]
sigmas = [sigma]
for i in range(n_iter):
    # Compute gradients
    mu_grad = (np.mean(x) - mu) / (sigma**2)
    sigma_grad = (np.mean((x - mu)**2)/sigma**3 - 1 / sigma)
    # Update mu and sigma
    mu, sigma = mu + gamma_mu*mu_grad, sigma + gamma_sigma*sigma_grad
    # Store mu and sigma
    mus.append(mu)
    sigmas.append(sigma)

The result converges quickly.


Overfitting in an unsupervised technique

I'm not sure if this is valid but how about two trivial clustering examples:

Every object belongs to cluster which contain only this object. So for example if you would like to cluster N cars, there will be N clusters - one for each car.
On the other hand there could be case when algorithm will pick one cluster which will contain all elements inside it - one cluster with all N cars.

Those will be valid clusters but obviously they will not give you any useful information.


Newton-Raphson or EM Algorithm in Python

scikit learn has the EM algorithm here.

Source code is available.

And if you are an R fan the mclust package is available here.


So I think you understood that a MS-TDNN has two parts: a conventional TDNN that calculates state probabilities for each frame and kind of a perceptron on top of it linking the states of several frames to a word. The later part is doing the segmentation. Its connections aren't trained, but just used for running BP and train the TDNN.

The segmentation is obtained by some other algorithm (I don't remember exactly, but something like dynamic time warping). In normal DNN-HMM hybrid systems the DNNs are trained separately to predict the correct state for each frame. The error function is calculated on frame level!

This is not optimal as we do not care so much about each state being correct, but the final text. WER is calculated on world level. MS-TDNNs try to solve this by having this extra layer that allows passing a word level based error to BP. This is very similar to todays methods called sequence training.

Neither TDNN nor MS-TDNN are recurrent networks.

If you are interested in RNNs and ASR systems that do not require HMMs I would recommend you to look at CTC objective that does indeed learn the alignment automatically.


How to decide the shape of input features, when each data file is of different length?

Data exploration

I would suggest exploring the data a little further, which might help decide what would be the best approach for this bird-song dataset.

For example, have a look at the spectrogram of each bird (there are only 66 different genus types), to see how you might extract more data from the samples. Here is the spectrogram of a sample taken from here:

We can see that there is clearly a repeating pattern! We can see those tall light green blocks intermittently appearing throughout. So although the sample is indeed just over 70 seconds of sound, the call of the bird seems to really only last about 2 seconds!

Either with a simple filtering algorithm or even building a model to find those chunks, you could extract those chunks and only work on those, perhaps along with the data regarding the gaps between those chunks.

That is just one example of data-specific pre-processing; I am sure there are many other ways to improve the information density.

Samplerate

This is another degree of freedom you could look at. One idea would be to accept different samplerates within your input to a model. One could adjust the sample rate to ensure the final samples all are all of the same length.

My idea would be to use the length of the shortest sample and then perform regular sampling of all longer sound snippets, such that they the resulting snippets are all of the same length as your shortest sample.

This method will obviously be compromising the quality of the data (irregularly across samples), but if the starting samplerate is sifficiently high, you might be able to get away with it!

Have a look at this useful article that describes many methods in (pre-processing) sound waves.

Two models

In your particular case, if you really only have two possible lengths: 8637686 and 3227894... it might be feasible to simple create two models, one for each sample length. It definitely isn't an optimal solution; however, it would allow for very quick development and model iterations, as you could use the same model and would only need to change on parameter to use both parts of the data.

Basics

As well as truncating your longer samples (cutting them to match the length of the shorter/shortest samples - you could using padding to simple make the shorted samples match the length of the longest sample.

Typically, this is done by simply adding zeros to the end of the vectors. You could also try adding zeros at the beginning and end to keep the information centered within each sample.

If you create a neural network using Keras, you would probably be best by first looking at the ZeroPadding1d layer.


What's the best way to select features independent of the model being used?

There is no best way, but if there was, Tensorflow would definitely not be it. There are three ways I can think of, and each of them is worse than the others in some cases (no free lunch):

Measure correlation of variables with the output variable, and take the variables with the highest correlation. This is a rather poor method as it considers linear correlation, and this is not a very good measure of dependence. Instead of this, although a bit more computationally intensive, is to measure the distance correlation between the output variable and all the variables, and select the features with highest distance correlation.
Fit a linear model with L1 penalization (Lasso). Lasso will automatically select variables for you setting the weights of the non important variables to 0. The variables with the nonzero weights are the ones you can select.
Fit a random forest or a gradient boosting, and take the variables with the highest feature importances. This has worked very well for me in practice.

I hope this helps.


MNIST - Vanilla Neural Network - Why Cost Function is Increasing?

Tensorflow's softmax function only works if the number of batches are in the rows and the output in the columns. If these are reversed, then you need to transpose the tensors in the cost function.


How to get probabilities values with keras?

I faced such a problem using CNN in Keras. Even, I thought that the output is being processed in the such a way like the argmax().

But, after investigating the model, I found that the model are learned and generalised so well that the labels got binarized ( 0s and 1s ). Remember, that we give the label of an image as a one hot vector ( like [ 1 0 0 0 ] ). The NN learned it so well that the output, usually being a probability vector , was now a one hot vector on which it was trained.
Another way to prove this is that you are using a softmax activation function at the output layer. A softmax function produces probabilities which sum up to 1. The sum of all class probabilities is 1. In the vector [ 1 0 0 0 0 ]. The sum of all numbers is 1. Hence the output is valid.

  This is just a different phase in training your model. You can try to increase the size of your dataset or lower the learning rate to 0.0001 or even smaller.


Do I have to scale/normalize my training data for LSTM Classification, even if I only have one feature?

Normalizing the features ensures that they take on reasonable values say between -3 and +3. This ensures that you don't run into a numeric overflow or under flow issues in your network. For e.g. just see what value np.exp(312) or np.exp(-312) takes on.
where, 312 is a value of the bitrate in your observation.
Certain activation functions such as the sigmoid might run into numerical precision issues if your data is not normalized. So, in this case it doesn't matter if your data contains only 1 feature.


Why is my Count (distinct) not working the way I would expect it to be? PowerBI

Count distinct is a measure (aggregate function). When the measure is used in the total row, it is not summing the distinct count values about but is calculating the function in the context of all the table data. Without seeing the data I would make the assumption that there are 85 distinct values between those 4 months. March, for example, happens to contain 83 of 85 of those values.

If you want a sum of the values above, you can use SUMX.

Example:

Count of ProjectName =
SUMX (
    VALUES ( 'put table name here'[MonthNameShort] ),
    CALCULATE ( DISTINCTCOUNT ( 'put table name here'[ProjectName] ) )
)

Comparison in Power BI:


Are there any methods for finding the value of variable which has significant influence on response?

It is therefore a problem of detecting the importance of the variables.
You have many solutions based on the analysis of a regression model, Stepwise regression algorithms (backwards, forwards), random forest...

However, the simplest solution is probably to use a decision tree: the results obtained are presented under a graphical form that is simple to interpret, and is an effective aid to decision support.

Once the graph is obtained, we can already have an idea of the importance of the variables by looking at the first nodes (it often happens that we use the conditions of these first nodes to create synthetic features that we reinject in a logistic regression).
Then to obtain more precise results, we can compute the total reduction of the criterion brought by that feature in the splits.

In python, with scitkit-learn, you could use the feature_importances_ attribute of a  DecisionTreeClassifier to get the details of the importance of the variables:

import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier, plot_tree

# init the classifier
clf = DecisionTreeClassifier()

# don't forget that you need to one-hot encode categorical
# variables before you fit a tree with sklearn!
clf.fit(X, Y)

# plot the graph
plot_tree(clf, filled=True)
plt.show()

print(clf.feature_importances_)

Explanation : The objective of each split is to find the division, or more precisely the variable and the division rule, which will contribute to the strongest decrease of the heterogeneity of the son nodes on the left $\kappa_l$ and on the right $\kappa_r$.
In the case where Y is a qualitative variable, several heterogeneity functions can be defined for a node: a criterion defined from the notion of Entropy or from the Gini concentration (actually there is also the CHAID criterion which is based on the statistical test of $\chi^2$).
In practice, it turns out that the choice of the criterion has little influence, and it is often Gini that is chosen by default.

In the node $\kappa$, where $p_\kappa^l$ is the number of element of the class $l$ in the node $\kappa$, and $m$ the number of classes.

\begin{align*}
\textit{Entropy} :
     &S_\kappa = - \sum_{l=1}^{m} p_\kappa^l log(p_\kappa^l) \\
\textit{Gini} :
    &G_\kappa = \sum_{l=1}^{m}p_\kappa^l(1 - p_\kappa^l) = 1 -
                \sum_{l=1}^{m}(p_{\kappa}^{l})^2
\end{align*}


I wouldn't interpret the first graph the same way. As far as know (and bear in mind that nobody knows everything), there's no clear way to distinguish between noisy features and bias (underfitting). But in either case there's no reason why the validation/CV loss would be much higher than the training loss, this is actually a clear sign of overfitting: if the model was not able to generalize, the training loss would be as high as the validation loss. Instead the model succeeds in representing some patterns, but it turns out that the patterns it captures don't apply (at least not as well) on the validation set. In other words, the model captures patterns which happen by chance in the training set: it overfits.
It's possible that adding more instances would help reduce the overfitting, as the CV loss appears to still decrease significantly in the 800-900 zone. But the difference is very large, so it's not sure that it can be solved only with more data.
I agree that the second graph shows some overfitting (as well), but not as much as the first one. In my opinion, the fact that the lowest difference between the two curves is not at the end of the curve points to either a model which is not stable yet (needs more data) or maybe there are simply not enough points on the curve to observe the real trend (i.e. the low difference at 240 and/or the higher one at 300 might happen by chance). This is indeed variance, but it's not very high
in my opinion.


Text classification with Word2Vec on a larger corpus

Use large amount of un-label data to finetune the BERT based model. You can train BERT in unsupervised manner. Then, use that bert to get embeddings of your input text of label data and train a classifier.


KNN:

The prediction of Lat/Lng from a given timestep could be performed by analysing the previous data of the user. This is a totally naive approach but could be tried,

Take an input timestamp from the user. Let's call it $x$.
From the previous data of the user, search for the timestamp which is the closest to $x$.
Once you find the closest timestamp, output the Lat/Lng associated with that timestamp.

  This approach is called K-NN ( K Nearest Neighbours ) classification.
  It will work well if the user is mostly at a common place at a given
  time ( timestamp ).

Clustering method:

This method will work on making clusters of Lat/Lng. For some a given time-interval, suppose, you will get a common timestep ( which will be the centre of the cluster ).

Once you have a cluster of timesteps, find the timestep ( centre of the cluster ) which is the closest to the input $x$.
For this timestep, you will get Lat/Lng which close to one another.
For predicting the Lat/Lng, use the centre of the cluster and output the associated Lat/Lng values.

  Basically, you are clustering similar data points which help in prediction. You can find where a user's location is clustered at given time intervals.


TicTacToe Linear Regression low accuracy and R^2 score

Linear regression will not work for this problem because the relationship between the board features and target variable that you are using is not linear.

Is this how data scientists would go about creating the training set for tic tac toe?

It is not 100% clear what your goal is. For simplicity I will select that your goal as "Predict the probability of X winning eventually given the current board state and completely random play in future by both sides." That appears to be what you are doing.
As an aside, this is not a direct path to training a neural network to predict the best moves to make in a game. For this simple game, it might work acceptably if that is your eventual goal, but if you want machine learning for game playing you should probably look into reinforcement learning, and specifically self-play with reinforcement learning, as a framework to manage the training data.
Back to your question, what you are doing is acceptable for creating a data set, although I would want to check:

For every tie game(full board) I do not count it as anything

If that means you are still storing the states that lead to a tie, but with a different label, then that is ok. If you are discarding data about ties, then that will skew the dataset and might impact your predictions - unless you are also discarding ties when testing.
This is also slightly unusual:

At the end I get my training data set which includes the board state (the feature set) and the outcome which is the percentage (a floating pint value. eg .8 is 80%) of games won for that state.

This is unusual in that you have pre-processed the data into a summary row when features are identical. This skews the dataset when used with an approximation function (linear regression - like most ML statistical learners - is an approximation function), because you lose the number of times those features occurred. Any balancing the prediction function does to make itself more accurate for common states is lost when you do this. It is more normal to keep all records separate and have the ML method resolve the best way to take averages. If you measure the accuracy of your completed model by taking random samples of new played games, it could have lower accuracy than possible due to this.
For data collection of records, it is more usual to keep all observations separate and not to summarise them before training a classifier. The classifier can then fit the data allowing for the frequency of each observation.
Other than the caveats about ties (which you may well have right), and premature taking of averages, plus the limitation that your dataset will only help predict outcomes in fully random games, then the dataset collection looks ok to me. Neither of the above problems are major enough to cause the problem that you noticed. The reason your predictions are not working with linear regression is mainly due to needing non-linearity in the prediction function.
A simple fix for this would be to use a non-linear predictor such as a neural network or maybe a decision-tree algorithm like xgboost.
If you use a neural network, the following may help:

Use sigmoid activation in the output layer and binary cross-entropy loss. This should help when your output is a probability.

Use the value $-1$ instead of $2$ for marking positions in the board played by O. This is not strictly required, but neural networks tend to learn faster and more accurately when the input data in centered around zero with close to 1 standard deviation.

It is worth noting that your averaged win rate table is already quite a reasonable predictive model for game playing. For TicTacToe it should work quite well because there are a limited number of states. After 20k games with random play, you will have a record of nearly every possible state, and some will have reasonably accurate average values (for instance each initial play by X will have ~2000 sampled continuations which should give you the win rate within a few percent). The weakness of this approach is that it cannot generalise to new unseen states, but actually that is quite hard to do in board games where fine detail matters.


How do I build a model to predict which service a customer will use on an app?

Start with the most frequent used ones... then explore the need to build a more complex model. Chances are the distribution of service is skewed and you don’t really need to build a model outside a top n.


Saving and loading keras.callbacks.History object with np.save and np.load

use allow_pickle=True in np.load

>>>np.save("a.npy",his)
>>>item = np.load("a.npy",allow_pickle=True).item()


numpy array quirks: does A.T return a view?

You are right that B points to the original array and A (at the end) is a transposed view of the array.
The key point here is something that trips up a lot of people: whenever you assign a new value to a variable by using = with no indexing (e.g., A = <anything>), it causes that variable to point to a new location, but it doesn't alter the data previously held in that variable.
Here is an annotated version of what is happening:
A = np.array([[1,2,3],[4,5,6]])
# create the array [[1,2,3],[4,5,6]] at some location in memory
# (let's call it loc 1)
# point variable A to loc 1

B = A
# point variable B to the same location as A (loc 1)

A = A.T
# right-hand side:
#    - get the array that A currently points to (from loc 1)
#    - create a transposed view of this array and store the view
#      somewhere in memory (let's call it loc 2)
# left-hand side: set variable A to point to loc 2
# Note: the `=` assignment does not alter the object at loc 1; instead,
# it points A to the new object that was created on the right-hand side

A[0,1]= -1
# Push the value -1 into position (0, 1) of the object that A
# currently points to, i.e., the view at loc 2. This updates
# the A view, but also propagates back to the original,
# un-transposed array at loc 1 (which B still points to).

# Note that the last two lines are equivalent to this:
X = A.T     # doesn't transpose B
X[0,1]= -1  # propagates through view back to B
# i.e., the A.T could have been assigned to anything, e.g. a new
# variable X. When you do an assignment with `=`, it creates a new
# variable every time, even if it happens to have the same name as
# an old one, as in your case.

This is the general issue of changing mutable data and getting unexpected side effects. The easiest way to think about it is that a bare variable assignment (A = <something>) never alters the underlying data that A currently points to. Instead, it just causes A to point somewhere new. On the other hand, assigning A[index] = <something> or A.attribute = <something> or using A.data_altering_method(...) will alter the underlying object that A points to, in place, so anything else that points to the same object will also see the alteration.
In your case, the only thing you did that changed the underlying data was A[0,1]= -1, and that did indeed propagate to B. A = A.T did not alter the underlying data, it just pointed A to something new (which happened to be a new view of the original data). So the change to the A variable didn't propagate to B. However, it is a little surprising that  A = A.T creates a view rather than a copy, so that later changes to A affect B. Your middle two lines are equivalent to B.T[0,1] = -1.
These answers also help:

https://stackoverflow.com/a/7838047/3830997
https://stackoverflow.com/a/38925257/3830997


Association Rules - Data Mining - Train and Test approach?

Finding association rules is an unsupervised learning task (or exploratory task). You don't actually know which rules you want to find before you actually find them, so there's nothing to test against. Validating against a separated instance set is usually done on supervised learning tasks such as classification and regression.


Compare multiple values from a DataFrame against a single row from another

You can join df and df_st on Reference:
df_merged = pd.merge(df, df_st, on="Reference", how="left")

Note: The how="left" would really depend on what you want in the joined table. You can then compare the values in the joined columns:
df_merged['Address_df'] == df_merged['Address_df_st']


How to predict based on multiple samples?

This is a reasonably standard problem for supervised ML:

The class is the variable "dropped_out"
Given the goal to predict a variable which is specific to a particular student, an instance must represent a student, not an exam.

This definition of what an instance should consist of seems to be the part that you didn't reach yet: you correctly saw that you need to join the two datasets but in your example you join them by exam id. As a result you obtain "instances" which each represent a particular exam by a particular student, and of course the same student might appear several times in the data. The solution is to join your datasets by student id in order to make a single instance contain all the information for one student, i.e. something like this:

AGE, RESULT_TEST1, RESULT_TEST2, SCORE_EXAM1, SCORE_EXAM2, SCORE_EXAM3,...., DROPPED_OUT

However it seems that the exams are not normalized, so I see two options:

Simplification: for each student, give only some summary statistics about their performance at exams, for example min, max, avg, std dev for both the score and the complexity. This gives a fixed number of features (8 in my example), each with a specific role so that the ML method can "make sense" of it.
Refactor the data: if possible, rearrange the exam data so that a column corresponds to the same exam for different students. This would mean that the exam complexity is not needed anymore, because the distribution of the grades is the only thing which matters. It's ok to have some missing/undefined values for the students who didn't take a particular exam, most ML methods can deal with that.

The second option is very likely to give better results than the first, but it might be impractical to transform the data this way.


Applying random forest model to a dataframe with multiple types of data

One thing to remember when we use Random Forest is when you use a categorical feature for training it shouldn't have more than 53 categories. Sometimes RandomForest takes numerical data as categorical. To overcome that make sure to convert all categorical as factors using this command.

df['col_name'] = df['col_name'].astype('category')

Lastly, make sure that all the levels in a categorical feature are presnent in both train and test data or else the model cannot hadle such situations.

I think you might get the model running if you can use any server(as the model can use the servers processing capacity to its fullest) and deploy on that and let it run before you leave for the day, before deploying make sure that you have tested the code on sample data. So, that it won't abend with some error.

Do let me know if you need any additional information.


Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model.

Doc2vec solution
You can train your doc2vec model following this link. You may want to perform some pre-processing steps like removing all stop words (words like "the", "an", etc. that don't add much meaning to the sentence). Once you trained your model, you can find the similar sentences using following code.

import gensim

model = gensim.models.Doc2Vec.load('saved_doc2vec_model')

new_sentence = "I opened a new mailbox".split(" ")
model.docvecs.most_similar(positive=[model.infer_vector(new_sentence)],topn=5)

Results:

[('TRAIN_29670', 0.6352514028549194),
 ('TRAIN_678', 0.6344441771507263),
 ('TRAIN_12792', 0.6202734708786011),
 ('TRAIN_12062', 0.6163255572319031),
 ('TRAIN_9710', 0.6056315898895264)]

The above results are list of tuples for (label,cosine_similarity_score). You can map outputs to sentences by doing train[29670].

Please note that the above approach will only give good results if your doc2vec model contains embeddings for words found in the new sentence. If you try to get similarity for some gibberish sentence like sdsf sdf f sdf sdfsdffg, it will give you few results, but those might not be the actual similar sentences as your trained model may haven't seen these gibberish words while training the model. So try to train your model on as many sentences as possible to incorporate as many words for better results.

Word2vec Solution
If you are using word2vec, you need to calculate the average vector for all words in every sentence and use cosine similarity between vectors.

def avg_sentence_vector(words, model, num_features, index2word_set):
    #function to average all words vectors in a given paragraph
    featureVec = np.zeros((num_features,), dtype="float32")
    nwords = 0

    for word in words:
        if word in index2word_set:
            nwords = nwords+1
            featureVec = np.add(featureVec, model[word])

    if nwords>0:
        featureVec = np.divide(featureVec, nwords)
    return featureVec

Calculate Similarity

from sklearn.metrics.pairwise import cosine_similarity

#get average vector for sentence 1
sentence_1 = "this is sentence number one"
sentence_1_avg_vector = avg_sentence_vector(sentence_1.split(), model=word2vec_model, num_features=100)

#get average vector for sentence 2
sentence_2 = "this is sentence number two"
sentence_2_avg_vector = avg_sentence_vector(sentence_2.split(), model=word2vec_model, num_features=100)

sen1_sen2_similarity =  cosine_similarity(sentence_1_avg_vector,sentence_2_avg_vector)


Predicting crowd density using real time streaming data

Are there any incremental ML models which could be suitable?

You have to be able to access the model when it's in production mode and actually be able to update it with a new version. all ML models should be adjusted when new data comes because of the so-called data shifting. you could periodically evaluate how much the model is performing.

I suggest modeling the problem as a sequence model, use multiple output at each day to predict densities for each postal code.


Football match prediction using regression

If your MSE is low, but then your predictions on the test set are way off, i can think only of over-fitting either you're leaking information from the training set to the test set ( or vice-versa ) or you simply have a complex model ( too many features ) which you could solve by increasing your data or going for less features.

That's a first diagnosis, maybe you could plot your learning curves so you can check your variance/bias and tell which one of those problems you have.


Some of the features were categorical, so they had to be converted to integers.

Then, I ran this to get the data in the right format:

import scipy.sparse as sp
X_train = sp.csr_matrix(X_train)
y_train = sp.csr_matrix(y_train)


Do we need to add the sigmoid derivative term in the final layer's error value?

Both courses are correct.

For the output layer only, and depending on which activation function and which loss function are in use, sometimes the effects of $f'(z^{(L)})$ are perfectly cancelled out by the calculation of $\frac{\partial J}{\partial \hat{y}}$

This is often arranged deliberately to simplify calculations. So:

MSE loss $(\hat{y} - y)^2$ is paired to linear activation function $f(x) = x$
Binary cross entropy loss $-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})$ is paired to sigmoid activation function $f(x) = \frac{1}{1+e^{-x}}$
Multi-class cross entropy loss $-\mathbf{y}\text{log}(\hat{\mathbf{y}})$ is paired to softmax activation function $f(x_i) = \frac{e^{x_i}}{\sum_j e^{x_j}}$

In all these cases, the end result is that, for the output layer logits only:

$$\frac{\partial J}{\partial z^{(L)}} = \hat{y} - y$$

Each of these results can be verified with some differentiation. E.g. it is possible to show that $\frac{d}{dx}(-y\text{log}(\hat{y}) - (1-y)\text{log}(1-\hat{y})) = \frac{1}{\hat{y}(1-\hat{y})}$

For all the other layers, you absolutely need to include the factor of $f'(z^{(l)})$ for whatever activation function is in use on the layer you just back propagated through.


Extracting encoded features after CatBoost

I don't believe this is possible, CatBoost does target encoding per split, so you end up with different values of encoding at different trees.

Before each split is selected in the tree (see Choosing the tree
structure), categorical features are transformed to numerical. This is
done using various statistics on combinations of categorical features
and combinations of categorical and numerical features.

https://catboost.ai/docs/concepts/algorithm-main-stages_cat-to-numberic.html
However, if you just want to use the CatBoost encoding algorithm. You can use CatBoostEncoder


Quote from the author of xgboost:

Both xgboost and gbm follows the principle of gradient boosting.  There are however, the difference in modeling details. Specifically,  xgboost used a more regularized model formalization to control over-fitting, which gives it better performance.
We have updated a comprehensive tutorial on introduction to the model, which you might want to take a look at. Introduction to Boosted Trees
The name xgboost, though, actually refers to the engineering goal to push the limit of computations resources for boosted tree algorithms. Which is the reason why many people use xgboost. For model, it might be more suitable to be called as regularized gradient boosting.

Edit: There's a detailed guide of xgboost which shows more differences.
References
https://www.quora.com/What-is-the-difference-between-the-R-gbm-gradient-boosting-machine-and-xgboost-extreme-gradient-boosting
https://xgboost.readthedocs.io/en/latest/tutorials/model.html


I think the best thing for a beginner is some practice to let the theory sink in. (If you need book suggestions, do a search.) If you are weak on computer science, contribute to an open source project like scikit-learn. If you are weak on data analysis, compete in a Kaggle competition. Write a paper and/or a few blog posts to prove your skills. Do this now while you are in school. An undergraduate physics degree should be enough to get your foot in the door if you do the above. A degree will rarely get you a job by itself; you still have to pass the job interview, and that's why I recommend getting practice.


Looking for an advice based on an output of a LSTM model

It looks like your indices for the predicted data are off by one. You could try to fix it with
predicted = predicted[1:]


How shouild I save model weights in Pytorch?

PyTorch has a state_dict which stores the state of the model (in this case, the neural network) at any point in time. Saving it would involve dumping those states into a file which is easily done with:
torch.save(model.state_dict(), PATH)

When reloading the model, remember to first create the model class with its default weights and load the state dict from the file.
Here's a link to saving/loading pytorch modules (https://pytorch.org/tutorials/beginner/saving_loading_models.html).


Here is one way to do it:

from collections import Counter

from numpy.random import binomial

# Generate mock sales data by randomly sampling if a salesperson makes a sale
p = 0.5 # Control "flatness" of distribution
n_records = 14
salesperson_id = binomial(n=100, p=p, size=n_records).tolist() # Binomial is a reasonable approximation for a discrete normal distribution
count_per_salesperson = Counter(salesperson_id)
id, count = zip(*count_per_salesperson.items())


Is there a documentation where it is explained why scikit-learn does not provide p-values?

There is a scoring function in sklearn package which is used in a feature selection procedure. This function first computes F-score and then converts it to p-value.


Which type of model to use and what's my target variable

I would say the variable 'spent' is your target variable. I'd suggest you to analyse the distribution of each feature and target to check if, eventually a linear regression could be appropriate.
You might try different other methods however, like a decision tree regression, which does not require any assumptions as it doesn't estimate parameters.
What I would suggest you to do, howeveer, is to concentrate on the distributions of parameters to check for eventual unbalances and make groups to make estimation robust.


How to do target encoding when data has repeated rows?

Yes, you can do target encoding. watch this video from @2:18 to @3:50. Hope this helps.

Update: Table B is enough to encode, you have mean values of y among each category. If a responce has like this [15-11-106 ,15-11-2], then you can replace those values with them [334.714, 298.994], now you can apply any statistic to get final encoding value, whether it could be mean/mode/median etc... of those values.

The mean target times the count of each code (Table B * Table C) is nothing but the sum of the target variable for each unique category(instead of mean of target variable you are going to choose sum of target variable). I don't think it's a good strategy.


Independent of the number of features, you will obviously need much more than

3 * 100.000 * 100.000 * 8 bytes

of ram (with double precision floats). That is 240.000.000.000 or about   240 GB. Not only is this a lot of RAM to have, but AP will have to do many passes over this RAM, so this will take forever, even after computing the distance matrix.

So clearly PCA won't help. You just cannot afford an algorithm with O(n²) memory and O(n²) runtime. PCA reduces d, but the runtime of AP itself does not depend on d.


How to prepare data for Named Entity Recognition with BIO annotation?

If your data always looks like this, there is little reason to use sequence labeling: every token belongs to an entity, so it's just a matter of correctly separating the entities and classifying them. But since the entities are already separated by line breaks, there's no need to train a model to separate them. So in the end you just have to classify the entities by category, and this doesn't require sequence labeling. But even for that, from your example it looks like the skills vs. education entities are already separated, so in the end I'm not sure what you want the model to learn?


How to Classify an Image in a Class and a Subclass?

The simple solution

Have one classifier which can classify all the combinations:

Fresh banana
Rotten banana
Fresh mango
Rotten mango
...

The good solution

Or have one network with multiple outputs. See the master Thesis of Marvin Teichmann for example.
https://arxiv.org/abs/1612.07695

So one path of the network would be:

Fresh vs Rotten

The other path of the network would be

Banana vs Mango vs ...

The difference to the simple solution is that you can give a probability to the cases, e.g. for the ground truth "fresh banana" you might say 99% banana but only 42% fresh.

The obvious solution

This one should not be done: have multiple classifiers


SVM hyperplane equations for linearly seperable data

Normalized / standardized data are necessary to train an SVM classifier. Before running an SVM you should always scale your variables: it can be done either through min-max scaling or with standardization (in statistics are called Z-scores).

It's a fundamental dataprep operation, without it a classifier such as SVM wouldn't be useful. That is because if variables are not on the same scale the ones with the larger variance will dominate all the others in the training.

The equations that you quoted are the support vectors. In the image next to them: the dashed lines whose distance must be maximixed by the SVM classifier. The values of 1 and -1 are the labels that the model is attributing to the target classes (in a binary classification problem).

Hope this helps, otherwise let me know.


I found both of your options slightly faulty. So, this is generally (very broadly) how a predictive modelling workflow looks like:

Data Cleaning: Takes the most time, but every second spent here is worth it. The cleaner your data gets through this step, the lesser would your total time spent would be.
Splitting the data set: The data set would be splitted into training and testing sets, which would be used for the modelling and prediction purposes respectively. In addition, an additional split as a cross-validation set would also need to be done.
Transformation and Reduction: Involves processes like transformations, mean and median scaling, etc.
Feature Selection: This can be done in a lot of ways like threshold selection, subset selection, etc.
Designing predictive model: Design the predictive model on the training data depending on the features you have at hand.
Cross Validation:
Final Prediction, Validation


Supervised learning for image segmentation

I would suggest you use Python with Ski-Image for Image Related operation. For Machine Learning typically for Deep Learning, you should use Keras with theano/Tensorflow backend with GPU capabilities whichever suits you.

In order to generate masked images, you should use data augmentation.Data Augmentation Documentation for Keras


It is quite true that papers or books use notations that sometimes seem obvious to people who are used to dealing with the mathematical aspects, but are meaningless for the others. Ways of understanding the math include:

Following theoretical courses or trainings
Reading dedicated books
Asking people around you
Asking people on forums such as this one, or Cross Validated for stats formulae
Getting it by yourself upon re-reading parts of the paper/book you didn't get at the first time

There are some notations/conventions that are implicitly accepted in data science / machine learning papers, such as:

Using $X$ as input, $y$ as output, $\theta$ as model parameters
Using $\hat{y}$ for the estimator of the true $y$
Assuming that vectors are column vectors

The list would be too long to include here.

Regarding the example above, what we face is a constrained optimization.

The $max$ statement means that we are looking for a maximum value of the expression that follows. What is below (namely, the $\Delta_{ij}$ values) the $max$ is the list of "free" parameters that change the value of the expression.

The $max$ statement is prefixed by $arg$, which means that we do not have interest in the expression's maximum value, but rather in the $\Delta_{ij}$ set that leads to that value.

Then we face a $s.t.$ statement, because this is no ordinary optimization, we also have to respect the several constraints expressed after $s.t.$. Those can be inequations, equations, belonging constraints, etc., either explicit ($\Delta_{ij} > 0$) or more implicit.


Forecasting Consumption for Multiple Products for Multiple Regions

Models

Xgboost can be used for Time Series. Here is a tutorial on Kaggle.
If Deep Learning is an option (the data set
has at least several thousand samples), you can try a RNN
(Recurrent Neural Network), with LSTM or GRU cells.

Features
You can transform categorical variables into indicator variables with the get_dummies function in pandas.


TensorFlow Time Series Tutorial Enhancement Gone Wrong

It seems I was returning multiple features as labels.  I had to modify the dense_1_step function to return a single feature...

def dense_1_step(batch):
  # Shift features and labels one step relative to each other.
  return batch[:-1], batch[-1:,1][0] # take second feature only

To make it the same as the output from the multivariate_data function.


How to iterate and modify rows in a dataframe( convert numerical to categorical)

As the error suggests, you don't have a column called Y1. Hence the error. Here is my suggestion to fix this. Assuming you input data looks like this -

15.55
15.55
15.55
15.55
20.84

Read it in pandas this way -

import pandas as pd

df = pd.read_csv('path/to/file.csv', header=None)

Provide a column name for this -

df.columns = ['Y1']

If you have more columns, just fill the df.columns list accordingly.

Finally, use the pandas best practices as per their latest documentation to assign a new column -

df = df.assign(Y2= (df['Y1'] > 18.250000).astype(int))

Output

print(df)

 Y1  Y2
0  15.55   0
1  15.55   0
2  15.55   0
3  15.55   0
4  20.84   1

Note: Since I don't have full visibility on what you are working on, I have assumed what might be the problems you are facing. If this doesn't work let me know.


How is "relevance" defined in information retrieval outside the context of systems with user feedback?

There is no formal definition for the concept of relevance, because relevance depends completely on the context and is therefore highly subjective. This is why the best way (some might say the only way) to evaluate relevance is to actually ask users what is relevant for them.

For any ML-based task, one needs to design a proper evaluation framework in order to  control and measure the quality of the results. Naturally the evaluation method should be chosen so that it reflects as much as possible the level of quality with respect to the goal of the task, i.e. what one would intuitively expect from it. Evaluation metrics are almost always simplified indicators of this "level of quality", so what matters is how well they correlate with what a user would expect from the system: sometimes even a perfectly standard evaluation measure might not to be suited to the goal of the task.

My point is that evaluation is a matter of analysis and design. There are infinitely many options, but the point is to select the most appropriate one for the job. Here are some of these options:

The ideal case is to have annotated data (e.g. user feedback) directly suited to the data and the task: then it's just a matter of counting how often the prediction is correct.
It's common to evaluate a system against another annotated dataset X, assuming that the task is similar enough so if the system works well on X then it will work well on the real dataset.
Another less than ideal way is to evaluate against the predictions of another reference system X: in this case X is considered the gold standard, so there is no way for the tested system to perform better than X.
Indirect evaluation: if there is another task being performed at a later stage with the predictions, and this task can be evaluated more easily than the IR task itself.
Heuristics: that would be the least reliable kind of evaluation, but it's better than nothing. It ranges from simply counting the number of words in common between the query and top N results to developing complex methods using third-party resources.


Two things:

Try reinstalling/updating Orange to the latest version.

Try reinstalling/updating the Geo package in Orange, from the Add-ons menu, to the latest version.

I am currently running Orange version 3.27.1 and Geo version 0.2.8 and the Choropleth widget appears.


Keras - no prediction probability for multiple output models?

As you can see here Keras models contain predict method but they do not have the method predict_proba() you have specified and they actually do not need it. The reason is that predict method itself returns the probability of membership of the input to each class. If the last layer is softmax then the probability which is used would be mutually exclusive membership. If all of the neurons in the last layer are sigmoid, it means that the results may have different labels, e.g. existence of dog and cat in an image. For more information refer here.

For more information as stated here in the recent version of keras, predict and predict_proba are the same i.e. both give probabilities. To get the class labels use predict_classes. The documentation is not updated. (adapted from Avijit Dasgupta's comment)


In training set:

You would better use some methods to "presume" certain values...average / pro-rata......which you could find in various learning sources

Leaving them as blank (not sure of Azure but in a low level programming platform) would cause either inaccuracy or problems.

For testing set:

You are not changing the Learned Parameters, but still that would probably causing some inaccurate measurement but Azure ML may have done something internally.

Reference from MSDN:

  How to deal with missing values?

  To deal with missing values, it is best to first identify the reason for the missing values to better handle the problem. Typical missing value handling methods are:

  Deletion: Remove records with missing values
  Dummy substitution: Replace missing values with a dummy value: e.g, unknown for categorical or 0 for numerical values.
  Mean substitution: If the missing data is numerical, replace the missing values with the mean.
  Frequent substitution: If the missing data is categorical, replace the missing values with the most frequent item
  Regression substitution: Use a regression method to replace missing values with regressed values.


You could e.g. use as objective "multi:softprob" instead of "multi:softmax". The model is identical, but you get a probability for each label instead if the most probable label.  Then you can specify a threshold, e.g. 0.5 and return "unknown" if no probability is above the limit.


Which programming languages should I use to integrate different analysis software packages?

If you are mostly stitching up together calls into other software, like unix utilities (awk, grep, sed ...), python, and matlab scripts, bash is just fine or possibly even best for the job to construct simple pipelines and workflows.

It's easy in bash to read user input, store it in variables, then launch other software depending on the set variables. It's perfectly fast enough for that, and nothing else gets any easier.

If you, however, were to use bash for preprocessing itself, like looping through files line by line, packing and unpacking tab-separated values into arrays etc., that would be excruciatingly slow and not recommended.


Use log-probabilities, then use k-shortest paths.

The probability of a path is the product of the probabilities on its edges.  If you log-transform everything (so that each edge is annotated with the log of the probability rather than the probability itself), the log-probability of a path becomes the sum of the log-probabilities on the edges.

Now you want to rank the paths by decreasing log-probability, and find the $k$ paths with the highest log-probability.  Note that the log-probability of a path is just the length of the path (the sum of the lengths of the edges).

Therefore, the problem becomes: given a graph with a length on each edge, find the $k$ shortest paths.

There are efficient algorithms for finding the $k$ shortest paths.  See, e.g., https://cstheory.stackexchange.com/a/32501/5038.


What makes a Tree-Structured Parzen Estimator "tree-structured?"

It means that your hyperparameter space is tree-like: the value chosen for one hyperparameter determines what hyperparameter will be chosen next and what values are available for it.

From a HyperOpt example, in which the model type is chosen first, and depending on that different hyperparameters are available:

space = hp.choice('classifier_type', [
{
    'type': 'naive_bayes',
},
{
    'type': 'svm',
    'C': hp.lognormal('svm_C', 0, 1),
    'kernel': hp.choice('svm_kernel', [
        {'ktype': 'linear'},
        {'ktype': 'RBF', 'width': hp.lognormal('svm_rbf_width', 0, 1)},
        ]),
},
{
    'type': 'dtree',
    'criterion': hp.choice('dtree_criterion', ['gini', 'entropy']),
    'max_depth': hp.choice('dtree_max_depth',
        [None, hp.qlognormal('dtree_max_depth_int', 3, 1, 1)]),
    'min_samples_split': hp.qlognormal('dtree_min_samples_split', 2, 1, 1),
},
])

From the original paper:

  In this work we restrict ourselves to tree-structured configuration spaces. Configuration spaces are tree-structured in the sense that some leaf variables (e.g. the number of hidden
  units in the 2nd layer of a DBN) are only well-defined when node variables (e.g. a discrete choice of how many layers to use) take particular values.


scikit-learn pipeline strategy with both numeric and categorical columns

One option is scikit-learn's ColumnTransformer applied to mixed types. ColumnTransformer is designed for the purpose of applying different preprocessing and feature extraction pipelines to different subsets of the features.


EagerTensor object does not support item assignment

You can't do this in Tensorflow.

For reference see: https://github.com/tensorflow/tensorflow/issues/33131


Because your two questions are phrased a bit differently, I'm going to answer the one in the title.

  Is Neural Network Architecture independent of Data?

Generally speaking, yes.

One simple reason why you might have to change the architecture is if your new task has a different number of classes. If so you must change the last layer of your network.

Another reason is that some deep CNNs perform multiple stages of spatial subsampling (e.g. through pooling). This requires the input to be larger than a minimum shape. E.g. InceptionV3 and InceptionResnetV2 require the input to be at least $75 \times 75$, Xception $71 \times 71$, and so on.

Even if you don't necessarily have to, sometimes you should. An architecture that does well in MNIST, won't necessarily do well on ImageNet, because the latter is much harder. It requires models with a higher capacity.


How are the findings learnt from the data set are generalized compared to Statistics?

Bias-Variance decomposition is one way, and V-C dimension/bound is another...

Both of these are metrics you can use to get a feeling of how confident you should be that your training results will generalize to out-of-sample.

V-C dimension focuses on the results of this learning algorithm outcome.
Bias-Variance focuses on the expected outcome of the algorithm itself.

Pick your poison - I sincerely hope that this was helpful.


Why do recurrent layers work better than simple feed-forward networks?

The first reason is the number of parameters. The former case that you've mentioned, for each neuron there should be corresponding entries that would increase the number of training parameters. The other reason is that by employing simple feed-forward neurons you are somehow discarding the temporal information of your data which means you are discarding the sequence information in your data. This is somehow like the spatial data which is obtained by convolutional layers in CNNs.


How do you get an RNN to learn in real time?

I think you misunderstood the fundamental of RNN.

  fix their weights to all be the same".

Yes that's exact what RNN is suppose to be. In any type of RNN, you always only have one real RNN cell. The unrolled RNN cells are merely representations of that RNN cell overtime. In a single training step, the weights of those unrolled RNN cells are always the same.

  Sp are RNN's any use in learning in real-time for temporal data?

Yes, you can train an RNN in real-time with a catch. Although in theory the sequence is arbitrary long, but in practice we can only backprop a certain step back due to


Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

The gradient of the average error doesn't always equal to the average gradient of errors.
The source for the difference between them lies in the non-linear layers of the model.

Example:

You can easily see it in the following example with the gradient of the sigmoid function:

The sigmoid function is defined as:

It has a very convenient derivative:

We now take 2 inputs  and calculate the mean of the sigmoid's gradient with respect to them:

We now calculate the sigmoid's gradient with respect to their mean:

These 2 results are clearly not the same. If you want further proof, just calculate the numerical results for:

You will get that the mean gradient is ~0.2233, and the gradient of means is ~0.235.


How to use the fillna method in a for loop

what you are looking for is replace().

And you don't need to write all the columns you can iterate over the columns name simply.

for col in train:
    train[col].replace("NA","XX",inplace=True)

You can do it on all the dataset in one line:

train.replace("NA","XX", inplace=True)

Or on specific columns:

for cols in na_data:
    train[col].replace("NA","XX",inplace=True)


Strategies to encode categorical variables with many categories

Generally, the logic of the categorical count transformation lies in the fact that features with similar frequencies tend to behave similarly. Have words in a corpus as an example, common words share little or no real information whereas uncommon words share more information with an algorithm.

Specifically, certain algorithms (tree-based methods) could even yield rules given an unspecified category from an event count. Say, for example, we have an unknown category whose count is 4. The algorithm may give a rule:

  If Column Count is < 5 and N is > 3 = X

That will be exactly the same as if an algorithm took a One-Hot encoded column and gave a rule:

  If One-Hot-Encoded-Column is > 0 = X

In that case, a tree-based algorithm will make several rules from many categories using the same count column. But how I said at the beginning, algorithms generalize among populations of similar counts so most likely you will find rules like:

  If Column Counts is < 10 and N is > 3 = X

Which will often contain different categories that behave similarly. Just check the models and look for parameters/ importance of the column in question to see this for yourself.

  Secondly, I was wondering if anyone may be willing to share other techniques of dealing with categorical variables.

Feature hashing became really popular there not so long ago. Hashing has very nice properties and it's a whole topic learned at schools but the main principle is that if you have a category with high cardinality you decide a minimum number of reduced categories (hashes) that all the categories will have to share. if two categories share the same hash or bucket, that is called a hash collision. Feature hashing doesn't deal with hash collisions because according to some authors (I don't have the reference here) may improve accuracy by forcing the algorithm to pick more carefully the features.

There are many ways we can encode these categorical variables as numbers and use them in the algorithm.

1) One Hot Encoding
2) Label Encoding
3) Ordinal Encoding
4) Helmert Encoding
5) Binary Encoding
6) Frequency Encoding
7) Mean Encoding
8) Weight of Evidence Encoding
9) Probability Ratio Encoding
10) Hashing Encoding
11) Backward Difference Encoding
12) Leave One Out Encoding
13) James-Stein Encoding
14) M-estimator Encoding

Find the below cheatsheet

More Info


Multivariate time series forecast with VAR confusion

In essence, you want to both incorporate the past historical values of the target timeseries and the (past and) current historical values of other timeseries to predict the current value of the target timeseries.
In the tutorial you gave, they define the VAR[1] model with two time series ($Y_{1, t}$, $Y_{2, t}$) as:
$$ Y_{1,t} = \alpha_{1} + \beta_{11,1}Y_{1,t-1} + \beta_{12,1}Y_{2,t-1} +\epsilon_{1,t}$$
$$ Y_{2,t} = \alpha_{2} + \beta_{21,1}Y_{1,t-1} + \beta_{22,1}Y_{2,t-1} +\epsilon_{2,t}$$
where the author defined $\alpha$ as the intercept, $\beta$ the lag coefficients and $\epsilon$ as the error term.
Let us consider $Y_{1,t}$ as the target time series. It seems that it could be modified such that it incorporates the past and current values of $Y_{2,t}$ as follows:
$$ Y_{1,t} = \alpha_{1} + \beta_{11,1}Y_{1,t-1} + \beta_{12,1}Y_{2,t-1} + \beta_{12,0}Y_{2, t} + \epsilon_{1,t}$$
Here, the term $\beta_{12,0}Y_{2, t}$ has been added, which signifies the dependence on the current value of the other time series. In fact, this equation is the only relevant one when determining the $\beta$ parameters, as $Y_{2,t}$ is known in the test set.
Generalizing this to an arbitrary number of time series $n$ and lag $l$, the model for your target timeseries becomes:
$$ Y_{1,t} = \text{VAR}[l] + \sum^{n}_{i=2}\beta_{1i,0}Y_{i, t}$$
with VAR$[l]$ indicating the original VAR model from the article.
EDIT:
Worked example based on sample data you provided. We have time series $X,Y,Z$. Let us model our time series with lag = 1. Then our model for $Z_{t}$ becomes:
$$Z_{t} = (\alpha + \beta_{ZZ, 1}Z_{t-1} + \beta_{ZY,1}Y_{t-1} + \beta_{ZX,1}X_{t-1} + \epsilon_{t}) + \beta_{ZY,0}Y_{t} + \beta_{ZX,0}X_{t}$$
The values between brackets is standard vector autoregression. We can then use a simple linear regression procedure. Note that we only have two samples to use for fitting, because at Day 1 there are no previous timepoints. Thus we have the following equations:
$$Z_{2} = 60 = (\alpha + \beta_{ZZ, 1}Z_{1}+ \beta_{ZY,1}Y_{1} + \beta_{ZX,1}X_{1}) + \beta_{ZY,0}Y_{2} + \beta_{ZX,0}X_{2}) = (\alpha + \beta_{ZZ, 1}30 + \beta_{ZY,1}20 + \beta_{ZX,1}11) + \beta_{ZY,0}40 + \beta_{ZX,0}22)$$
$$Z_{3} = 90 = (\alpha + \beta_{ZZ, 1}Z_{2}+ \beta_{ZY,1}Y_{2} + \beta_{ZX,1}X_{2}) + \beta_{ZY,0}Y_{3} + \beta_{ZX,0}X_{3}) = (\alpha + \beta_{ZZ, 1}60 + \beta_{ZY,1}40 + \beta_{ZX,1}22) + \beta_{ZY,0}60 + \beta_{ZX,0}33)$$
Using python we can solve this as follows:
from sklearn.linear_model import LinearRegression

#Z-vals
Z_1 = 30
Z_2 = 60
Z_3 = 90

#Y-vals
Y_1 = 20
Y_2 = 40
Y_3 = 60

#X-vals
X_1 = 11
X_2 = 22
X_3 = 33

LR = LinearRegression()

result = LR.fit([[Z_1, Y_1, X_1, Y_2, X_2],
                [Z_2, Y_2, X_2, Y_3, X_3]],[Z_2, Z_3])

print(f'Alfa is {result.intercept_}')
print(f'Parameters are {result.coef_}')

which yields
Alfa is 21.95159629248201
Parameters are [0.46343975 0.30895984 0.16992791 0.30895984 0.16992791]

And thus our predictive model for Z has become (after rounding):
$$Z_{t} = 21.95 + 0.46Z_{t-1} + 0.31Y_{t-1} + 0.17X_{t-1} + 0.31Y_{t} + 0.17X_{t}$$


What's the difference between finding the average Euclidean distance and using inertia_ in KMeans in sklearn?

You will notice that the inertia is the sum of squared distance between each point and its nearest cluster center. Thus by removing the normalization term / data.shape[0] and instead replacing that by a squared **2 these two expressions will be equivalent

distortions_2.append(sum(np.min(cdist(data
                              , kmeanModel.cluster_centers_
                              , 'euclidean')
                              , axis = 1)) **2)

and

distortions_2.append(kmeanModel.inertia_)

For example using this artificial data source, where we generate 5 Gaussian distributions with 300 points in each.

params = [[[ 0,1],  [ 0,1]],
          [[ 5,1],  [ 5,1]],
          [[-2,5],  [ 2,5]],
          [[ 2,1],  [ 2,1]],
          [[-5,1],  [-5,1]]]

n = 300
dims = len(params[0])

data = []
y = []
for ix, i in enumerate(params):
    inst = np.random.randn(n, dims)
    for dim in range(dims):
        inst[:,dim] = params[ix][dim][0]+params[ix][dim][1]*inst[:,dim]
        label = ix + np.zeros(n)

    if len(data) == 0: data = inst
    else: data = np.append( data, inst, axis= 0)
    if len(y) == 0: y = label
    else: y = np.append(y, label)

num_clusters = len(params)

We will then apply KMeans using different number of clusters

k = [1,2,3,4,5,6,7,8,9,10]

inertias = []
dists = []

for i in k:
    kmeans = KMeans(i)
    kmeans.fit(data)
    inertias.append(kmeans.inertia_)
    dists.append(sum(np.min(spatial.distance.cdist(data, kmeans.cluster_centers_, 'euclidean'), axis=1)**2))

plt.plot(range(1, len(inertias)+1), inertias, label = 'Inertia')
plt.plot(range(1, len(dists)+1), dists, label = 'Distance')
plt.legend()
plt.show()


How to aggregate data where instances occur over different time intervals

I had the same problem. you can use aggregation functions. For example use Max, Min, Avg, count, std or some calculation like the slope of the line. In cases which you have a different period, you can divide your value to days of each period.


Can not install spacy package on windows 10 via pip

I was getting this exact same error trying to install spaCy into a 32-bit Python virtual environment. The latest version of spaCy I was trying to install (2.3.2) requires 64-bit Python.  When I rebuilt my virtual environment as 64-bit and then tried to install spaCy, it worked fine.
Just make sure you are trying to install spaCy into a 64-bit Python environment and you should be good to go.


Why GA convergence curves continue as two parallel lines?

As you know MATLAB plots GA result with two curves, one for the best values and other to show the mean values and when this two curves touch each others it means algorithm has been converged.

More accurately when the two curves touch each other it means that all the individuals in the population perform exactly the same, because that's the only way for the best and the mean to be equal (and usually this happens because all the individuals are identical). That can happen only by chance and/or if the mutation probability is very small, because as soon as there is an individual with a mutation it's unlikely that this individual performance will be equal to the best.

To me in a plot like yours successful convergence was reached around 60-70 iterations. The residual difference is due to the random mutations.


Is is possible to train on Hypergraphs in Keras?

As far as I'm aware, deep learning on hypergraphs is still a relatively new area, so I don't think there's any ready-made solution for hypergraphs. I did find this repo, which implements some models in keras to accompany a recent paper on hypergraph learning, but it is hardly a library.
You may also check out this paper, which cites a pair of techniques for converting a hypergraph to a graph:

Notably, previous methods [for hypergraph learning] typically decompose the hyperedge into pair-wise relationships where the decomposition methods can be divided into two categories: explicit and implicit. For instance, given a hyperedge (v_1,v_2,v_3), the explicit approach would decompose it directly into three edges, $(v_1,v_2), (v_2,v_3), (v_1,v_3)$, while the implicit approach would add a hidden node $e$ representing the hyperedge before decomposition, i.e., $(v_1,e),(v_2,e),(v_3,e)$.

If you're okay with doing this, then there are a few options besides Spektral:
Deep Graph Library provides a Tensorflow.keras compatible API, although their documentation seems to favor PyTorch.
There is also Graph Nets, which is built on top of Tensorflow.keras, although it's quite bare-bones at the moment.


It's really hard to understand your code but it seems that you are only seeing the last fold's y_pred . Create a list and in each iteration append the prediction to it.


Unable to use pip package obtained from building Tensorflow 2.3 from source

It's working with symbolic link:
libcudart.so.11.1 -> libcudart.so.11.0
I guess it's because the nightly build has CUDA 11.1 fully supported.


Another good normalization is zScore normalization. Was already implemented in python in the module scipy.stats


How to split the data as per the label name count using sklearn

Use a stratified test split.

import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
X = np.array([[1, 3], [3, 7], [2, 4], [4, 8]])
y = np.array([0, 1, 0, 1])
stratSplit = StratifiedShuffleSplit(y, n_iter=1, test_size=0.5, random_state=42)
for train_idx, test_idx in stratSplit:
    X_train=X[train_idx]
    y_train=y[train_idx]

print(X_train)
# [[3 7]
#  [2 4]]
print(y_train)
# [1 0]

Source: https://stackoverflow.com/a/35473089/9524424


Recommended data cleaning techniques for multivariate time series prediction?

3 questions:

Does it make sense to transform the data into a stationary time series when using LSTM?
Always Stationarity is always desired property and data should be transformed (read more)
Should the data be normally distributed when you are using PCA? No
There are multiple assumptions around PCA IF you use some matrix factorization techniques. For example if using SVD you should make sure that your matrix is of full rang.
Does it make sense to impute/fill (by forward filling or something else) big gaps or is it just better in that case to ignore the missing data completely? If for certain features a lot of data is missing you should drop them all together. Dont try to impute it, you will add false information.


Is there any problem with dropping only part of the OneHot generated features?

I think you can keep as many as you want, and it'll be alright. Sometimes it's even worth to delete the very rare classes to have more stable features.
In addition, for linear regression, you shouldn't include all of them, as you might have a collinearity issue.
To sum up, no problem with not keeping them all.


How to add extra word features other then word Embedding in Recurrent Neural Network model

One option is to concatenate them, the second is to treat them as separate inputs. For example Keras offers such neural model: https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models


Continuously training one model with different dimensions each time

You could try to summarise the data before handling it to the model:

If you have 5 readers, you could gather the information of "mean", "max", "min", "standard deviation" of them, this will work on 10, 15, any N of readers. Unless there is something which stops you from calculating mean, max, etc.


Import csv file contents into pyspark dataframes

"How can I import a .csv file into pyspark dataframes ?"
-- there are many ways to do this; the simplest would be to start up pyspark with Databrick's spark-csv module. You can do this by starting pyspark with

pyspark --packages com.databricks:spark-csv_2.10:1.4.0

then you can follow the following steps:

from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)

df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('cars.csv')

The other method would be to read in the text file as an rdd using

myrdd = sc.textFile("yourfile.csv").map(lambda line: line.split(","))

Then transform your data so that every item is in the correct format for the schema (i.e. Ints, Strings, Floats, etc.). You'll want to then use

>>> from pyspark.sql import Row
>>> Person = Row('name', 'age')
>>> person = rdd.map(lambda r: Person(*r))
>>> df2 = sqlContext.createDataFrame(person)
>>> df2.collect()
[Row(name=u'Alice', age=1)]
>>> from pyspark.sql.types import *
>>> schema = StructType([
...    StructField("name", StringType(), True),
...    StructField("age", IntegerType(), True)])
>>> df3 = sqlContext.createDataFrame(rdd, schema)
>>> df3.collect()
[Row(name=u'Alice', age=1)]

Reference: http://spark.apache.org/docs/1.6.1/api/python/pyspark.sql.html#pyspark.sql.Row

"Also, please tell me how can I import xlsx file?" -- Excel files are not used in "Big Data"; Spark is meant to be used with large files or databases. If you have an Excel file that is 50GB in size, then you're doing things wrong. Excel wouldn't even be able to open a file that size; from my experience, anything above 20MB and Excel dies.


Should the different layers of deep learning models have same size or they should be changed based on a rule

Actually it is not completely clear which deep neural nets you are referring to but I guess you are referring to Dense, aka fully connected, networks. It depends on your data but for simplification, based on the answer here, neurons of the first hidden layer try to find lines for separating the data, and the neurons of the subsequent layers try to consider them simultaneously, I suggest reading the hole answer. As you increase the number of neurons in the first layer, you try to increase the number of lines for separation, The last shape of the link, and as you try to increase the number of e.g. the second layer, the number of convex shapes will increase. If you increase them too much you will over-fit your data.

Increasing the number of filters, neurons or kernels, in convolutional nets has another meaning. If you increase the number of filters, it means that you try to find more features that can represent your data better. In conv layers you actually try to find features of data. Although you are somehow doing same procedure in a dense net but because the neurons are less connected and they get updated similarly in conv nets, the features learned by conv nets are completely different from dense layers. for more information I recommend you taking a look at here.


decision -tree regression to avoid multicollinearity for regression model?

To answer your questions directly, first:

Is there a decision tree regression model good when 10 features are
high correlated?

Yes, definitely. But even better than decision trees, is many decision trees (RandomForest, Gradient Boosting (xGBoost is popular). I think you'd be well served by learning about how decision trees split, and how they naturally deal with collinearity. Maybe try this video Follow the logic until the 2nd tier of splits, and you'll be able to imagine how the correlated variables are suddenly not important because they're relative to the split above them.

Is there a scientific or math explanation or recommendation (to use
decision tree regression)?

The math explanation of why collinearity is "bad" for linear models, comes down to the coefficients and how you interpret them. One of the side effects is that they can undermine the statistical significance of a variable, as well as flip their coefficients the wrong direction. It usually doesn't affect the accuracy of the model very much, but most people want linear models so that they can interpret the coefficients (which is totally messed up with collinearity). I suggest reading maybe this article to start.
One of the things that you mentioned, include all variables if possible? is not really something you should be concerned with. The goal of a model is to explain the most, with the least. If you're forcing as many variables as possible into the model, then it's possible that you'll be fooled into thinking a model is good, when in fact it isn't if you were to test it on new data. In fact, sometimes less variables will give you a better model. This is exactly the kind of problem that multicollinearity causes with linear models - that you can't really judge very well what variables are significant or not. Stepwise selection doesn't work very well when there are correlated features.
In general, I think decision trees - especially Random Forests - will be a good start for you. But remember not to force all of the variables into the model just for the sake of it. Experiment with using less variables and manipulating the tree structure such as leaf size and max depth. And as always - test your model on validation data and holdout data so that you don't overfit a model and fool yourself into thinking it's a strong model.