Which algorithms should I use for identifying similar characteristics between data points (the intersections)?

You could achieve this by creating and analyzing a confusion matrix of characteristics that appear together. Here is an example:

From this image, you can see (from the price-price section) that price appears 4 times. Then, you can also see that (from the price-quality section) that price and quality appear 3 times together. So, you can conclude that 75% of the time that price is a characteristic, quality is also a characteristic.

Other information you can extract:

price appears the most
price and quality is the most common pair
service_and_support is only 50% of the time paired with quality

Below is the code to generate this plot:

import seaborn as sn
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

characteristics = {'quality':0, 'service_and_support':1, 'price':2, 'customer_relations':3}

data_points = [
    ['quality', 'service_and_support', 'price'],
    ['quality', 'price'],
    ['quality', 'customer_relations', 'price'],
    ['service_and_support', 'customer_relations', 'price'],
]

counts_matrix = np.zeros((len(characteristics), len(characteristics)))

for data_point in data_points:
    for characteristic1 in data_point:
        for characteristic2 in data_point:
            counts_matrix[characteristics[characteristic1]][characteristics[characteristic2]] += 1

keys = list(sorted(characteristics.keys(), key=lambda x: characteristics[x]))

df_cm = pd.DataFrame(counts_matrix, index = keys, columns = keys)
plt.figure(figsize = (10,7))
sn.heatmap(df_cm, annot=True)

plt.show()


Advice on what Machine Learning Algorithms to study for a Job to candidate matching algorithm

Very likely, there is too much noise in the data. Meaning there are hidden pieces of information that are dominating the decision for acceptance or not, which you don't find in the data. For example, if the client finds the consultant sympathic. Or if the client had a good day when deciding about the consultant.

I doubt that the rule system is overly complicated, so my advise is to stay with it.

If you want to get started with machine learning, look for MNIST examples (a handwritten digits dataset) and I recommend to use Keras with Tensorflow as a Backend.

(by the way: I'm a machine learning ;-) )


I guess you have not figured out the concept of dropout very well. First, the reason we apply it is that we add some noise to the architecture in order not be dependant on any special node. The reason is that it was observed that while training a network, after overfitting, the weights for some of neurons increases and cause the network to be dependant on them. By exploiting dropout, we are not dependant on any node anymore due to it is possible to drop it while training.

Now, answers to your question. First, you have to bear this point in mind that the probability shows the chance of dropping a node in a layer. Consequently, chance 0.5 does not mean you, for instance, will have those two nodes. It just means after employing dropout, the chance of dropping for each node is half. Dropout is used for layers. It is customary to use it in fully connected layers. You set the hyper-parameter and it is the chance of keeping the nodes in the layer. While testing, you don't drop any node. We don't multiply neurons to the probability. The probability specifies the chance of existence of that node.

Okey-doke! I update the answer. As you can read in the paper,

  At test time, it is not feasible to explicitly average the predictions from exponentially
  many thinned models. However, a very simple approximate averaging method works well in
  practice. The idea is to use a single neural net at test time without dropout. The weights
  of this network are scaled-down versions of the trained weights. If a unit is retained with
  probability p during training, the outgoing weights of that unit are multiplied by p at test
  time as shown in Figure 2. This ensures that for any hidden unit the expected output (under
  the distribution used to drop units at training time) is the same as the actual output at
  test time. By doing this scaling, 2n networks with shared weights can be combined into
  a single neural network to be used at test time. We found that training a network with
  dropout and using this approximate averaging method at test time leads to signicantly
  lower generalization error on a wide variety of classication problems compared to training
  with other regularization methods.

I guess the easiest way to understand it is to watch this video. As you can see there are different implementations for that but the reason it is multiplied is that for any hidden unit the expected output (under the distribution used to drop units at training time) is the same as the actual output at test time. To be concise, it is done in order not to change the distribution that the outputs of the layer have.


Quick guide into training highly imbalanced data sets

Max Kuhn covers this well in Ch16 of Applied Predictive Modeling.
As mentioned in the linked thread, imbalanced data is essentially a cost sensitive training problem. Thus any cost sensitive approach is applicable to imbalanced data.
There are a large number of such approaches. Not all implemented in R: C50, weighted SVMs are options. Jous-boost. Rusboost I think is only available as Matlab code.
I don't use Weka, but believe it has a large number of cost sensitive classifiers.
Handling imbalanced datasets: A review: Sotiris Kotsiantis, Dimitris Kanellopoulos, Panayiotis Pintelas'
On the Class Imbalance Problem: Xinjian Guo, Yilong Yin, Cailing Dong, Gongping Yang, Guangtong Zhou


Resource and useful tips on Transfer Learning in NLP

If you use pre-trained models on data that are distinct from the data they were originally trained on, it's transfer learning. Your two-class sentence corpus is distinct from the data that the GloVe embeddings were generated on, so this could be considered a form of transfer learning. This might be a helpful explainer for general ideas around pre-training (and why it's a worthy pursuit).

Recent work in the NLP transfer learning space that I'm aware of is ULMFiT by Howard and Ruder of fast.ai, here's the paper if you prefer that. OpenAI also has recent work extending the Transformer model with a unsupervised pre-training, task specific fine-tuning approach.

As for your task, I think it might be helpful to explore research around sentence classification rather than digging deeply into transfer learning. For your purposes, it seems that embeddings are a means to have a reasonable representation of your data rather than prove that Common Crawl (or some other dataset) extends to your corpus.

Hope that helps, good luck!


As mentioned in another answer, you should consider using a database. In the long run it will make your life easier.

But, R can help you automate things. You can store excel files in specific folders and/or with specific names (or patterns of names ex. file1_day1, file1_day2 etc) and then create R scripts that process them and produce the report you want.

Since you are using excel files (nothing personal with excel, it is a great program) I am inclined to think that your data will fit in a decent computer running R. In any case, if you end up using R, you should check out data.table package.

Furthermore, R can help you create more complicate reports. Plus, its learning path is not too steep.


Looking for a data set that shows hospital patients' illnesses and their vital signs, during their stay

I'm assuming you are looking for medical datasets, not only what you have stated in the question, but also some image recognition datasets which need something like CNN.

Try this one. This dataset contains several medical features including blood sugar, serum cholesterol etc, and wants you to find out the presence of heart disease.

In the meanwhile, there are some medical competitions and datasets on Kaggle, including the famous Data Science Bowl.


I think you can approach it as a logistic regression problem, where you will have a on/off (1/0) feature called "exposure_to_image". And your goal will be to detect if that coefficient is statistically significantly different than 0. If yes, then the probability of eating the item was influenced by the exposure.

As for the assumption that they would have eaten it anyway:
That's the whole point of having the control group. So your dataset will have people where exposure_to_image=1, and people for who exposure_to_image=0. All of them will have some baseline probability of wanting to eat the cake. But that coefficient tells you by how much that baseline is affected.


what indicators can be used to classify a group of stocks?

Question 1:
Geograhic location, market sector, it all depends on what you're trying to achieve with the clustering.
Question 2:
Yes. Adding too many indicators will cause your data to overfit and at some point you'll end up with every cluster having 1 member. I would use a regularisation penalty term to counter this. Either L1 (Lasso Regression) or L2 (Ridge regression) would work for this. Or you can use Elastic Net which would use a portion of both.


Is there an advantage to use decision trees in OneVsAll than using "classical multiclass trees"?

In general OneVsAll is a device useful when you want to transform a binary classifier into a multi-class one. This happens when models cannot handle multiple classes or when it is difficult to adapt it. By doing so, you have to normalize the probability estimates, otherwise they are not too useful, being rather non comparable. That alone is probably a sufficient reason why nobody use this when a multi class version is available.
There are cases when something similar might be useful. For example when you have a lot of classes. One might try to group the target levels, build a classifier for groups and later do subsequent classifiers for each group. But there are not overlaps between predictions.


Syntaxnet parser could surely help you in parsing the sentences and in tree represenation.

If you plan on solving using RNNs, I believe Tree LSTM will be a better choice than LSTM, as it also preserves dependency information.
Full paper.

Use Tree LSTM, if you need a vector embedding for the whole sentence. For use cases like classification, sentiment analysis. It works and there is a good probability that the vector could have all the information but you may not be certain and thats why its still a black box. But, if your use case is a clear representation of tree structure and logic among the terms (which you wanted), better go with parsers like Syntaxnet and try rule-based models for the use case mentioned.


looking for public dataset for stock market

You can have a look at the kaggle stock dataset.

https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs

This questions are normally done in OpenData stack exchange.

https://opendata.stackexchange.com/


Extracting embeddings of Categorical Variables

As far as I understand, usually embeddings are initialized with random values. If there are pretrained embeddings, they can be loaded, but of course there are no such things for categorical variables.

So if you simply create an embedding of a categorical feature, you'll get a vector representation, but it will have random variables. So at first you'll need to train the neural net. After it is trained, you could take the output of the embedding layer and use it in regression model.

But are you sure that you need the embedding? You could do mean encoding or frequency encoding.


Ranking tumors from most malignant to most benign

Most classification algorithms actually provide continuous scores that are compared to a given threshold to give a binary output. Using that score directly give you a ranking. But unless you give us a specific algorithm, it would be difficult to help you further.

You usually can find that trough the performance metrics. AUC for exemple, has a general definition obtained by varying the above mentionned threshold. However, it has a more natural interpretation (and way faster approximation) when you use a definition based on individual scores.

Some models can even be calibrated in probabilities. That means, under some conditions, the continuous score can be intepreted as a probability to be of a given class.


Cardinality vs width in the ResNext architecture

Like in InceptionNet, which increases width, the different paths are depth-concatenated as well as each path is different (i.e different filter sizes for different paths).

While in ResNeXt, which increases cardinality, the different paths are merged by adding them together, and each path is similar.

Read this article for more information An Overview of ResNet and its Variants


The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is.

According to some: Standardization of categorical variables might be not natural. Neither is standarization of Tf-Idf because according to stats stack exchange:

  (it's) (...) usually is a two-fold normalization.

  First, each document is normalized to length 1, so there is no bias
  for longer or shorter documents. This equals taking the relative
  frequencies instead of the absolute term counts. This is "TF".

  Second, IDF then is a cross-document normalization, that puts less
  weight on common terms, and more weight on rare terms, by normalizing
  (weighting) each word with the inverse in-corpus frequency.

Tf-Idf is meant to be used in its raw form in an algorithm. Other numerical values are the ones that could be normalized if the algorithm needs normalization or the data is just too small. Other options can be using algorithms resistant to different ranges and distributions like tree based models or simply using regularization, it's up to the cross-validation results really.

But categorical features like bag-of-words, tf-idf or other nlp transformations should be left alone for better results.

However, there is also the idea of normalizing one-hot coded variables as something that can be done as a standard step same as in other datasets. And it's presented by a prominent figure in the field of statistics.

https://stats.stackexchange.com/a/120600/90513


Structure of NN for input data with drop out

There is no Special reason that neural networks are suitable, if anything it could be too much (depending on the complexities of your time-series).
In any case, you need to deal with you Nan values and if you are looking for a fast model, that imputes that automatically, I would suggest lgbm. Since it will impute NAN with values that reduce the out of fold loss the most (making the highly optimal imputation).


Tokenizing words of length 1, what would happen if I do topic modeling?

The libraries usually exclude 1-length tokens and tokens with no alpha-numeric characters because typically they are noise and do not have any descriptive power. That is, these tokens are usually not helpful, say, in distinguishing between relevant vs not relevant documents.

However, if in your domain you feel like 1-length tokens can be helpful, feel free to use them as well. For example, if all the document that contain 1 belong to the same topic, it may be a good idea to preserve this token. 1 has descriptive power in this case: it can help distinguish between one particular topic and the rest.

Now, your next question is about LSI. For LSI there is no difference if a column in the document-term matrix corresponds to a 1-char token or to a 5-char token. So you can use LSI in your analysis.


Does it make sense to train an Autoencoder for Dimensionality Reduction using Mini-Batch Gradient Descent?

In general, using an autoencoder for dimensionality reduction using mini-batch gradient descent is perfectly fine.

You should pay attention to the neural architecture you use in order to have the appropriate inductive bias, e.g. image autoencoders tend to use convolutional layers to profit from pattern locality.

You should also assess how well the input data is reconstructed, to ensure that the information is properly retained in the compressed representation.

However, as you did not specify the final destination of the data (i.e. what the reduced-dimensionality data is going to be used for), we can't tell whether the dimensionality reduction would be appropriate in for your specific purposes.


When I try to predict with my model I get an Attribute error

In cell 10 of your notebook you write:

model = build_model() #Perfect

In cell 11 you write:

model = model.summary() #Uh oh!!

later you call

example_result = model.predict(example_batch) #model.summary() returns None

and python tells you

  AttributeError: 'NoneType' object has no attribute 'predict'

This is because you reassigned model in cell 11 to, well, nothing. You should remove the model = in cell 11 and your code will run perfectly!

redo cell 11 to read:

 model.summary()


Assigning vector of weights to Neural Network

From the documentation we can find out that:

weights: (case) weights for each example – if missing defaults to 1.
Wts: initial parameter vector. If missing chosen at random.

Though I'm no R expert, my understanding is:

weights is something like sample_weight in python, which controls the weight of each sample. It's useful when dealing with imbalanced class issue.
Wts is the initial weights of neural network. Random weights will be initiated if this parameter is not passed.


Bias variance tradeoff boosting (xgboost) vs random forest (randomized bagging) which to use when?

The decision trees used in gradient boosting are typically shallow decision trees (with only a few nodes). Limiting the depth or number of nodes in the decision tree makes them simple. This is different from fully developed decision trees used as standalone models.


John Langford's documentation on GitHub could help. You can find something on the Learning to Search Sub System page.


What open-source books (or other materials) provide a relatively thorough overview of data science?

One book that's freely available is "The Elements of Statistical Learning" by Hastie, Tibshirani, and Friedman (published by Springer): see Tibshirani's website.

Another fantastic source, although it isn't a book, is Andrew Ng's Machine Learning course on Coursera. This has a much more applied-focus than the above book, and Prof. Ng does a great job of explaining the thinking behind several different machine learning algorithms/situations.


How to show two pictures in one cell in Jupyter Notebook? (matplotlib) (python)

One way to do it (without getting into lots of the inner-workings of Jupyter notebooks), it to use two matplotlib Axes in one plot. Then you show one image in each of these:
from skimage import data
image_coffee = data.coffee()
image_horse = data.horse()
fig, axs = plt.subplots(1, 2, figsize=(15, 8))  # one row of Axes, two columns = 2 plots

The axs variables is a list containing the two Axes, so just access each one and plot your image on it like this:
axs[0].imshow(image_coffee)
axs[1].imshow(image_horse)

If the plots don't pop automatically, either run plt.show() or make sure your notebook has executed %matplotlib inline in a cell.


I would need to understand certain aspects of your dataset to give you a better answer but:

The reason you're receiving NULL Values:

Depending on the class distribution, You may receive null predictions due to not having classes("repoId") in both samples of the original data. So classes that exist in test data may not exist in the training data. So when you apply the transform to the test data it has no basis on which to make a prediction on the supplied data. When you use "coldStartStrategy" it just omits those records altogether.

I would recommend first setting "ColdStartStrategy" to False just to see if all your records are just returning Null prediction values.

If that is the case, You would then need to check class distributions in both the training and test samples of your data. This should be the class distribution for "repoId". Then you would have to sample the data in a way that makes sure classes are present in both samples then reapply.

Potential flaws in your process:

Use seeding in "randomSplit" so your samples are always reproducible therefore you can detect issues regardless of when you ran your program
You may have assumed equal class distribution in your dataset
Normalizing your "Repocount" values would benefit the algorithm, but likely has no bearing on the overall results


How to increase accuracy of model from tensorflow model zoo?

Are you training the models from scratch? If yes, then can you try using pre-trained models and fine-tune for your specific dataset.

You'll have the experiment with hyperparameters. Learning rate and optimizer (e.g., sgd, adam, rmsprop, adadelta) will have largest effect on model training and performance.


K-means

Your data has $7$ dimensions so k-means is worth to try. See the PCA of your data and check if any cluster is visible there as K-means will have a tough time if clusters are not Gaussian. the setup is:

Each person is a point in $7D$ space (a $50\times7$ matrix)
Apply PCA and inspect it.
If different clusters visible then you will have a result

Fuzzy C-Means

I would suggest soft clustering algorithms. Soft clustering means that output is not binary (each sample belong only to one cluster and does not belong to others) but it assigns a membership score for belongness of each sample to each cluster. It minimizes the same objective function as K-means but with a weight which is calculated in each iteration and can be found here. Libraries usually have this algorithm under the name FCM.

K-means Through GMM

Another soft version of K-means is called Gaussian Mixture Models in which you try to estimate Gaussian kernels whose superposition describes the data (as you see again the Gaussian distribution of data is crucial here). The setup is:

Choose a $k$ as the initial number of clusters and produce $k$ arbitrary Gaussian kernels (i.e. with arbitrary $\mu$ and $\sigma$)
Use Expectation Maximization algorithm for updating the new clustering structure at each step.

Spectral Clustering

Define a Similarity Matrix from the data by any means. For example calculate the distances between points in $7D$ space and reverse that. Or apply a RBF kernel for determining the similarity between them. Then calculate the second eigenvalue-eigenvector pair according to sorted eigenvalues. Use K-means to cluster those elements of eigenvectors.

If you have more detailed questions just drop a comment here. Good Luck!


Do timesteps must have the same temporal distance in training a RNN?

As always in ML modelling problems: it depends.

The critical factor here is that you are predicting based on properties of a sequence. The sequence does not need to be sampled at fixed time steps, or even be time based at all. E.g. in natural language processing the sequence of letters or words is only very loosely associated with the timing of the same items when spoken or read. The sequence does need to be drawn from a self-consistent source where moving from one item to next item in the sequence has the same meaning with respect to what you are trying to predict.

If the interruptions you see would cause large differences in the expected distribution of your input vector X or output Y, then you may not be able to use them as-is. You may still be able to rescue them as input data for a RNN, and make predictions based on them, if you add whether or not there has been an interruption - and if it might be relevant to your purpose, for how long - as additional features of input items in X.

Another option is to follow the time sequence strictly but add placeholder value of X (with a different flag set) for "unkown", maybe along with some rough guess for the missing features (or even a ML prediction from a different model), which you might do if the quantity being aggregated over is still active and important where there are missing records, but the aggregation service you are using as input has failed.

Depending on how much data you have available, and whether you need to make predictions at times when your aggregation service has failed, you could do anything from discarding incomplete data to ignoring the time differences, to interpolating missing records.

The first question you need to answer is "what does it mean for a sequence item to be missing":

If the whole system that provided the aggregation data is switched off, and there is no concept of pending items that will enter the system the moment it is switched on, then you can probably safely ignore the missing records and ignore the precise timing.
If your prediction target is sensitive to elapsed time, and yo uneed to make predictions even when there have been interruptions to the aggregation system, then you will want to augment the sequence data somehow to account for the missing records, and train/predict with that added feature engineering.
If only the aggregation is switched off (or has a chance of failure causing gaps in the record), and you don't need to run any predictions when it is not available, then you may be able to make the rule that the prediction system will only function when there are 6 recent consecutive records available, and discard the data that is causing you to worry. This makes sense if interruptions are rare or the predictive model is not critical.

If you are not sure which of these things to do, then the next approach is to do all of them that seem that they could be intuitively correct, tuning each model separately, then see how each different approach performs on your test set.


How to split natural language script into segments?

I haven't seen anything like this before but it seems quite feasible. You need an ontology to separate the main concept into its subconcepts, then you need a classifier to distinguish between your broader categories; description, methodology, classifier, application, and example. That is, I would manually label some transcripts at the paragraph level. If you don't have paragraph-segmented text, smooth the classification probability over the sentences so they share the same label. Or use a paragraph segmentation model as in Automatic Paragraph Segmentation with Lexical and Prosodic Features.

The classifier could be a CRF or an RNN. The modern way to induce the ontology would be through word embeddings; cf. e.g, Learning Semantic Hierarchies via Word Embeddings. Formerly I would have recommended a hierarchical topic model, such as hLDA; cf. e.g, Non-Parametric Estimation of Topic Hierarchies from Texts with Hierarchical Dirichlet Processes or Unsupervised Terminological Ontology Learning based on Hierarchical Topic Modeling

Welcome to the site and good luck!


Difference between $Q(s,a)$ ,$V^*(s)$ and $V^\pi(s)$ in Markov Decision Process?

Your confusion seems to come from mixing up between some policy $\pi$ and an optimal policy $\pi^*$. Your summary is generally correct, but missing these extra details.
Let me try go through it again. Starting with the MDP definitions:

First of all, we have the transition probabilities $T(s,a,s') = P(s'|s,a)$ which are conditional probabilities of arriving at state $s'$, given that we've taken action $a$ in state $s$
The (expected) reward is generally associated with the state-action pair $R(s,a)$ - there is some slight variations about it in literature, but that's a subject for a different discussion.
Then we have the policy $\pi(a,s) = P(a|s)$ - the probability of taking an action $a$ in state $s$ for an agent following the policy.

Given all three things above: $T$, $R$ and $\pi$ (plus a discount factor $\gamma$) you can define your value functions as average discounted rewards collected by the agent that follows $\pi$.

$V^\pi(s)$ is the average reward that the agent following $\pi$ will collect, when starting from the state $s$.
$Q^\pi(s,a)$ is the average reward that the agent following $\pi$ will collect, when starting from the state $s$ and taking an action $a$.

Quite often authors drop the index $\pi$ and assume implicitly that we are dealing with some policy $\pi$, but one should keep in mind that some (maybe unspecified) policy is always in the context when we are talking about value functions $V$ or $Q$.
These value functions satisfy the following recursive relationships for any policy $\pi$ (note that there's no $\max_a$ in these):
$$V^\pi(s) = \sum_a\pi(a,s)\left[R(s,a) + \gamma \sum_{s'}T(s',s,a)V^\pi(s')\right]$$
$$Q^\pi(s,a) = R(s,a) + \gamma \sum_{s',a'}T(s',s,a)\pi(a'|s')Q^\pi(s',a')$$
Now, some policies maximize the expected reward - these policies are called "optimal" and standardly denoted with a star: $\pi^*(a,s)$ - optimal policy. The  index $\pi$ for the value functions is usually replaced with a star as well: $V^*$ and $Q^*$. The optimal value functions satisfy Bellman equations (the ones that have $\max_a$ in them):
$$V^*(s) = \max_a\left[R(s,a) + \gamma \sum_{s'}T(s',s,a)V^*(s')\right]$$
$$Q^*(s,a) = R(s,a) + \gamma \max_{a'}\sum_{s'}T(s',s,a)Q^*(s',a')$$
Hope this clarifies it.


Python3 Tensorflow: Difference Between truncated_normal and uniform_normal when creating weights

A simple Google search would show you the differences.

Truncated Normal outputs random values from a truncated normal distribution. The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.
Random Uniform outputs random values from a uniform distribution. The generated values follow a uniform distribution in the range [minval, maxval). The lower bound minval is included in the range, while the upper bound maxval is excluded.

Regarding when is it better to use one or the other, we don't know. Weights just need to be random, and there is no conclusive research that shows that one or another distribution is better for specific scenarios. That being said, there are better ways to initialize weights that completely randomly. These include the very popular Xavier initializer.


Why do people use CrossEntropyLoss and not just a softmax probability as the loss?

Let me explain it with logistic regression first:

The output of logistic regression is $\hat{y}= \dfrac{1}{1 + e^{-\theta x}}$
Consider the minimum square loss function $\mathcal{L} = (1 -\hat{y})^2$
substitute $\hat{y}$ in the second equation $\mathcal{L} = (\dfrac{e^\theta x}{1 + e^{-\theta x}})^2$

Tha above formula is not convex, so optimization is much more difficult since the gradient may point to local optima.

Whereas cross entropy looks like.$$\mathcal{L} = -y\log \hat{y} - (1-y)\log (1-\hat{y}) $$ which has two advantages.

It is convex when subtitute $\hat{y}$ for its value (considering that $y$'s value is either 1 or 0)
For each row in your dataset you are taking on count both labels at the same time

When we generalize from binari logistic regression to multiclass classification, we use the same ideas of using convex loss functions with respect to the output and also loss functions which minimize at the same time for all labels.


Tips and tricks for designing time-series variational autoencoders

I can speak from a more theoretical point of view, but honestly I haven't had much success with VAEs.

1) How deep should my encoder and decoder network be? Are there any good guidelines?

That depends entirely on your dataset. If you have highly nonlinear data, then a deep network should do well. The successive nonlinearities allow you to capture higher-order correlations in the input, which might be good for your situation.

Do you have any idea of the number of truly independent dimensions in your dataset? Have you done something like PCA (a linear autoencoder)? That might tell you where to aim your dimensionality of your bottleneck layer, and that might help you determine your layer sizes to get there.

2) Should one use fully connected dense networks, or stacked conv1Dnets?

I'd say try both.

3) What activation functions are good choices?

Hugo Larochelle once said that you should always start with ReLUs. See if you get a good enough result with them, as they aren't as prone to the problem of exploding/vanishing gradients, which is something you're going to face with time series data. Remember to initalize them with small positive values, though, to avoid dead neurons.

[EDIT: actually, vanishing and exploding gradients can still be an issue with ReLUs, it's just mitigated a bit. Also, initialize the bias with small positive values. Andrej Karpathy said so in a lecture]

4) Can we say anything about the 'best' dimensionality of the latent dimension?

In some contexts yes, we can. The 'best' dimensionality will be the one that results in the highest lossless compression. That is, the one that compresses your input data the most without losing information when you reconstruct. Good luck finding that optimum, though.

5) Like everyone else I imagine, my loss function is the sum of a reconstruction term, and the KL divergence regularization term. Is there something else one should consider?

You could incorporate various types of regularization that will modify your loss function. You could use dropout, for example, or L1 or L2 regularization.

EDIT: you could also consider an attention-based model, for example see this discussion.

6) Batch normalization? Currently, on my problem, it doesn't make a blind bit of difference. But it should. Should one always use batch-norm?

'Always' is a strong term. I would say not always, but batch normalization ought to be a good choice. Have another look at your implementation before you give up on it.

7) In the decoder layer, is it better to up-sample before returning to the original input dimension?

That's the default presentation - an autoencoder is usually shown as a symmetric construct from the input to the hidden to the reconstruction. I would say experiment with not doing so, but I don't know if one is better than the other.


Possible scenarios:

You can predict "Likely to Buy Probabilities": Create features that covers before purchase (e.g: product click, product view, purchases from same category etc). After, train a classifier and use prediction probabilities  (predict_proba_) to obtain scores. Then sort them pick x% and report them to business unit. They can offer discounts, campaigns to converge customers fast.
Try to find segment transitions. For example coke buyers move to lets say pie segments faster than vegetable segment. Identify those transitions and apply advertisement on cross categories.
If you have transaction data you should try to use apriori based algorithms and collaborative filtering methods for recommendations.
Additionally you can do RFM analysis and compare RFM segments and already created segments to find best customers of yours.

Hope this is helpful!


What are the main types of NLP annotators?

Here are the basic Natural Language Processing capabilities (or annotators) that are usually necessary to extract language units from textual data for sake of search and other applications:

Sentence breaker - to split text (usually, text paragraphs) to sentences. Even in English it can be hard for some cases like "Mr. and Mrs. Brown stay in room no. 20."

Tokenizer - to split text or sentences to words or word-level units, including punctuation. This task is not trivial for languages with no spaces and no stable understanding of word boundaries (e.g. Chinese, Japanese)

Part-of-speech Tagger - to guess part of speech of each word in the context of sentence; usually each word is assigned a so-called POS-tag from a tagset developed in advance to serve your final task (for example, parsing).

Lemmatizer - to convert a given word into its canonical form (lemma). Usually you need to know the word's POS-tag. For example, word "heating" as gerund must be converted to "heat", but as noun it must be left unchanged.

Parser - to perform syntactic analysis of the sentence and build a syntactic tree or graph. There're two main ways to represent syntactic structure of sentence: via constituency or dependency.

Summarizer - to generate a short summary of the text by selecting a set of top informative sentences of the document, representing its main idea. However can be done in more intelligent manner than just selecting the sentences from existing ones.

Named Entity Recognition - to extract so-called named entities from the text. Named entities are the chunks of words from text, which refer to an entity of certain type. The types may include: geographic locations (countries, cities, rivers, ...), person names, organization names etc. Before going into NER task you must understand what do you want to get and, possible, predefine a taxonomy of named entity types to resolve.

Coreference Resolution - to group named entities (or, depending on your task, any other text units) into clusters corresponding to a single real object/meaning. For example, "B. Gates", "William Gates", "Founder of Microsoft" etc. in one text may mean the same person, referenced by using different expressions.

There're many other interesting NLP applications/annotators (see NLP tasks category), sentiment analysis, machine translation etc.). There're many books on this, the classical book: "Speech and Language Processing" by Daniel Jurafsky and James H. Martin., but it can be too detailed for you.


This is most likely because sklearn has chosen your majority class for you and you simply need to weigh your classes the other way around with "0" being the majority class.


The dictionary word2id does not contain the word "i". I can run the Kaggle Kernel and do not get the error.
When I run the code my word2id dictionary looks like this:

{'i': 0,
 'feel': 1,
 'awful': 2,
 'about': 3,
 'it': 4,
 'too': 5,
 'because': 6,...}

The word "i" is definitely in there, I think you did not run the code to create the word2id dictionary properly. Make sure to run all the cells of the kernel.


Question on bias-variance tradeoff and means of optimization

There are a lot of ways bias and variance can be minimized and despite the popular saying it isn't always a tradeoff.

The two main reasons for high bias are insufficient model capacity and underfitting because the training phase wasn't complete. For example, if you have a very complex problem to solve (e.g. image recognition) and you use a model of low capacity (e.g. linear regression) this model would have high bias as a result of the model not being able of grasp the complexity of the problem.

The main reason for high variance is overfitting on the training set.

That being said there are ways of reducing both bias and variance on a ML model. For example the easiest way of achieving this is getting more data (in some cases even synthetic data help).

What we tend to do in practice is:

First, we increase the capacity of the model in order to reduce the variance on the training set as much as possible. In other words we want to make the model overfit (even reach a loss of 0 on the training set). This is done because we want to make sure the model has the capacity of sufficiently understanding the data.
Then we try to reduce the bias. This is done through regularization (early stopping, norm penalties, dropout, etc.)


Can Random Forest regressor or Decision trees handle missing values and outliers?

No, not all of your assumptions are true

1. Missing values

The sklearn implementation of RandomForest does not handle missing values internally without clear instructions/added code. So while remedies (e.g. missing value imputation, etc.) are readily available within sklearn you DO have to deal with missing values before training the model.

This involves understanding how and why values are missing (MCAR VS. MAR VS. ...) and determining the best way to deal with those values (median/mean imputation, MICE, removing values, etc.).

2. Outliers and Skewness

Again this isn't a done deal! You need some EDA to understand your data here and handle it accordingly. Strong biases in the distribution of the dependend variable (so-called class imbalances) can heavily influence the result. Look into ways to handle those like SMOTE.

Outliers are less of a problem, granted they really ARE outliers. Check via EDA and confirm this isn't a problem to be dealt with.

3. Feature selection and transformation

It is true that many ML models favor a more-is-more approach to feature selection. The main benefit of using RandomForest, XGB over classical statistical approaches is that they cope much better with irrelevant predictors.

Still feature selection also means feature engineering which is still helpful and necessary. Additionally for practical purposes you should still aim to reduce predictor amount if you want to use your model again. Because no matter how irrelevant every predictor used in the training model NEEDS TO BE PRESENT in the unseen data as well.

Lastly you will need to transform data! Simply example, RandomForest needs one-hot-encoding (OHE) for factor variables and string variables. You do not need the same scale for all your numerical variables but they all need to be numerical!


Why is T test reweighting on a word X word co-occurrence matrix so effective?

IT is very similiar to PMI, here you just expand it to the whole dictionary matrix (matrix representation of the whole vocabulary), normalize it by subtracting quantitive representation of the sum of words found in row i column j and than standardize. (Like when using sklearn Standardize(), similiar atleast)
Intuition? Well why is tf-idf working (generally as text quantification technique), you are focusing on essential n-grams and minimising away the rest, with this re-weighting you are getting close to tf-idf representation in a sence.


In a regular fully connected layer (Dense), the computation is done using the following Matrix operation : $$R = A*W + B$$
With all matrixes being vectors (if batch size = 1), exept $W$, which has size(inputsize, outputsize).
for a 3D input, the way TF computes the output is simply by applying this formula only to the last dimension, considering all other dimensions as similar to batch sizes.
So in your example with input of size (4,3,2), Tensorflow 'flattens' the vector into a (12, 2) matrix, computes the result as if it was a 2D input, this gives a (12, 5) vector, and then 'unflattens' the vector to give back its 3 dimensions (4,3,5).


Each time the result of XGBoost is different

You are missing some seeds there:

Numpy
Pandas
StratifiedKFold

I will say that StratifiedKFold(shuffle=True) without a seed can make your results change.


How to generate ratings without training data?

If you model your system by means of reinforcement learning, you will make your system learn from the user's feedback. The system will provide a rating based on the input features (it may be just a random rating during the first stages, since you will not have any prior information), and then the user will tell the system how well it predicted such rating. Based on the difference between the suggested and the actual user's ratings, the reinforcement learning algorithm will refine the recommendation system's model in order to provide more accurate ratings in the future.

Sutton's book on reinforcement learning (http://incompleteideas.net/book/the-book-2nd.html) is a good introduction to the reinforcement learning field.


Output trained parameters of Keras model

There is a way to do this. See the documentation! It is very good.

I assume you mean that you build a network using Keras (which contains recurrent GRU layers) and would like to save the model after some training, then restart from the same point e.g. with new data or just to push the model further.

Example

Assuming you want to analyse some images of shape (150, 150, 3):

from keras.models import Model, load_model
from keras.layers import Input, GRU, Dense

my_model = Model()

my_input = Input(shape=(150, 150, 3))    # using Tensorflow's dims: 'channels last'
gru_layer - GRU()(input)
output = Dense(32)(gru_layer)

# tell the Model class which layers are the start and end of the model
model = Model(inputs=my_input, outputs=my_output)

# save it!
my_model.save('my_model.h5')

# some time later.... a new python session
my_reloaded_model = load_model('my_model.h5')

# continue using the model

Explanation

At the point the model is built, we can compile and train it the usual way e.g. my_model.fit(...)

Once training is finished, we can save the model two different ways:

just the model weights, using my_model.save_weights('my_weights.h5')
the whole model and its meta data, using my_model.save('my_model.h5')

In either case, you will need the library h5py (see the documentation)

Option 2 preserves more information from the model. Quoting the documentation, it saves:

  the architecture of the model, allowing to re-create the model
  the weights of the model
  the training configuration (loss, optimizer)
  the state of the optimizer, allowing to resume training exactly where you left off.

To continue on the with model where you ended and saved, it is as simple as:

my_model = keras.models.load_model('my_models.h5')

Now you can train it further or introspect the model and so on.

Links

Details on the Model class
Details of GRU
Details of saving models


Intuition behind using non-hypercubic kernels in density estimation

Histograms and methods based on binning have a number of well-known problems. Different anchor points etc. can introduce artificial patterns that make interpretation unreliable. Smooth kernels don't use a grid and thus smooth out the noise.

This also has the advantage that it makes it easier to get a single overall picture of the data because it takes into account neighboring points and smooths the data into areas where no data is observed.

Smooth kernels can also be justified by their favorable statistical properties. Popular methods like fastKDE use the fact that one can find "an empirical kernel that is optimal in the sense that the integrated, squared difference between the resulting KDE and the true PDF is minimized."


The initial weights of h for GRU and h,c for LSTM are are often set to zeros, setting random weights is also an option. Also people have tried to learn the initial hidden states.
Since the hidden states are updated with every cell, if your sequences are long enough, it would not make a big difference how you initialize the hidden states.


Compute parameters of a PDF (probability density function) for which no closed form expression is available

Your problem could be solved either by direct numeric integration or by MCMC.

Numeric integration can be performed most easily by scipy:

import numpy as np
import scipy.stats
import scipy.integrate

def weird_density(x):
    """ The function I want to sample """
    return scipy.stats.lognorm.pdf(x, 1.0)

def quad_quantile(fun, q, precision=1e-10, minus_inf=-10e10, lower=-10e10, upper=10e10):
    """ Use bisection to evaluate a quantile """
    while upper - lower > precision:
        med = (upper + lower) * 0.5
        cdf_med = scipy.integrate.quad(fun, minus_inf, med)[0]
        if cdf_med < q:
            lower = med
        else:
            upper = med
    return (upper + lower) * 0.5

print('true mean  :', scipy.stats.lognorm(1).mean())
print('integrated mean:', scipy.integrate.quad(lambda x: weird_density(x) * x, 0, 100)[0])
print('true median  :', scipy.stats.lognorm(1).ppf(0.5))
print('integrated median:', quad_quantile(weird_density, 0.5, 0.001, 0, 0, 100))

The output is

true mean  : 1.6487212707
integrated mean: 1.6484641126903046
true median  : 1.0
integrated median: 0.9998321533203125

However, if the distribution is multidimensional, direct integration may be intractable. In this case,  MCMC methods (Markov chain Monte Carlo) can help.

What they do is just make a correct (in some sense) sample from the distribution for which you know the PDF. When you have a sample, you can calculate all your parameters from it as classical sample statistics, just like from any observed data.

MCMC algorithms may be implemented manually, like in the example below. Another option is to use the PyMC3 library or its analogs.

One of the best known MCMC algorithms, Metropolis-Hastings, works as follows:

def sample_next(x):
    """ Generate random next point"""
    return scipy.stats.norm(loc=x).rvs(1)[0]

def metropolis_hastings(density, generator, n_samples, starting_point=1, random_state=None):
    """ Generate sample from the given density function """
    np.random.seed(random_state)
    result = [starting_point]
    for i in range(n_samples):
        current_point = result[-1]
        next_candidate = generator(current_point)
        acceptance_ratio = density(next_candidate) / density(current_point)
        if np.random.uniform() <= acceptance_ratio:
            result.append(next_candidate)
        else:
            result.append(current_point)
    return np.array(result)

sample = metropolis_hastings(weird_density, sample_next, 1000, random_state=1)

print('sample mean:', np.mean(sample))
print('sample median:', np.median(sample))

This code prints

sample mean: 1.2872857165
sample median: 0.932826883898

You see that the results are not very close, but with longer sampling (1000 points is too few) they will converge. And the shape of distribution is already well matched:

import matplotlib.pyplot as plt
plt.hist(sample, bins=30, normed=True)
idx = np.linspace(0, sample.max())
plt.plot(idx, scipy.stats.lognorm(1.0).pdf(idx))
plt.legend(['true density', 'sample density'])
plt.title('convergence of MCMC sample')
plt.show()


How to use Variational Autoencoder's μ and σ with user-generated z?

If you are training a VAE the encoder is essentially parameterizing the variational posterior distribution of $z$ given $x$, i.e.
$$ q(z | x) = \prod_{i=1}^{N_z} q(z_i | x) = \prod_{i=1}^{N_z} \frac{1}{\sqrt{2\pi}\sigma_i(x)}  \exp \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} \right] $$
where $\mu_i(x)$ and $\sigma_i(x)$ are given by the encoder and $z$ is in the $N_z$-dimensional latent space.
I would think of the problem as if the $x$ are parameters of a probability distribution and the $z$ are some observations you made.
The "nearest" encoding from your training data $x$ would be the encoding with the highest likelihood, i.e. you compute the likelihood for each data point for a given $z$ by evaluating the above expression and take the $x$ with the maxmal value.

The log-likelihood is usually used in these scenarios because it's more convenient, but it is equivalent, as the likelihood is non-negative and the log is a monotonic function.

In the comments you mentioned to use a distance metric. The log-likelihood provides a nice interpretation, because it gives you something similar to the negative euclidean distance between $\mu(x)$ and $z$, but scales and shifts by terms determined by the standard deviation:

$$ \log q(z|x) = \sum_{i=1}^{N_z} \log q(z_i|x) = \sum_{i=1}^{N_z} \left[ -\frac{(z_i - \mu_i(x))^2}{2\sigma_i(x)^2} - \log \left( \sqrt{2\pi}\sigma_i(x) \right) \right] $$

So intutively, by maximizing the (log-)likelihood, you are minimizing the euclidean distance between a given z and the encoding $\mu(x)$ of $x$ from the training data set, but you pay penalties for having large variances.

(Furthermore, if you do it this way, there is no sampling of $\epsilon$ required.)


Caret + RStudio: Error "Please make sure `y` is a factor or numeric value" when training

Have you done any pre-processing or data manipulation before training the model?

Its hard for me to say what is the reason the error, but I have tried to do the same task and it worked whithout errors, here is my code:

library(mlbench)
library(caret)
df <- PimaIndiansDiabetes

#Using x, y arguments

modelo <- train(x = df[, 1:8] , y = df$diabetes, method='mlp')

#Using formula
modelo <- train(diabetes ~.,  data = df, method='mlp')

Both approaches valid, and produce result like this:

> modelo <- train(x = df[, 1:8] , y = df$diabetes, method='mlp')
> modelo
Multi-Layer Perceptron

768 samples
  8 predictor
  2 classes: 'neg', 'pos'

No pre-processing
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 768, 768, 768, 768, 768, 768, ...
Resampling results across tuning parameters:

  size  Accuracy   Kappa
  1     0.6545936   0.0000000000
  3     0.6416382  -0.0008031577
  5     0.6397277   0.0002662915

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was size = 1.

Please note that your results might differ a little.


Analysis of passed/not passed students by enrolled year

To start with, you could use a simple thresholding.
If you have the dataset $S$ where an element has the form $(x,y,c) \in S$, $x$ denotes the year,  $y$ is a binary value (exam passed or not), and $c$ is the student id.
you can obtain a classifier by using
$\{(x,y,c) \in S \mid x \leq \theta\}$ and $\{(x,y,c) \in S \mid x > \theta\}$.
Now you can check all plausible $\theta$ values (..,2008,2009,..,2020) to see how good your data can be seperated. For example you could use Matthews correlation coefficient to evalute which $\theta$ is best.
But If you want to see if there is some linear correlation, yes you could compute the correlation coefficient, but I would assume that there is no linear correlation. Essentially you are computing the Point-biserial correlation coefficient.
Also with the thresholding method, in case you see that all $\theta$ perform more or less the same, and badly, you can see that there is no linear correlation.
You should have a look at histograms to see how your data is distributed.


Methods / Algorithms for rank scales based on cumulative scoring

To solve challenges #3 and #4, let's limit the overall available rank volume. For example, sum of this rank for all the users will be 1 (100%).

From challenge #2 I understood, that you accept 2 different ranks: (1) place from 1 to 100, and (2) simple sum of all earned points (fixed and relative). Did I got it right? I so, there is no need to worry about unlimited growth, or fixed scores inflation. Let's just use percentages, not 1-100 ranks.

These percentage ranks could be calculated based on interaction behaviors (vote/selecting answer/etc), using PageRank-like algorithm. Such algorithm will consider all previous reactions (and ranks of acted users), obtained by an exact user. Unfortunately, you cannot use PageRank algorithm "as is", because it supports only "positive" links, but you can look for it's extensions. For example, look at this paper with PageRank extension  for both positive and negative links (as users can down vote). You can iteratively estimate percentage rank (TrustRank, TR) using this algorithm.

The second task is to calculate reward/penalty rate in points for each single action. Let's determine (predefine) maximal reward/penalty rate (X) for each type of action. And will use coefficient to discount it, based on TrustRanks of acting users (e.g., author and voter). Slightly modified Sigmoid will map this ratio from [-Inf,+Inf] range to [0,1].

Here for peer users you will have ~0.5 of predefined maximal rate. If "voter" has TR twice more than "author", "author" will recieve ~0.75 of predefined value, and so on. You can tune steepness with additional parameter, or try to find any other mapping transformation function.

Anyway, now simply multiply maximal penalty/reward by this coefficient, and you'll get the number of point, you need to deduct or add. The only issue, I see, is a user with zero TR - such user as a voter will "give" nothing, and as an object of voting, will recieve the maximal amount of points regardless voter's rank. To avoid this, you can predefine minimal TR (like 1e-10), and don't let user's TR to fall beyond this value.


How do I group similar type of skills together?

If all you have is the name/word, then I see only two ways:

You can look at word similarity, basically by measuring how similar the spelling is. There are a number of possible metrics (e.g. Levenshtein distance). This will probably not get you very far.
You can look at semantic similarity. You can use pre-trained word embeddings to map the words to a new vector space where you can calculate the distance between the word embeddings, e.g. with word2vec or other implementations. Alternatively, you can use a dictionary/corpus (e.g. https://www.nltk.org/index.html) that provides you a word taxonomy that allows you to compute semantic similarity between words via relationships like synonyms etc. See here for different approaches.


Different runs of feature selection algorithm giving different set of selected feature. How to choose the best set among them?

You can set a seed or a random state for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.

This can be done by set_seed(...) in R, and adding the random_state = ... in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.


packages installed after activating conda environment

It will only affect the environment that you activated.


How to create a model to recognize matching label and ROI with OCR

There are multiple ways to do it. Here is one technique:
Run OCR on your image and search for your desired text . in this case it is Total.
Get the pixels of that word - run image_to_data method of ocr to get pixels.
Now you can extract the ROI using those pixels via OpenCV.


Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

BERT is a transformer based model. The pre-trained BERT can be used for two purposes:

Fine-tuning: This involves, fine-tuning with the new data to a specific task such as classification or question-answering, etc. Here, the BERT itself acts like a classifier.
Extracting embeddings: Here, you can extract the pretrained embeddings. The difference between Word2Vec (or other word embeddings) and BERT is that BERT provides contextual embeddings, meaning, the embeddings of each word depends on its neighbouring words. However, since it's contextual embeddings, we can make an assumption that the fist token which is '[CLS]' captures the context can be treated as sentence embeddings as be used as input to 'SVM' or other classifer. But, in the case of RNN, you may want to take the pretrained embeddings of each token to form a sequence.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.


What's the issue with my code for visualizing linear regression in 3 dimensions with matplotlib?

Following the suggestion given by @Ubikuity, I changed the ax.plot to ax.scatter, but there were still some issues with the dimensions. It was due to the transposition of the matrices, that was being done in the wrong place. The code below works correctly.
X = df[["Idade","LF"]]
y = df["DGAF"].values.reshape(len(df["DGAF"]),1)

reg = LinearRegression()
reg.fit(X, y)

fig = plt.figure()
ax = Axes3D(fig)

ax.scatter(X['Idade'],X['LF'], y,s=10)

x_pred = np.linspace(0,100,100)
y_pred = np.linspace(0,100,100)
xx_pred, yy_pred = np.meshgrid(x_pred, y_pred)

model_viz = np.array([xx_pred.flatten(), yy_pred.flatten()])

ax.set_xlabel('Years since oil change')
ax.set_ylabel('LF')
ax.set_zlabel('DGA Score')

ax.scatter(model_viz[0], model_viz[1], reg.predict(model_viz.T),color='red')
plt.show()
```


What tool to find expected and hidden patterns in data?

You are unlikely to find a tool that will truly do this for you. Various bits of software may produce some nice graphs. However we're not at a stage where tools can generate interesting questions to ask, ask them, and then explain the results to you.

If this is commercial, your best bet is to hire someone with an appropriate background and set them some questions. Tell them what your biggest goals and costs are, explain the data to them, and see what happens.  So the answer is a contractor.


pandas count values for last 7 days from each date

The main obstacle is figuring out whether a date is within the last 7 days of the month. I'd recommend something hacky like the following:

from datetime import datetime, date, timedelta
def last7(datestr):
    orig = datetime.strptime(datestr,'%Y-%m-%d')
    plus7 = orig+timedelta(7)
    return plus7.month != orig.month

Once you have that, it's relatively simple to adapt your previous code:

df3 = df1[df1['is_buy'] == 1 && last7(df1['date'])].groupby(['id', 'month']).agg({'is_buy': np.sum})

Now we just join together df2 and df3 and we're done.


Cross-validation: K-fold vs Repeated random sub-sampling

If you have an adequate number of samples and want to use all the data, then k-fold cross-validation is the way to go. Having ~1,500 seems like a lot but whether it is adequate for k-fold cross-validation also depends on the dimensionality of the data (number of attributes and number of attribute values). For example, if each observation has 100 attributes, then 1,500 observations is low.

Another potential downside to k-fold cross-validation is the possibility of a single, extreme outlier skewing the results. For example, if you have one extreme outlier that can heavily bias your classifier, then in a 10-fold cross-validation, 9 of the 10 partitions will be affected (though for random forests, I don't think you would have that problem).

Random subsampling (e.g., bootstrap sampling) is preferable when you are either undersampled or when you have the situation above, where you don't want each observation to appear in k-1 folds.


Why doesn't training RNNs use 100% of the GPU?

I get about this same utilization rate when I train models using Tensorflow. The reason is pretty clear in my case, I'm manually choosing a random batch of samples and calling the optimization for each batch separately.

That means that each batch of data is in main memory, it's then copied into GPU memory where the rest of the model is, then forward/back propagation and update is performed in-gpu, then execution is handed back to my code where I grab another batch and call optimize on it.

There's a faster way to do that if you spend a few hours setting up Tensorflow to do batch loading in parallel from pre-prepared TF records.

I realize you may or may not be using tensorflow under keras, but since my experience tends to produce very similar utilization numbers, I'm going out on a limb by suggesting that there's a reasonably likely causal link to draw from these correlations. If your framework is loading each batch from main memory into the GPU without the added efficiency/complexity of asynchronous loading (which the GPU itself can handle), then this would be an expected result.


value represents the number of items in each class.

If you look at the top node, you should view it as:

There are:

35100 samples of class 0
16288 samples of class 1
which sums up to 51388 samples total


Mathematical algorithms to generate labels for training data

Assuming we have a lot of data, just not many data are labeled.

Try to tackle the problem first by using an unsupervised approach, following that, use the learned features for any downstream task.
see Deep Generative Models.

  generate training labels based on some pre-determined experts rules

If you have some function that can generate the data distribution then why you bother yourself with such finding yet another function that approximate the former.
you could use this approach for evaluation purposes, e.g. compare different models regarding the problem being solved.

  What are the approaches to label more data for training

you could aggregate (average) different models that tackle the same problem but with a different dataset.

  Could Maximum Likelihood Estimator be used here.

As I said, we could maximize the likelihood of the data, giving high probability to samples that are very similar to the training data is very useful at exploiting the internal structure (learning the data manifold) that could potentially disentangle the factors that are responsible for generating the data which presumably part of them could be a direct signal for predicting the target y that you are trying to predict.


Great answer by Leevo, just let me point out one thing: Perfect multicollinearity means that one variable is a linear combination of another. Say you have $x_1$ and $x_2$, where $x_2 = \gamma x_1$. This causes various problems as discussed in this post.

The main takeway (to put it simple) is, that $x_1$ and $x_2$ basically carry the same information (just "scaled" by $\gamma$ in the case of $x_1$). So there is no benefit of including both. In fact there is a problem with including both since multicollinearity will "confuse" the model because there is no unique effect of $x_1, x_2$, when considered jointly, on some outcome $y$.

Look the following situation (R code):

y = c(5,2,9,10)
x1 = c(2,4,6,8)                           ### = 2   * x2
x2 = c(1,2,3,4)                           ### = 0.5 * x1
cor(x1, x2, method = c("pearson"))

The correlation between $x_1$ and $x_2$ equals 1 (so of course a linear combination). Now when I try to make a simple linear OLS regression:

lm(y~x1+x2)

The result is:

Coefficients:
(Intercept)           x1           x2
        1.0          1.1           NA

The second term has been dropped by R (due to perfect multicollinearity).

We can run a regression on each term separately:

Call:
lm(formula = y ~ x1)

Coefficients:
(Intercept)           x1
        1.0          1.1

...and...

Call:
lm(formula = y ~ x2)

Coefficients:
(Intercept)           x2
        1.0          2.2

Now you can see that the coefficient for $\beta_2$ is simply $2\beta_1$ because $x_1$ is $2 x_2$. So nothing to learn from including both, $x_1, x_2$ since there is no additional information.

Basically the same problem can occur if correlation between $x_1,x_2$ is really high. See some more discussion in this post. Thus given strong correlation, one should be cautious to include both variables. The reason is that in this case, your model cannot really tell apart the effect of $x_1$ and $x_2$ on some outcome $y$, so that you may end up with weak predictions (among other problems).


To debug this case, I suggest you try the following steps:

Reduce the features step-by-step until you end up with using just 1 feature and see whether the accuracy changes or not.
Add a sine-wave and a random noise to the feature set and see whether it effects any of these optimization algorithms.
Re-evaluate how you selected or derived these features, check if these are highly correlated.
Are your classification targets highly imbalanced? If so, then under/over sample them to achieve a more balanced training set. Then check the performance of you algorithm after training over this balanced dataset.

As already highlighted by Jan van der Vegt, its extremely odd that changing the no of features from 8 to 40 has no impact on test set accuracy.


As I said, maybe there is already a public domain vector file that fits your needs, e.g.

library(raster)
map <- getData("GADM", country="HUN", level=2)
plot(map, col=rainbow(length(map)), border=NA)

Otherwise, try a dedicated tool to polygonize your raster image and save it as a shapefile. E.g. using QGIS (German):

After saving the shapefile, you can load and process it in R:

library(raster)
map <- shapefile("evk_106_20140101.shp")
plot(map, col=rainbow(length(map)), border=NA)

(Side note: I assume it's ok for any copyright owner oto modify and use the image for demonstrational purposes in the above way. If not, drop a comment and I'll delete it.)


How to best visualize or capture time interval between lab measurements?

The average or median seem reasonable here. A boxplot might be a good choice of visualistion:

The y-axis could be your time dimension, and boxplot shows the normal distributions of differences, including the median value and outliers. You could then compare certain measurements against this. But it would of course depend on if you have enough data in your sample to represent the total population.
Or simply calculate the standard deviation of differences, and report this with your median (the mean value may be warped by outliers).


How to deal with position bias in search?

What are your output (prediction) and inputs?

The bias in itself is very important in predicting the click-through rate, but definitely less useful if you want to assess your ad copy/ page title in relation to the click-through rate.

One potential way to do that is to model the impact of the position of the ad / (or page, if it's for SEO), and try to normalize the data either by including the impact factor in your model or adjusting the result by the impact factor.

I would suspect it would be a log function, as the number of clicks tend to drop significantly after the top 1/2 of the first page.


MLR is similar to caret, MLR offers a high-level interface to various statistical and machine learning packages.

According to the package description:

  Interface to a large number of classification and regression
  techniques, including machine-readable parameter descriptions. There
  is also an experimental extension for survival analysis, clustering
  and general, example-specific cost-sensitive learning. Generic
  resampling, including cross-validation, bootstrapping and subsampling.
  Hyperparameter tuning with modern optimization techniques, for single-
  and multi-objective problems. Filter and wrapper methods for feature
  selection. Extension of basic learners with additional operations
  common in machine learning, also allowing for easy nested resampling.
  Most operations can be parallelized.

Please check the Reference and Homepage.


Problem with Linear Regression and Gradient Descent

Gradient descent probably didn't converge. I'd sugest scaling your feature on a [-1,1] or [0,1] scales, try increasing your learning rate and maybe add some code which checks the gradient's slope and if there is perhaps sequence of <0,>0,<0 it means that the learning rate is locally too large and cost function can't converge. In that case, learning rate should be decreased.


Correct Way of Displaying Features in Decision Tree

If you pass header=None, pandas.read_csv assumes that the first row contains data and names the columns '0' to '12'.

Instead you should pass header=0 to specify that the column names are in the first row or equivalently skip the header argument.

You can then still continue with X = balance_data.values[:, 1:12], because calling values returns a numpy array without the column names.

Alternatively, you could also select your feature columns like so:

feature_names = ['A','AAAA',....]
X = balance_data[feature_names].values

You can then pass the same list of feature_names to graphviz.

Also note that you don't have to pass a numpy array to scikit-learn's functions. It can handle pandas DataFrames as well, so values is optional.


I got the following error : 'DataFrame' object has no attribute 'data' can you help please

As pointed out in the error message, a pandas.DataFrame object has no attribute named feature names. You probably meant something like df1.columns.


First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems.

You said you have cosine similarity between your records, so this is actually a distance matrix. You can use this matrix as an input into some clustering algorithm.

Now, I'd suggest to start with hierarchical clustering - it does not require defined number of clusters and you can either input data and select a distance, or input a distance matrix (where you calculated the distance in some way).

Note that the hierarchical clustering is expensive to calculate, so if you have a lot of data, you can start with just sample.


Speaking as someone from a finance background, the `usual' model for a stock price process is

$\frac{dS}{S}=r dt + \sigma dW_t$

i.e. we assume the returns (not the absolute price changes, i.e. dS/S is approximately the daily percentage change in price) as having a 'drift' equal to the risk free rate (the interest rate r) and a random shock $dW_t$ with standard deviation $\sigma$

So it's basically the opposite of what you've been asked, I would expect a trend (but it's very hard to estimate with confidence) and I would not expect any seasonality, you could fit an arithmetic model by taking the daily differences though.


Choosing weights on random forest for imbalanced data with the aim to minimize false positives

I totally missed out on the fact, that the RandomForestClassifier accepts the values “balanced” and “balanced_subsample” to automatically assign the weights as per documentation


When optimizing the MSE, the correlation between prediction and target increases?

It should not be lower but it does not always have to be higher.

Let's consider these two vectors:

A | B
1 | 2
2 | 3
3 | 4

The correlation between A & B is super high as I've just modeled a deterministic relationship (B = A +1). However if I perceive B as a predicted value and A as the real value then the MSE would not be 0.

Now let's say I improve my prediction and it's perfect:

A | B
1 | 1
2 | 2
3 | 3

We haven't impacted the correlation between those vectors at all because we simply removed the constant shift between the vectors. Their relation is still 1:1, however the MSE if B is the prediction and A is the real value is now much, much better than in the first example.


Create multiple matrices from 2 bigger ones in R

This will extract the rows from matrix A and B.

matrix(c(A[x,],rep(B[x,],times=3)),nrow=2,byrow=T)

If you want to get them into a list (recommend)

single<-lapply(c(1:1000), function(x) matrix(c(A[x,],rep(B[x,],times=3)),nrow=2,byrow=T))

To put them all in diagonal matrix with adiag from magic package

d<-adiag(single[[1]])

for(i in 2:1000){
  d<-adiag(d,single[[i]])
}

I could make it work without the loop (anyone any suggestions?)


How to build a data analysis pipeline procedure

Nomially, I would simply just try to write a bash script (or powershell in windows) and just string the commands together.  However, this approach is rather fragile as in things get overwritten, and if it's an end to end process that has long batches.

I tend to use a workflow packages like luigi or airflow when stringing dependent task together.  The idea for Luigi is that you can break each action into a task.  Each task has three needed functions.

Requirement - what needs to exist before this task runs?
Output - where is the output going?
Run - what is the task?

So essentially you just chain a bunch of task and define your run function to call your previously built scripts using something like subprocess.  In requirement, you would reference the last step, and for output you would point towards where the file is being written.

The pro for doing this is that if your process breaks in task 50 out of 100, you don't have to rerun all 50 task, luigi will go down the dependency tree until it finds a requirement not fulfilled.

Calling R from Pytho
Luigi


How to quantify the numerical influence of categorical variables?

Yes you're talking about training a model, then doing sensitivity analysis to determine the numerical influence of each explanatory variable in isolation:

Some considerations/tips:

you said the three features are categorical. Presumably student_age is ordered categorical, and the other two are unordered. But you didn't say how many levels they each have (cardinality), or how many bins. You don't necessarily want to (say) bin student_age in bins of 1 year each. You might bin in say 2/3-year increments, or quantiles (see the R cut command), e.g. student_age: [0-17, 18-20, 21-23, 24-28, 29-34, 35+]. Similarly, if you have a long tail of smaller institutions with few exemplars, you might bin them together, or bin by country/ region/ size/ focus/ ranking.
what is the product of cardinality of each variable, vs the size of the dataset? Do you have 1 exemplar for each possible combination? >10? <0.1? If you don't have that much data, you might prefer to use Linear Regression rather than NN, or at least a shallower NN rather than deep. With Linear Regression you can allow quadratic interaction terms, nonlinear terms etc - again
(with Linear Regression you can directly see the influence just by directly reading the model coefficients e.g. Princeton: -1.4, Princeton.Age22_24: -3.6)
if you specifically want to use NN, see the lime package. Also see the many articles/tutorials on sensitivity analysis with NNs, and CrossValidated answers on sensitivity analysis.
deep NNs suffer from a lack of explainability; you can view the many individual layer weights but it's near-impossible to picture how that translates into effect on output.


Can we cross-validate the neuron no. and hidden layer no. of a feed-forward neural network?

Yes. validation is the best way to choose model hyperparameters such as neurons number.
Validation with neural net is not substantial different from validation with other models.
The metric should be chosen according to the task the network performs. The validation method should be chosen according to the amount of data, training time, certainty level you want to achieve and compute and time resources you have.


What model to use for matching two datasets

Let the credit card statement be the ground truth, and the receipts be the noisy inputs. For a given line item, find the receipt with the smallest distance. If the distance is small enough, declare a match. This is a threshold you will have to experimentally determine. You can let the distance be the sum of the distances for the amount and date. A heuristic for these individual distances is simply the edit distance. A more sophisticated approach would be to model the OCR error using labeled data to determine the most likely input; print something with known text in a similar typeface to the receipts so you can learn which characters are commonly confused with one another, and thereby estimate the most likely input sequence. Going another step further, you could jointly model the density between price, store, and item, so you can recognize for example that a vegetable bought at your local grocery store probably does not cost \$100 but \$1.00

If your OCR software gives you confidence estimates over the characters and possible guesses, you can use that too.


You can use OpenCV's Scene Text Detection https://docs.opencv.org/3.3.1/d4/d61/group__text.html


Is any feed-forward neural network a composable function?

I suppose you could write $ h\circ [g\circ f\ |\ f]$ if you argue that matrix augmentation is itself a multivariate function.
But to avoid pedantic arguments, you could use multivariate function composition (although it is uglier).
Define a multivariate concatenate function for column vectors:
$ c(\vec{x_1},\vec{x_2},...,\vec{x_n})=\left[
\begin{matrix}
\vec{x_1} \\ \vec{x_2} \\ \vdots \\ \vec{x_n}
\end{matrix}
\right]$
In a network $f\rightarrow g\rightarrow h$ having skip connection $f\rightarrow h$, you could then write: $ h\circ c|_{g\circ f,f}$.


How to import statsmodels module to use OLS class?

See https://stackoverflow.com/a/56284155/9524424

You need to have a matching scipy version (1.2 instead of 1.3)


Building a static local website using Rmarkdown: step by step procedure

In most things, related to R, there are many approaches to solve a problem, sometimes too many, I would say. The task of building a static website, using RMarkdown, is not an exception.

One of the best, albeit somewhat brief, sets of workflows on the topic include the following one by Daniel Wollschlaeger, which includes this workflow, based on R, nanoc and Jekyll, as well as this workflow, based on R and WordPress. Another good workflow is this one by Jason Bryer, which is focused on R(Markdown), Jekyll and GitHub Pages.

Not everyone likes GitHub Pages, Jekyll, Octopress and Ruby, so some people came up with alternative solutions. For example, this workflow by Edward Borasky is based on R and, for a static website generator, on Python-based Nicola (instead of Ruby-based Jekyll or nanoc). Speaking about static website generators, there are tons of them, in various programming languages, so, if you want to experiment, check this amazing website, listing almost all of them. Almost, because some are missing - for example, Samantha and Ghost, listed here.

Some other interesting workflows include this one by Joshua Lande, which is based on Jekyll and GitHub Pages, but includes some nice examples of customization for integrating a website with Disqus, Google Analytics and Twitter as well as getting custom URL for the site and more.

Those who want a pure R-based static site solution, now have some options, including rsmith (https://github.com/hadley/rsmith), a static site generator by Hadley Wickham, and Poirot (https://github.com/ramnathv/poirot), a static site generator by Ramnath Vaidyanathan.

Finally, I would like to mention an interesting project (from an open science perspective) that I recently ran across - an open source software by Mark Madsen for a lab notebook static site, which is based on GitHub Pages and Jekyll, but also supports pandoc, R, RMarkdown and knitr.


Perhaps you could transform the latitude and longitude into spherical coordinates. In this coordinate system, the cosine of the angle between the vectors has a natural geometric interpretation.


Changing duplicate a line hotkey in Jupyter notebook

You can change the Chrome shortcut like this:
In System Preferences > Keyboard > Shortcuts > App Shortcuts, click the [+] button, and select Google Chrome as the Application. Put Bookmarks->Bookmark All Tabs... in the Menu Title and a different Keyboard Shortcut.


Not all the algorithms provide you feature_importance, some of them give you coeff(weights W1..W7) values.
So according to me, the way you are extracting feature_importance is correct.

  However, this approach has the problem that out of n attributes, I
  will only get feature importance score for (n - 1) attributes as the
  nth attribute is the target variable.

Model gets trained based upon the features and target value. So you will get the feature_importance only for features and not for the target.

Using statistical approach you can also analyse the relationship between the various features and the target value.

As a side note:

Make sure you normalized the whole features on the same scale for
better and faster training.


What Naive Bayes method is being used in this example?

A basic Naive Bayes is being used in this example. Each feature can have a number of different values within the ranges of 2 or 3.

Bernoulli Naive Bayes requires that each feature be either true or false or 0 or 1.

Multinomial Naive Bayes allows features to be of values 0+ as it is counting occurrences of features.

Hope this helps. The Naive Bayes wikipedia pages summarizes this pretty well as well.


Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce

rhadoop (the part you are interested in is now called rmr2) is simply a client API for MapReduce written in R. You invoke MapReduce using R package APIs, and send an R function to the workers, where it is executed by an R interpreter locally. But it is otherwise exactly the same MapReduce.

You can call anything you like in R this way, but no R functions are themselves parallelized to use MapReduce in this way. The point is simply that you can invoke M/R from R. I don't think it somehow lets you do anything more magical than that.


Can I create a good Speech Recognition Engine while having millions of recorded conversations?

Yes, having lots of recorded conversations is great for building a speech recognition system. You will still have to create training samples (Each sample will be parts of Wave file --> text), but you will need lesser number of samples.

High level steps are :

Train a GAN on raw audio
Train a Language model on raw text data (it needs not be from these conversations, but has to be from same domain). For example, if conversations are related to medical, train language model on medical text.
Merge these models and train on labeled samples

For step 1, Google WaveNet is a good example (it is eventually used for Text-to-Speech, it is a component in Speech-to-Text as well)

https://deepmind.com/blog/wavenet-generative-model-raw-audio/

Papers that cover design and overall approach :

https://arxiv.org/abs/1711.01567
https://arxiv.org/abs/1803.10132


Model to predict based on frequency of occurrence

For something like this, you could go with a simpler approach. One idea is to sample randomly among the cities that a given passenger has visited using the amount of times each city has been visited as probabilites.

Here's a way you could do so. I've added a few more examples to the dataframe so that the application is seen more clearly. Say you instead have:

     Passenger    Trip
0       John     London
1       Jack     Girona
2       Jack      Paris
3        Joe     Sydney
4        Joe  Amsterdam
5        Joe  Barcelona
6        Joe  Barcelona
7       John     London
8       John      Paris
9       Jill    Newyork
10       Jim     Sydney
11      Jack      Paris
12     James     Sydney

You could define a function like the folllowing in order to randomly sample from the existing data in the dataframe:

def random_sample(df, name):
    import numpy as np
    # group the dataframe by Passenger and count
    # the different trips
    g = df.groupby('Passenger').Trip.value_counts()
    # Make the probabilities add up to 1
    freq = g[name] / g[name].sum()
    # random destination based on
    # its probabilities
    random_name = np.random.choice(a=freq.index, size=1,
                     p = freq.values)[0]
    # return likelyhood of next randomly chosen
    # destination and destination
    return freq[random_name], random_name

 Usage

Say we want to select a a randomly samples destination for say Joe and also to know which is the likelihood. Considering that the destinations where Joe has been are:

Trip
Barcelona    2
Amsterdam    1
Sydney       1

We could get for example:

for _ in range(5):
    freq, dest = random_sample(df, 'Joe')
    print('Chosen destination {} with a probability of {}'.format(dest, freq))

Chosen destination Sydney with a probability of 0.25
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Barcelona with a probability of 0.5
Chosen destination Sydney with a probability of 0.25


RandomizedSearchCV() not scoring all fits

Some of your hyperparameter values aren't allowed (colsample_bytree and subsample cannot be more than 1), so probably xgboost errors out and sklearn helpfully moves on to the next point, recording the score as NaN.
Half of your values for colsample_bytree are disallowed, which supports seeing half of your scores as NaN; and that will happen regardless of the fold, which explains why you always see them in groups of 10.


Should we normalize test data by choosing maximum and minimum value of training data?

You should use your training data fit function (by fit function i mean the function of the scaling that is learning from your data)simply because your test dataset size and feature values could be different than your training which would result in a scale different than the scale used on your training data. imagine you want to infer your model with 1 row which will be a problem if you try to fit 1 row on the minmaxscaler.

Example :
Let's say your max value is 100 and min value is 0 on your training data for a particular feature, you scale that to [0,1] => 100 will be 1, 0 will stay as so.
70 will be 0.7 , 60 = 0.6 and so on. You move on to the test set and apply scaling on it but it happens that the max value for that same feature in your test set is 70 ( 100 unfortunately for you happens to exist only in the training set ). Your 70 in the test set if you scale using the test set will be a 1. You feed the 1 ( 70 ) to your model who's trained to consider a 1 as a 100 thus the error.


Getting a transition matrix from a Adjacency matrix in python

Considering a is your adjacency matrix 2D numpy array :

a / a.sum(axis=0)

Should do the trick (divide all elements by columns sum)


How to perform (modified) t-test for multiple variables and multiple models on Python (Machine Learning)

This is a simple solution to my question. It only deals with two models and two variables, but you could easily have lists with the names of the classifiers and the metrics you want to analyze. For my purposes, I just change the values of COI, ROI_1, and ROI_2 respectively.

NOTE: This solution is also generalizable.
How? Just change the values of COI, ROI_1, and ROI_2 and load any chosen dataset in df = pandas.read_csv("FILENAME.csv, ...). If you want another visualization, just change the pyplot settings near the end.

The key was assigning a new DataFrame to the original DataFrame and implementing the .loc["SOMESTRING"] method. It removes all the rows in the data, EXCEPT for the one specified as a parameter.

Remember, however, to include index_col=0 when you read the file OR use some other method to set the index of the DataFrame. Without doing this, your row values will just be indexes, from 0 to MAX_INDEX.

# Written: April 4, 2019

import pandas                       # for visualizations
from matplotlib import pyplot       # for visualizations
from scipy.stats import ks_2samp    # for 2-sample Kolmogorov-Smirnov test
import os                           # for deleting CSV files

# Functions which isolates DataFrame
def removeColumns(DataFrame, typeArray, stringOfInterest):
    for i in range(0, len(typeArray)):
        if typeArray[i].find(stringOfInterest) != -1:
            continue
        else:
            DataFrame.drop(typeArray[i], axis = 1, inplace = True)

# Get the whole DataFrame
df = pandas.read_csv("ExperimentResultsCondensed.csv", index_col=0)
dfCopy = df

# Specified metrics and models for comparison
COI = "Area_under_PRC"
ROI_1 = "weka.classifiers.meta.AdaBoostM1[DecisionTable]"
ROI_2 = "weka.classifiers.meta.AdaBoostM1[DecisionStump]"

# Lists of header and row in dataFrame
#  `rows` may act strangely
headers = list(df.dtypes.index)
rows = list(df.index)

# remove irrelevant rows
df1 = dfCopy.loc[ROI_1]
df2 = dfCopy.loc[ROI_2]

# remove irrelevant columns
removeColumns(df1, headers, COI)
removeColumns(df2, headers, COI)

# Make CSV files
df1.to_csv(str(ROI_1 + "-" + COI + ".csv"), index=False)
df2.to_csv(str(ROI_2 + "-" + COI) + ".csv", index=False)

results = pandas.DataFrame()
# Read CSV files
# The CSV files can be of any netric/measure, F-measure is used as an example
results[ROI_1] = pandas.read_csv(str(ROI_1 + "-" + COI + ".csv"), header=None).values[:, 0]
results[ROI_2] = pandas.read_csv(str(ROI_2 + "-" + COI + ".csv"), header=None).values[:, 0]

# Kolmogorov-Smirnov test since we have Non-Gaussian, independent, distinctive variance datasets
# Test configurations
value, pvalue = ks_2samp(results[ROI_1], results[ROI_2])
# Corresponding confidence level: 95%
alpha = 0.05

# Output the results
print('\n')
print('\033[1m' + '>>>TEST STATISTIC: ')
print(value)
print(">>>P-VALUE: ")
print(pvalue)
if pvalue > alpha:
    print('\t>>Samples are likely drawn from the same distributions (fail to reject H0 - NOT SIGNIFICANT)')
else:
    print('\t>>Samples are likely drawn from different distributions (reject H0 - SIGNIFICANT)')

# Plot files
df1.plot.density()
pyplot.xlabel(str(COI + " Values"))
pyplot.ylabel(str("Density"))
pyplot.title(str(COI + " Density Distribution of " + ROI_1))
pyplot.show()

df2.plot.density()
pyplot.xlabel(str(COI + " Values"))
pyplot.ylabel(str("Density"))
pyplot.title(str(COI + " Density Distribution of " + ROI_2))
pyplot.show()

# Delete Files
os.remove(str(ROI_1 + "-" + COI + ".csv"))
os.remove(str(ROI_2 + "-" + COI + ".csv"))


Tokenize text with both American and English words

Grouping related tokens is called text normalization.

There is not an established Python package that does this. You could create a custom dictionary or write a function to rewrite the tokens.


deriving the gradient of batch normalization

Your derivative computation is correct, so I think your understanding of what BN does is slightly off. If you want a more thorough proof that your computation graph is correct, you can backpropagate from $\bar{x} = x-\mu$ using the partial derivatives with respect to each input in the batch, i.e. $\frac{\partial\bar{x}_i}{\partial x_j}$ from $\bar{x}_i = x_i - \mu$. There are two cases to calculate: when $i=j$:

$$
\frac{\partial\bar{x}_i}{\partial x_i} = \frac{\partial x_i}{\partial x_i} - \frac{\partial\mu}{\partial x_i} = 1 - \frac{1}{B}
$$

And when $i \neq j$:

$$
\frac{\partial\bar{x}_i}{\partial x_j} = \frac{\partial x_i}{\partial x_j} - \frac{\partial\mu}{\partial x_j} = -\frac{1}{B}
$$

of which there are $B-1$ cases. Therefore:

$$\begin{align}
\frac{\partial\bar{x}}{\partial x} &= \sum_i^B \sum_j^B \frac{\partial\bar{x}_i}{\partial x_j} \\
 &= \sum_i^B 1 - \frac{1}{B} - \frac{B-1}{B} \\
 &= \sum_i^B 0 \\
\frac{\partial\bar{x}}{\partial x} &= 0
\end{align}$$

Note also that $\frac{\partial\mu}{x}=1$, as you also correctly pointed out. What this means is that $\frac{\partial\text{BN}}{\partial x}$ is entirely dependent on the demeaned inputs ($\bar{x}=x-\mu$) and not the original inputs themselves; more importantly, backpropagation through a BN layer is invariant to the scale of the inputs. This directly results in some of the observed benefits of using BN, such as a smaller dependence on the initialization scheme and easier/faster training.


Does L1 regularization always generate a sparse solution?

I think with your red cross you have in mind the case where $w_1 = w_2$. You are right that in this scenario the solution will not be sparse, since you will also get $w^*_1 = w^*_2$.

However, whenever the coordinates of the red cross are off the line $y=x$ ($w_1\neq w_2$), you can shrink the regularization region so much that eventually you will end up with a similar picture as Figure 3.4 above.

Say your red cross has the coordinates $w_1=r_1$ and $w_2=r_2$ and say your regularization strength is such that $||w||_1 = |w_1|+|w_2| \leq C$ where $C$ defines the size of the shaded region.

Then, for the circular contours, the optimal solution will be where the line $w^*_2 = C- w^*_1$ (for the boundary region) and the line through the red cross at $(r_1,r_2)$ running orthogonal to the boundary region intersect. You can see that the latter has the form $w_2 = f(w_1) = 1\cdot w_1 + r_2 - r_1$. (Orthogonality gives you slope +1 because the boundary region has slope -1.) Setting them equal gives you the intersection at coordinates $w^*_1 = \frac{C+r_1-r_2}{2}$ and $w^*_2 = \frac{C-r_1+r_2}{2}$.

Now, if $r_2=r_1$ you have $w^*_1=w^*_2$ independent of $C$ and the solution won't be sparse.

If $r_1>r_2$ (i.e. you are below the first diagonal) shrinking $C$ to $r_1-r_2>0$ yields $w^*_2=0$ but $w^*_1>0$. You end up in the right corner of the regularization region. (Similar to my image.)

If $r_2>r_1$ (i.e. you are above the first diagonal) shrinking $C$ to $r_2-r_1>0$ yields $w^*_1=0$ but $w^*_2>0$. You end up in the top corner of the regularization region. (Similar to your image.)

As Emre mentioned in the comments, this holds in general and you can "see" this from the Kuhn-Tucker conditions of the optimization problem.


Efficiently Sending Two Series to a Function For Strings with an application to String Matching (Dice Coefficient)

Your function does a lot of pythonic data crunching. In these cases numba can be useful.
In the below code I split your function into two: sorting and scoring. I then converted your bigrams from strings to integers (to comply with numba datatypes) and decorated the scoring subfunction with numba's @autojit.
from numba import autojit
import numpy as np

def dice_coefficient(a,b):
    try:
        if not len(a) or not len(b): return 0.0
    except:
        return 0.0
    if a == b: return 1.0
    if len(a) == 1 or len(b) == 1: return 0.0
    a_bigram_list = [a[i:i+2] for i in range(len(a)-1)]
    b_bigram_list = [b[i:i+2] for i in range(len(b)-1)]

    a_bigram_list.sort()
    b_bigram_list.sort()

    lena = len(a_bigram_list)
    lenb = len(b_bigram_list)
    matches = i = j = 0
    while (i < lena and j < lenb):
        if a_bigram_list[i] == b_bigram_list[j]:
            matches += 2
            i += 1
            j += 1
        elif a_bigram_list[i] < b_bigram_list[j]:
            i += 1
        else:
            j += 1
    score = float(matches)/float(lena + lenb)
    return score

def dice_coefficient_new(a,b):
    try:
        if not len(a) or not len(b): return 0.0
    except:
        return 0.0
    if a == b: return 1.0
    if len(a) == 1 or len(b) == 1: return 0.0

    a_bigram_list, b_bigram_list = dice_coefficient_sorting(a,b)
    score = dice_coefficient_scoring(a_bigram_list,b_bigram_list)

    return score

def dice_coefficient_sorting(a,b):

    a = np.array([ord(i) for i in a])
    b = np.array([ord(i) for i in b])

    a_bigram_list = 256*a[:-1]+a[1:]
    b_bigram_list = 256*b[:-1]+b[1:]

    a_bigram_list.sort()
    b_bigram_list.sort()

    return a_bigram_list,b_bigram_list

@autojit(nopython=True)
def dice_coefficient_scoring(a_bigram_list,b_bigram_list):

    lena = len(a_bigram_list)
    lenb = len(b_bigram_list)
    matches = i = j = 0
    while (i < lena and j < lenb):
        if a_bigram_list[i] == b_bigram_list[j]:
            matches += 2
            i += 1
            j += 1
        elif a_bigram_list[i] < b_bigram_list[j]:
            i += 1
        else:
            j += 1
    score = float(matches)/float(lena + lenb)

    return score

Let's time it:
N = np.power(10,5)

a = ''.join([str(unichr(i)) for i in np.random.randint(97,123,N)])
b = ''.join([str(unichr(i)) for i in np.random.randint(97,123,N)])

%timeit dice_coefficient(a,b)
%timeit dice_coefficient_new(a,b)

Output:
1 loop, best of 3: 204 ms per loop
10 loops, best of 3: 52.9 ms per loop

So for 100K elements you get a speedup of 4x!
For further optimisation you could parallelise your global for loop (for example also using numba or multiprocessing).
(Note: I edited my rushed first answer which didn't work)


What is "SwarmPackagePy.cso.cso at 0x187cf21e340"

This is not really a "data science" question but more of a "(Python) programming" question. You can find more comprehensive answers at stackoverflow.
Other than that, if you go to the source code of SwarmPackagePy.cso (I was not able to find any documentation), you will see that it creates an instance (object) of the cso class and what you get when you print this object is the location or address of this object in the memory.