Method for predicting winner of call for tenders

Having the classifier try to predict one of 2000 possible values is going to be tough. A common approach is to either bucket the possible targets/labels or decompose the problem. For example:

Instead of trying to predict the exact winning company, group the winners into similar groups and predict the group. For example, predict if the winner will be a large public company, small/medium business, or an independent consultant.
The nature of procurement probably varies across the buyer and what is being bought. A model that is good at predicting which company will win the business of a Fortune 500 company probably has a different structure than who is good at winning the business of a small city. In a similar fashion, the competition for who is going to build a bridge is going to be different and not include the companies who will bid on implementing a new website. Partition your data into similar competitions and then try to predict the results. This will hopefully have the side effect of reducing the number of potential targets.


The probability that a vector $x$ is drawn from $p(x)$ in some region $R$ of a sample space is given by $ P = \int_{R} p(x')dx'$.  Given a set of N vectors drawn from the distribution; it should be obvious that the probability k of these N vectors fall in $R$ is given by $P(k) = \binom{N}{k} p^{k} (1-p)^{N-k}$.  From the properties of a binomial p.m.f the mean and variance of the ratio $\frac{k}{N}$ are ${E}[\frac{k}{N}] = P$ and ${var}[\frac{k}{N}] = \frac{P(1-P)}{N}$.  Therefore, as $N  \rightarrow \infty$ the distribution becomes more defined and the variance smaller.  Hence, we can expect a decent estimate of the probability P to be obtained from the mean fraction of points that fall within the region $R$.  Hence $P \cong \frac{k}{N}$,

Now consider if the region $R$ is small such that $p(x)$ does not vary considerably within it, then $\int_{R} p(x')dx' \cong p(x)V $.  Combining this result with the one above.  We see that $p(x) \cong \frac{k}{NV}$.

That's where the formula you found basically comes from.  Therefore if we want to improve $p(x)$ we should let V approach 0.  However, then $R$ would become so small that we would find no examples.  Thus we really only have two choices in practice.  We have to let V be large enough to find examples in $R$ or small enough such that p(x) is constant within $R$.

The basic approaches include using KDE (parzen window) or kNN.  The KDE fixes V while kNN fixes k.  Either way, it can be shown that both methods converge to the true probability density as N increases providing that V shrinks with N and that k grows with N.

The formulas used in the picture are just arbitrary examples that fulfill this requirement.


Effect of removing duplicates on Random Forest Regression

I agree with the previous answer about your specific question: your model is likely to be less accurate if you just remove training data.
Moreover:

With very large sample counts some models (like Random Forests) become very large (several GB when pickled).

It sounds like the size of your final model is the reason why you're considering pruning your dataset. Assuming you are using scikit-learn, I'd recommend looking at the documentation and the parameters that you can tweak that impact your final model size.
Specifically, parameters like n_estimators, max_depth, min_samples_split, and min_samples_leaf all have default values that tend to generate large models. For example, max_depth is set to None and therefore (as per the docs) "nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples." If you experiment with these you are likely to be able to generate much smaller models, in terms of size-on-disk, without needing to prune your dataset.


(CNN+)RNN-HMM hybrid for learning phonemes from a spectogram

Have you seen this...

http://ieeexplore.ieee.org/document/7953168/

  ... We propose to use a recently developed deep learning model, recurrent convolutional neural network (RCNN), for speech processing, which inherits some merits of recurrent neural network (RNN) and convolutional neural network (CNN). The core module can be viewed as a convolutional layer embedded with an RNN, which enables the model to capture both temporal and frequency dependance in the spectrogram of the speech in an efficient way.

Sound pretty accurately to what you are searching for, or trying to implement.


Best way to represent data as features vectors in Python

Numpy is best.  You can just take

np.array(dict.values()).

Eg,  dict = Paris_attck

Most ML libraries operate on numpy arrays.


Disparity between training and testing errors with deep learning: the bias-variance tradeoff and model selection

Your question is, what model is better between one that seems more overfitted (larger difference between train and eval set) but it has also higher scores or one that has less variance between train and eval set but at the same time it has worst results. Everything assuming that you have done a correct train test split and there is no data leakage and distributions keep the same in every split (this is important to check).

There was a discussion about this some time ago. The answer seems to be kind of subjective since the quantitative analysis has been performed

Normally there is the following trade-offs:

Complexity: Occams razor and complexity vs interpretability. In your case, both models are almost with the same complexity (it's not a linear regression against DL, just a couple layers more) and the interpretability stays the same,
Generalization. You want your model to behave in the best possible way in production, an overfitted model in train seems to have more probable cause to fail due to a change of distribution in production.

You only have 3 data points so it's hard to say what it will be best. My suggestions will be that:

Add some more layers (6,7,8) just to see when your test results start to go down (you can still overfit much more) and then visualize the data and keeping both concepts defined before choosing what are the best architectures for your model
Investigate with more parameters (adding one more layer seems to be a high difference hyperpameter), like learning rate, layer size, activations functions and so on...
Consider using one of the famous architectures for your problem, they are developed in every framework and tested by a lot of people, they are there because the seem to be the best at their task, give them ago. There has been already a lot of electricity wasted in deep learning hyperparameter tunning.


You could trivially prototype your solution in Orange (it's a Python library that uses scikit-learn under the hood). It also has a neat GUI.

But with only two features (state and renting_reason), I'm afraid your models might not be as precise as you'd want them to be, unless you have lots of examples to learn from.


Why PyTorch is faster than sklearn models?

It is difficult to answer your question without the access to your code. The best way to understand the difference is to profile the code and see where the bottlenecks are for your specific problem.
For this, you can use different profiling modules in python:

cProfile
python line profiler


POC - Get an idea to create a Predictive Model

Recommender systems are usually either collaborative ("other people who bought x also bought y") or content-based ("item y has similar properties as item x"). Given your very few data features, collaborative filtering seems appropriate. In the basic setting it could use only the purchases of your users. Ideally, you would also have some sort of rating of their purchases.

Also, nothing stops you from using a hybrid approach with additional data if available.


Terminology - cross-validation, testing and validation set for classification task

Confusion 1)

From wikipedia :

  k-fold cross validation is a technique for assessing how the results
  of a statistical analysis will generalize to an independent data set

They also say :

  In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times, with each of the k subsamples used exactly once as the validation data. The k results can then be averaged to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used,[7] but in general k remains an unfixed parameter.

So you now see that you don't have k% for the testing but you always use 1/k % of the your dataset as test set. Note : you can choose to keep 2/k or more but it will be a lot more complicated to code.

Confusion 2)

In the scikit learn they above all refer to this tool as an "evaluating performance tool" counter to what wikipedia authors may suggest. The point is that CV allows you to assess how robust and reliable your prediction will be according to your initial dataset.

The final mean obtained at the end of the CV is a mean score on the k testing sets. It is often good to have a look at all the intermediary results to assess the variances of them that can be a good explanatory estimation in case of bad generalization capability of your model.

Edit : why running another training after CV

Cross validation can also be used as an optimization tool to find the best hyper-parameters of your model. In this case, you should take the better hyper-parameters (among the k different parameters; one for each fold) and use them to do prediction on the full set to see if the optimized (i.e. choosen from CV) hyper-parameters are good on the full dataset. The notion of "best" parameter can be seen as the hyper-parameters from the model who gave the best score in your CV process.

Note you can still put aside a validation set from the dataset, on which you won't do CV. This validation set can be used as the last test of your model prediction's quality. See also here

Finally, you can use each k-fold model to predict an estimation and then take the mean of them as the final prediction of your model as Wikipedia authors have suggested, but this idea is closer to ensemble learning or a kind ofBootstrap method without replacement, than to CV

hope it helps


How many people can use a single Hadoop cluster at one time?

For initial learning you could very easily do proof of concept work against individual VMs (a 4GB VM with a pseudo cluster is enough to do basic mapreduce examples in). If you're going to use Spark I would lean towards higher memory counts if they're within budget. I'd also keep an eye towards more, lower cost nodes. A stack of Intel i7 NUCs (or similar) with 2TB consumer SSDs and 32GB of RAM will cost less than $1k per node and 10-15 of them could easily handle a class of 24 students.


Package/function to convert the word to one hot vector in Python NLP

There are several ways to convert words to one hot encoded vectors. Since I do not know the data structure you store your data. I assume it is going to be a list

  from keras.preprocessing.text import Tokenizer
  samples = ['The', 'dog','mouse','elephant']
  tokenizer = Tokenizer(num_words=len(samples))

This builds the word index

  tokenizer.fit_on_texts(samples)

one hot representation

  one_hot_results = tokenizer.texts_to_matrix(samples, mode='binary')

By changing the mode from binary to 'tfidf' or 'count', you can make a matrix of any type, apart from one hot.

You can achieve the same result using other packages like sklearn. But it does involve a bit more lines of code.


is there any rule to apply pca to the imbalance data?

PCA is method for unsupervised learning so you shouldn't add the label.

Imbalanced data refers to the label.

So for me it does not make too much sense to talk about PCA and imbalanced data.


Which tribe does Probabilistic Graphical Models fall under?

Probabilistic Graphical Models (PGMs) are:

Connectionist: RBMs are PGMs and neural networks (source)
Bayesian: Bayes Networks are bayesian (Wikipedia article)
Symbolist: Markov Logic Networks (source)
Analogizers and Evolutionaries: According to Domingos, they are also in Markov Logic Networks.

So the answer is that you can't simply categorize such a general technique as probabilistic graphical models in a single one of those categories.
See also: "The Five Tribes of Machine Learning (And What You Can Learn from Each)," Pedro Domingos


How does one decide when to use boosting over bagging algorithm?

Bagging and boosting are two methods of implementing ensemble models.

Bagging: each model is given the same inputs as every other and they all produce a model

Boosting: the first model trains on the training data and then checks which observations it struggled most with, it passes this info to the next algorithm which assigns greater weight to the misclassified data

Because of this both bagging and boosting reduce variance. However boosting is better at improving accuracy vs a single model whilst bagging is better at reducing overfitting.

I would advise training a single version of the ensemble aka decision tree for random forest and then seeing where improvements can be made.
Good article explaining boosting vs bagging


What will the required time to process 500GB of images using NVIDIA GEFORCE 930M GPU

The large size of your data is acceptable for deep learning and big data projects. Your system is also acceptable, though it is not powerful. If you have enough hard disk to store them all, it will suffice which means you can train your network. The elapsed time for each epoch depends on multiple aspects. For instance, some elements which are important are, the batch size and your vectorized implementation, the bottle-neck between the disk and RAM, the bottle between RAM and GPU, the size of the model, the size of training data, the memory size of your GPU alongside the size of your RAM, the size of each data, the load which is imposed to your GPU by your OS, and so forth. The easiest way is to code your network and try it yourself.

As I've mentioned, by the current settings you can train your network, but you may not have very fast computation. However, you can use some techniques to faciliate your training phase as much as possible. For instance, you have two main bottle-necks. The first bottleneck, which exists between disk and RAM, can be dealt with using generators. Namely, you can employ generators to decrease the number of disk calls. The other bottle-neck, between RAM and GPU, can be handled using vectorized implementation of your neural network. After loading your network, you can find the appropriate batch size to use all available GPU memory.

I also want to point out that the current GPU you have may have space limitations. This can incur difficulties when your network is very large. In such cases, you won't be able to load your entire network to your GPU.


Which solutions are there for RL agents when not all actions are always available?

You could use an out-of-the-box learning algorithm if the availability of actions is deterministic and depends only on the state the agent is in, i.e. if availability of actions does not violate the Markovian assumption.
You need only substitute the full action space with the 'available action space' in any place where the action space shows up.
As an example, consider the update rule in Q learning, then replace the Q(S,A) with Q(S, A') where S' indicates the available actions for that particular S.
Please note that the referenced paper targets a setting in which availability of actions is not Markovian.


Does fine-tuning of transferred layers perform better than frozen transferred layers?

Transfer learning means to apply the knowledge that some machine learning model holds (represented by its learned parameters) to a new (but in some way related) task. Whereas fine-tuning means taking some machine learning model that has already learned something before (i.e. been trained on some data) and then training that model (i.e. training it some more, possibly on different data).

From this we can conclude that if we are to use the learning of one model and concentrate only on some specific part of it, we can use transfer learning and train the network again with only a little amount of data. Whereas if we have sufficient data that we want the model to be trained upon, fine-tuning the model will come to the rescue. Refer to the previous question and this article


Knowing Joint probability distribution between feature-label space

The interest is to predict $y$.

If we know the real distribution $P$ of $(x,y)$, there is no need to build any more machine learning model. Given $x$, we can directly consult $P$ to know the probability of $P(y|x)$. For the case of discrete $Y$, we have $P(y|x)=\frac{P(x,y)}{\sum_z P(x,z)}$.


Cosine Similarity for Ratings Recommendations? Why use it?

Rating bias and scale can easily be accounted for by standardization. The point of using  Euclidean similarity metrics in vector space co-embeddings is that it reduces the recommendation problem to one of finding the nearest neighbors, which can be done efficiently both exactly and approximately. What you don't want to do in real-life settings is to have to compare every item/user pair and sort them according to some expensive metric. That just doesn't scale.

One trick is to use an approximation to cull the herd to a managable size of tentative recommendations, then to run your expensive ranking on top of that.

edit: Microsoft Research is presenting a paper that covers this very topic at RecSys right now: Speeding Up the Xbox Recommender System Using a
Euclidean Transformation for Inner-Product Spaces


Faster R-CNN: Labels regarding the positive anchors when there are many classes

As well as the algorithm used for region proposals in R-CNN and in Fast R-CNN, RPN task in Faster R-CNN is to propose regions that may or may not be an object, so its
cls head with 2k scores corresponds to the classification of every anchor as an object or as a non object (this way all of them may be non objects):

But it could also be a k scores logistic regression (k sigmoids 0 to 1) as they stated in the fourth note of the paper:

  For simplicity we implement the cls layer as a two-class
  softmax layer. Alternatively, one may use logistic regression to
  produce k scores

After proposal generation, Fast R-CNN is used to classify and improve every selected proposal, i.e. grey area in the following image:

So you may better conceptualize the difference between Fast and Faster R-CNN considering the region proposal generation algorithm as a black box, the one could be filled with a RPN head connected to the output convolutional feature map of the deep net used as backbone, or the standard Selective Search used by R-CNN and Fast R-CNN


In ML you really need good examples and then things for which you don't know the outcome. You learn from the good examples and then apply this "knowledge" to the examples for which you wish to know the outcome.
I agree that Mathematical Optimisation would probably be a better route to take in a problem such as this.
Alternatively, if you want to imply some kind of connection between sets, you could create a categorical (dummy) variable that designates such. If I understand you correctly here is an example
         item1    item2   basket1  basket2  basket3  GroundTruth (target)
order 1  Banana   Apple   True     False    False    warehouse1
order 2  Toy      Carrot  False    True     False    warehouse2
order 3  Picture  Shoe    False    False    True     warehouse3

You could also include the items in the warehouses in a similar way though if you have lots of different items, lots of warehouses, lots of baskets and not relatively enough training example this is going to get pretty sparse pretty quickly.


A math question about solving the Lagrangian of Support Vector Machine

Since you have an inequality constraint you need to meet the necessary Kuhn-Tucker Conditions which for non-negativity constraints are:

$\xi \geq 0$
$-r \cdot \xi = 0$

It is not enough to set the derivative w.r.t. to $r$ to zero as is the case for equality constraints and Lagrange multipliers.

Therefore, there are two cases to distinguish:

Either the condition is tight ($\xi=0$) and you can have $r \neq 0$
Or $\xi > 0$ and $r = 0$.


Is it possible to know the output vectors of MLP Classifier of scikit learn?

If you are interested on the probability output of your model, simply call mlp.predict_proba(X)


Understanding cosine distance with word vectors

Word2Vec algorithm does not go inside words. Word “king” is never used as a gerund, so there is no reason why it should be similar to gerunds.

My guesses are:

Your corpus might by wrongly tokenized. Maybe there are some OCR-related errors with word splitting something like “li-↲ king”.
You might be using a different algorithm for getting the embeddings (e.g., FastText) that goes inside the words and infers the word embedding as some embeddings of character n-grams the word consists of.

On the other hand, words similar to man look fine. If you think about how Word2Vec (and also FastText) is trained, you should not ask a question: ”Do the words have as similar meaning as possible?” but rather “Does the word appear similarly frequently in a similar context Shakespear's works?”

(Of course, when the embeddings are trained on data which is large enough, there is almost no difference between these two questions.)


How to find whether a dataset is blanced or imbalanced?

Typically, the representation of each class in a multi-classification problem should be equal. Say if there are 4 classes, then the ratio of count of samples in each class should ideally be n:n:n:n, most classification data sets do not have exactly same number of sample count in each class, which is fine and a lit bit of difference often does not matter. But if the difference is huge, say for example 100:5:9:13 then it matters and it is an imbalanced dataset.

coming to 400 GB of data to read - Depending on the type of your file, you can read it in chunks and then read and save the target variable( the one which has multi class labels) in another variable.

You can visualize this variable (containing target) using a bar chart which will show you the count of variables for each class. Along with that you can also calculate the distribution of your classes to get better understanding of data.


We are using PostgreSQL to store big data and are concerned it may crash the on-board Neural Net. Any thoughts?

Yes, Orange does not handle very large data sets very well. In fact, if you try to load in something like the file size that you are talking about (gigabytes of data) it will crash. When ever you want to create a large scale neural network, you will need to use packages like Tensorflow and Keras. You can not do this in Orange.


How to use CNN to deal with a 2D regression problem?

It should even be easier than classification: you do not need the final layer.

Your input layer should have 18 nodes $(x_i,y_i,t_i)|i=1..6$

Hidden layers as you see fit (experiment with it, depending on data and results).
I would expect the best results from a fully connected layer and a 'relu' layer.

The output layer has 3 nodes $(x_7, y_7, t_7)$

Train the model.

Evaluation is just feed forward of a set of 18 values, and retrieving 3 results from the output layer nodes.


Classification method when idea conditions are known?

Welcome to the community Martan!

If I understood your question well, you have a set of patterns (you call them idea condition) and a set of query points (samples) and you want to determine (predict) their labels according to their affinity/similarity/closeness to the pattern.

If it's right, then K-Nearest Neighbor algorithm is what you are looking for. Please note that in high dimensions Euclidean distance is distorted however 8 dimensions is fine.

Hope it helps and in case I did not understand well, please comment here so I can update my answer. Good Luck!

Update

What you mentioned in your comment about training in KNN is right. Let me clarify the thing.

Classification as A Supervised Learning Process: This means that you already had some data and their classes. So you can partition your space according to all those labels. Image below shows partition of different classes (i am not the best painter specially in MS paint :D). Having this, a new point falls into one of these partitions so you can determine its label. Building such a partitioning is done in the training process as you said.

Nearest Neighbor Search: But isn't this method pretty natural? You really do not need to know Machine Learning to perform such an algorithm and indeed you use KNN in daily life (today you see your friend with some of his colleagues and a new person you dont know and you instantly think he is probably a colleague. The other day you see your friend with his family and a guy you dont know and you guess he is most probably a relative. That is conceptually KNN!). In your example you dont need to learn how to partition the space as you already have your predefined labels with fixed position (we do training to find this out. you already have it, so just go on!). Now you can just make a nearest neighbor search and say to which class a point belongs.


even though there are some learning rate decay schemes for Standard Gradient descent, convergence of these methods are not justified beyond convex functions or functions whose gradients are globally Lipschitz. Even for those latter maps, when you do not have a good lower bound for the Lipschitz constant L, basically you have no way to say which learning rate is good.

In my joint paper, we proposed a method to modify learning rate using the backtracking version of Gradient Descent, which works quite stably and effectively. More details about the theoretical background and the accompanying source codes can be found in my answer in this link:

Does gradient descent always converge to an optimum?


Is there any consensus on choosing an appropriate ML approach?

Well, let's say in this way. Although there are numerous learning approaches, each is useful for a particular situation. It is possible that for a problem you have multiple choices. Each of learning approaches has a special application domain and that is why people usually know where to use decision trees and where to choose neural networks, e.g. in situations that all of your inputs are real-valued numbers, attempting to use decision trees is not a wise choice. I try to explain the main things that an ML practitioner usually considers.

Number of Available Features

The number of features is important due to not being able to visualise them easily in cases there are so many features. This may lead to not being able to recognize whether the data is linearly separable or not. So many features do not imply that the dataset is not linearly separable. Consequently, say you want to use neural nets for modelling the problem. You shouldn't begin with a complicated network with so many layers and neurons. You have to begin with a single neuron, equivalent to logistic regression for classification tasks, to figure out whether your data is linearly separable or not. If you observe that you don't have good performance, you can add extra neurons and layers. How? Take a look at How to set the number of neurons and layers in neural networks.

Feature Space

About choosing ML approaches, it is simple to consider the limitations of each algorithm with respect to the feature space. E.g. decision trees are not very good for problems with many features which some of them are numerical features. They may get really big. SVMs are not very good for non-linear problems with so many features due to the fact that you have to specify the kernel size. For different regions in the feature space, a single kernel size may not be valuable. To generalize, problems that have very large input space are usually handled using neural networks. If the problem has so many features but they are e.g. binary features or categorical features with a small number of choices, then the problem has smaller feature space, input space, and other ML approaches can be considered.

Size of Dataset

Depending on your problem, feature types and feature ranges, the input space may be very small or very big. Consequently, the number of possible data may differ depending upon the input space. For large input space, as mentioned, neural nets are very powerful for mappings.

Distributions and Bayes Error

For different tasks, e.g. classification, you have to do statistical analysis for knowing your available dataset better. You have to investigate if there are input patterns that there are same but their labels differ. If so, why? Is that for expert error or not. The current feature space is not valid for understanding the problem. After addressing these questions you can employ Bayes error to investigate the best approach will have what accuracy on your data.


Training with data of different shapes. Is padding an alternative?

Whether or not padding is ppropriate really depends on the entire structure of your dataset, how relevant the different variables/columns are and also the type of model you want to run at the end.

Padding would be used, whereby you would have to fix the length of each sample (either to the length of the longest sample, or to a fixed length - longer samples would be trimmed or filtered somehow to fit into that length). Variables that are strings can be padded with  empty strings, variables with number can be padded with zeros. There are however many other ways to pad, e.g. using the average of a numerical variable, or even model-based padding, padding with values that "make most sense" for filling gaps in that specific sample. Getting deep into it like that might more generally be called imputation, instead of padding - it is common in time series data, where gaps aren't always at one end of a sample.

Below I outline one approach to padding or standardizing the length of each sample. It it not specifically padding.

As you did not mention a programming language, I will give and code snippet in Python, but the same is easily achievable in other languages such as R and Julia.

The Approach

Based on the two examples you provide, it seems each example would be a calendar day, on which there are a variable number of observations.

There are also columns that are strings, and others are strings of numbers (e.g. column 5 in sample 2).

In time-series analysis in general, it is desirable to have a continuous frequency of data. That means have one day give one input. So my approach would be to make your data into a form that resembles a single input for each of the variables (i.e. columns) of each sample.

This is general approach, and you will have to try things out or do more research on how this would look in reality for your specific data at large.

Timestamps

I would use these as a kind of index, like the index in a Pandas DataFrame. One row = one timestamp. Multiple variables are then different columns.

Dealing with strings

I will assume that your dataset has a finite number of possible strings in each column. For example, that column 4 (holding names), will always hold a name from a given set. One could perform set(table2['column 4']) to see which values there are (removing duplicates). Or even:

# Gather a single list containing all strings in column 4 of all samples
all_names = []
[]    # list comprehension to loop

# Check how many strings there are
from collections import Counter
counter = Counter(table2['column4'])
print(len(counter))                 # see how many unique values exist
print(counter)                      # see how many times each string appears
print(counter.most_common(5))       # see the most common 5 strings

Now assuming this shows there is a finite number (more than likely the case), You could look into using a sparse representation of each sample (that means for each day). For example, if all the words in the entire dataset were: ['hi', 'hello', 'wasup', 'yo', 'bonjour'] (duplicates removed), then for one single sample with column 4 holding e.g. ['hi', 'hello', 'yo', 'hi'], your sparse representation for this sample would be: [2, 1, 0, 1, 0], because the sample has two 'hi', one 'hello', zero 'wasup' and so on. This sparse representation would then be your single input for column 4 for the timestamp (that single sample).

It might be worth looking into something like the DictVectorizer and CountVectorizer from Scikit-Learn.

Dealing with number columns

As I mentioned right at the beginning, you could pad these to a chosen length, perhaps matching the length of the string based representation above (or not!), depending on your final model.
You can then pad the inputs with a value that makes sense for your model (see the kind of options I mentioned at the beginning of my answer).
This should land you with, once again, a single vector for the given day, containing the information in the numerical column.


"Apriori" algorithm is used for "Association Rules" learning.

In very simple terms its trying to determine that if people who buy chocolates ,do they buy roses also with that? or do they buy chocolate with ice-cream more? or its chocolate+roses+ice-cream always together? or any combination of it.

So, the data which contains these purchase transactions are analysed and then "frequent item sets" are determined.

From this "rules" are derived , eg. here a rule can be, chocolates are bought with roses, or roses are bought with chocolates.

Each rule has confidence and support.

Now in you dataset, do you don't seem to have transactions, rather it has attributes regarding chocolates and its rating.which doesn't suit associative learning since the objective is different.

For the "learning" purpose, if its predicting that there are NO rules, means there are NO rules to determine.

Here something to get you started-

https://en.wikipedia.org/wiki/Apriori_algorithm

https://en.wikipedia.org/wiki/Association_rule_learning


Thoughts on Feature Engineering of a duration_in_program Variable

What you are describing sounds much like a survival model, where you have observations that are still surviving (still in the program) and observations that died along the way (attrition).

You would want to look into survival methods and how to censor your data. There are basic regression methods for this where you are trying to predict time-until-event. XGBoost can also handle censored data for survival.


How to adjust cofounders in Logistic regression?

While I'm not sure what you mean when you say "adjust for confounders", I suppose your question is about model choice (or variable/feature selection).

Here are some thoughts on this problem:

Clearly define what you want to achieve: If you want to achieve a good prediction (so you are not up to causal modeling), choose a suitable metric to measure model fit. For a classification problem, you could look at the confusion matrix or AUC (but there are many more options).
Choose a baseline model which you believe is a good starting point for your problem. Look at relevant metrics of this model.
Try to come up with improvements: You could - for instance - work on your features and see if your "new" model performs better (based on the chosen metrics) compared to the baseline model. You can also add interaction terms (between variables) by simply multiplying variables. So when you have a model $y=\beta_0+\beta_1 x_1+\beta_2 x_2 + u$, you can also go for $y=\beta_0+\beta_1 x_1+\beta_2 x_2 + \beta_3 x_1 x_2  + u$ (Note: $\beta$ are the regression coefficients, $u$ is the error term).

Make sure you train your model on one part of your data and you test your model on another part of the data (train/test split).

Some notes on feature selection: Generally it is not a good idea to select features only by looking on significance. Why? Significance tells you if the estimated confidence interval of the coefficient of some variable/feature "crosses zero" (the interval has positive and negative values). This says little about the contribution of a variable to overall model performance. Variables can also be "jointly significant", so even non-significant (single) features can be important (in interplay with other features). Alternatively, a linear transformation of some feature may make it significant, e.g. in the case of adding polynomials (squared terms for instance) or making a logistic transformation. So in essence: don't kick out features only based on significance. By doing so, you have a fair chance of introducing the "omitted variable bias".

In case you have a lot of possibly relevant features and you are sick of doing model selection "by hand", you may also look into Lasso (or Ridge) regression. Under this approaches features are "shrunken" (automatically) when they are not so useful for a good prediction. Here is a very good intro to Lasso/Ridge/Elastic Net by Trevor Hastie and Junyang Qian. The code is in R, but the tutorial is very good.

You would surely gain from looking at the book "Introduction to Statistical Learning". Chapter 4 covers Logit in a very instructive way. Plus there is Python code for the Labs in the book, which could give you a good starting point. Reading your question, the book and the Lab code from the book, would be a natural statring point for you.


FeatureHasher assigns each token to a single column in the output; it does not do the sort of binary encoding that would allow you to faithfully encode more features to a smaller space.  Instead, the point is that you don't have to look up which column a given category maps to as you would in OneHotEncoder: you just have to apply the hash function to the value.  Using a lower-dimensional target space will guarantee collisions, but even a larger target space may produce a few collisions.


How does generalised advantage estimation work?

I found very intuitive the explanation of the GAE in the Supplementary material of this paper: DeepMimic. You do not need to read the paper. Just go straight to the Supplementary material section on page 143:15. For the λ-return you can find lots of information in the Reinforcement Learning book of Sutton and Barto. Hope it helps!


I don't have much hands-on experience in this, but nevertheless I think this kind of processing of streamingdata is called "online learning"; and finding optimal hyperparameters for any type of learning (online or batch) automatically such that the system is self-tuning is an area of active research.

Prof. George Hinton talks about it generally in this video "Lecture 16C : Bayesian optimization of neural network hyperparameters" in this playlist: https://www.youtube.com/playlist?list=PLiPvV5TNogxKKwvKb1RKwkq2hm7ZvpHz0
(13min)

If you like audio, a listener asked a similar question in this  Episode of the Talking Machines podcast:
(about 8 minutes in)

Clever switching of the learning algorithm also has been explored, see "Bagging" or "Boosting" on Wikipedia, or Ensemble learning

For a relatively easy-to-use Java-based GUI tool you might try "MOA"- Massive ONline Analysis


How can discard "peaks" values from a chart?

I would say you have two options to choose from:

Aggregate your data
Use a low-pass-filter

Aggregating

Aggregating your data over long time periods is not cheap (whatever that means). It is a common way of displaying trends over long periods of time.

Take a look at how Bloomberg displays the exchange rate of EUR over USD.

You can filter by time period, and if you select 1 day, you see the hourly rate:

or you can filter by month, and see the rate aggregated by day. For them, the last rate is the important one, so that is the value you see. For you, however, will depend on your use case. You can choose to pick one of the following: mean, median, mode, max, min, first, last or anything else that makes sense for you.

Signal processing filtering

If you want to take a more mathematical approach to your problem, you could use a signal filtering technique to reduce high-frequencies from your signal.

What you need is a low-pass filter.

If you are using Excel, you certainly can do this.
However, I would recommend you do that in some programing language like Matlab or Python, that contain easier to use signal processing packages.


Modern reference on general Deep Learning theory

For the latest practices of deep learning architectures, I follow the Kaggle latest Competition's notebooks.
Say, for example, In the recent finished Cassava Leaf Deasese Computer Vision Competition, People are sharing the experimental notebooks on different State of the art Architectures Like Vision Transformer(Various versions pretrained  - vit_large_patch16_384, vit_base_patch16_384 etc. models), Facebook's Data-efficient Image Transformers , EfficientNet (Noisy Student version, Imagenet Version) and etc.
 https://www.deeplearningbook.org/ this book undoubtedly one of the great book. But for the latest practices if you follow the Kaggle notebooks you get a good explanation with codes.
I suggest you,  Explore this Notebooks Section of the Casava Leaf Disease Classification Contest to get more Models and Various recent new released Loss Functions and so on.


Many separation line using RBF kernel in SVM

The OneClassSVM is an unsupervised algorithm that is supposed to learn the normal data distribution. This means that the algorithm will model the boundaries of areas of high likelihood for your data points to be drawn from.

Even though you know which points are outliers, the SVM will only know that there is roughly a proportion of outliers of nu, therefore the 14000 boundary is not intuitive at all in terms of distribution.

What the SVM will do that explains your multiple lines is:

If you have a hole in what you consider the "normal zone" - let's say as an example you have no data points between 2560 and 2985 - the SVM can decide that this is an area of low likelihood for your distribution and therefore build two vectors to exclude it from the learned normal distribution.
At the opposite, if you have several points clustered together above 14000, since the SVM has no clue this is supposed to be the anomalous zone, it can detect a zone of high likelihood for the data points to be drawn from, and build two vectors to include it in the learned normal distribution.

Now regarding your metrics:

You probably have an imbalanced dataset since we are talking about anomaly detection (otherwise it makes no sense!), so you should not use global accuracy here because it is misleading.
There is one thing that should alert you: The train AUC is very low, while the test AUC very high and this is not normal. So your algorithm poorly learns the target normal distribution.

The high test AUC can be explained, but first let's use some notations:

The real normal data distribution will be called *the target normal distribution", in the sense that you want to learn it. It is the interval [0;14000[
The real anomalous area (cannot talk about a distribution in this setting) will be called "the target anomaly area". It is the interval [14000;+infinity[
The learned normal distribution will be called the "learned normal distribution", obviously.
The learned anomalous area will be called the "learned anomalous area"

So the learned distribution consists in small clusters around areas of higher training data density. Since you (supposedly) have a few outliers and a lot of normal points, the learned distribution still intersects well with the target normal distribution, but not with the target anomalous area, where only a few areas will belong to the learned normal distribution. You can verify this on your graph: there are only a few vectors in the target anomalous area.

Now when you apply the algorithm on the test set, it is not likely that a point from the test set in the target anomalous area falls into the intersection of the learned normal distribution and the target anomalous area, because this intersection is very small. But this is only due to the small number of data points in this anomalous area, not to a successful learning process. Hence these points will still be labeled as anomalies and you get a misleading high AUC score.

At the contrary, the target normal distribution and the learned normal distribution intersects correctly, because there was more data points to learn from. So most of the normal points in the test set are classified correctly.

To sum up, your algorithm learns a completely different distribution from what you excpected. You can try to use a higher gamma value to smooth the kernels. but you will probably not avoid the classification of the areas of higher density above 14000 as normal data points. So watch out for the train AUC!

It would be very interesting to investigate these statements with graphs, you could as an example plot the [14000; +infinity[ zone, highlight the decision function boundaries and plot train and test set data points. You should get confirmation of what I am saying above (I will do it later and add the graphs if I find some time!)


Dimensionality reduction with PCA limitations

You should not use PCA if you only have categorical variables, and thus the distance function in PCA is invalid.

Correspondence analysis is a common alternative.


What is the best way to compare various classification models?

Since the problem you are trying to tacle is image classification, then classification accuracy is the appropriate measure of comparison. Also you could consider the Precision, Recall and F1 metrics (for multi-class problems). See below there is description of an extension about these multi-class metrics.

A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems

Another appropriate metric is AUC/ROC, which has to be extended cosnidering the multi-class case. See the link below

Receiver Operating Characteristic (ROC)


Connect Orange 3.20 to postgresql database

To summarize my linked answer. While you may be able to connect Orange to Postgresql, Orange does not handle large datasets very well (it will cause it to crash). Generally, it does not support datasets the size that you would be storing in SQL sized databases. Personally, I think Orange works, and is most effective, with smaller sized datasets for illustrative tasks and exploratory data analysis.


What is the advantage of keeping batch size a power of 2?

This is a problem of alignment of the virtual processors (VP) onto the physical processors (PP) of the GPU. Since the number of PP is often a power of 2, using a number of VP different from a power of 2 leads to poor performance.
You can see the mapping of the VP onto the PP as a pile of slices of size the number of PP.
Say you've got 16 PP.
You can map 16 VP on them : 1 VP is mapped onto 1 PP.
You can map 32 VP on them : 2 slices of 16 VP, 1 PP will be responsible for 2 VP.
Etc.
During execution, each PP will execute the job of the 1st VP he is responsible for, then the job of the 2nd VP etc.
If you use 17 VP, each PP will execute the job of their 1st PP, then 1 PP will execute the job of the 17th AND the other ones will do nothing (precised below).
This is due to the SIMD paradigm (called vector in the 70s) used by GPUs. This is often called Data Parallelism : all the PP do the same thing at the same time but on different data. See here.
More precisely, in the example with 17 VP, once the job of the 1st slice done (by all the PPs doing the job of their 1st VP), all the PP will do the same job (2nd VP), but only one has some data to work on.
Nothing to do with learning. This is only programming stuff.


Bayesian network in Python: both construction and sampling

Please use the function from_samples() to build a Bayesian n/w from the data.


How is the modulo number selected to build the hash table in DHP algorithm?

One possible reason to define the function this way is that it works for their example. It is an academic paper so it incentivized to find a novel solution that works on a specific dataset. In the paper, it is a synthetic dataset.

When building hash tables it is best practice to pick a prime (7 in this case) to minimize the number of collisions when using the modulo operator. The fact that is small prime is a clue that they building a toy system.


PoS tagging works for natural language only and identifies grammatical parts of the sentence, nothing more. LSTM is an algorithm that can be used to predict series. Named Entity Recognition (NER) and Terminology Extraction could work if you have already data to engage in Information Extraction (IE). However, In order to use a these techniques you need to have a trained model and in order to train one you need data. In your case that would involve identifying and tagging parts of the sentence by hand and later training a model with that data.
The best approach, in my opinion, is just use regex to identify parts of the sentence as one of the approaches of Information Extraction and use hard coded rules to best identify what you are trying to replace later.
For instance, if you want to search the version in the example:

import re
s = 'Releasing version. 0.0.1 for Stackoverflow, on 01/01/2019. The coverage is 99% and build is passed.'
re.search(r'Releasing version(.*?)for', s).group(1)
' 0.0.1 '

Check out these resources that will help you parse a log in Python using Regex.
https://pythonicways.wordpress.com/2016/12/20/log-file-parsing-in-python/
https://medium.com/devops-challenge/apache-log-parser-using-python-8080fbc41dda
But if you prefer to use Named Entity Recognition or Terminology Extraction techniques you could hack a NER model and train it yourself with your data. Keep in mind though, that according to Poibeau, Thierry; Kosseim, Leila (2001). "Proper Name Extraction from Non-Journalistic Texts". Language and Computers. 37 (1): 144–157.:

Research indicates that even state-of-the-art NER systems are brittle, meaning that NER systems developed for one domain do not typically perform well on other domains.

And according to Wikipedia: Considerable effort is involved in tuning NER systems to perform well in a new domain; this is true for both rule-based and trainable statistical systems.
So even if you pull that off, accuracy will be less than if you just extracted data using regex. An approximate of a reduction of 97% to 93% only for named entities (companies, names, etc.) Accuracy reduction will be much less in you case.
Check this link for more information about Information Extraction: https://web.stanford.edu/~jurafsky/slp3/17.pdf


$\varepsilon$ is referred as a noise term with 0 mean. The distribution is random in the real world, but you can make assumptions on its distribution.

For example the Gaussian Process machine learning suggests that it follows a Gaussian distribution i.e :  $\varepsilon \sim \mathcal{N}\left(0, \sigma^ 2 \right)$.

The variance $\sigma$ of the latter distribution can be seen as an hyperparameter that we obtaining by maximizing the likelihood function, or by having prior information on the data set.
You can find more information in this book Rasmussen, C. E., and C. K. Williams. "I (2006) Gaussian Processes for Machine Learning." (2006)

In some cases by looking at your a priori data, and estimating the possible error sources, you can a priori expect the type of the error distribution and try to assess this assumption a posteriori (mainly using data-driven method).


What is the intuition behind using Monte Carlo to solve a differential equation

The trick is to convert ODE/PDE into an integral equation and then Monte Carlo comes to play.
Here are some examples.
http://jotterbach.github.io/2018/08/08/MonteCarloODE/


When should mutual information be used for feature selection over other feature selection methods like correlation, ANOVA , etc?

Pearson correlation and ANOVA will measure the linear variance between x,y.
A mutual information regression method is an entropy-based method which can find even non-linear correlation.
Let's see 2 pair of data (x, y_1) and (x, y_2).
##0. Data setup
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y_1 = 3.39*x_1
y_2 = np.array([1, 1, -1, -1, 0, 0, -2, -2, 2, 2])
df_1 = pd.DataFrame([x,y_1]).T
df_2 = pd.DataFrame([x,y_2]).T

##1. Pearson corr
df_1.corr()
df_2.corr()

##2. f_test
from sklearn.feature_selection import f_regression, mutual_info_regression
f_test, _ = f_regression(x.reshape(10, 1),y_1)
f_test
f_test, _ = f_regression(x.reshape(10, 1),y_2)
f_test

##3. Mutual reg
mutual_info_regression(x.reshape(10, 1),y_1,n_neighbors=1,random_state=123)
mutual_info_regression(x.reshape(10, 1),y_2,n_neighbors=1,random_state=123)

##4. Tree-based feature importance is also based on entropy reduction
##and is more reliable as  KNN needs a good k value

 We can observe that Mutual importance is able to catch the relation of x with not just y_1 when it is simple Linear bust also with y_2 when it is not linear(but has a strong relation)
You should use either MI if you are only looking for a filter-based method. Otherwise, Tree feature importance would be best for most of the cases.
Check this link -
F_test Vs MI


Deep learning based Resume Parser and Scoring

Resume parsing and scoring can be done by identifying relevant keyword in the text. NER (Named Entity Recognition) is a technique to identify relevant keywords in text. There are different framework available to work with NER and NLP as a whole. I will suggest you to look into spacy
to train a custom NER model.


How to go about cooling load forecast for a district level dataset?

For a good overview of common statistical methods, check our Rob Hyndman's Forecasting Principles and Practices.
For a comparison of different models, including machine learning and deep learning, check out the M4 competition and their companion repo. The current state-of-the-art on the M4 data set is N-BEATS as far as I know.
For a replication and extension of the M4 study in Python, check out sktime (disclaimer: I'm one of the developers) and our preprint.


Text-Classification-Problem: Is Word2Vec/NN the best approach?

1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task.

2) Word2vec is definitely something worth trying and comparing to model 1. I would suggest using the Doc2Vec flavor for looking at sentences/paragraphs.

Quoc Le and Tomas Mikolov. Distributed Representations of Sentences and Documents. http://arxiv.org/pdf/1405.4053v2.pdf

Gensim(python) has a nice Doc2vec model.


How to combine two Deep learning model weights into one

Merging models is not as simple as it seems.
In my opinion, I see multiple possibilities :

Don't merge models, merge datasets and retrain : this is in my opinion the most reliable solution, models are fitted from a dataset which represent a certain distribution of data and features. If you can retrain : retrain, especially if datasets are differents.
Create a model over models : use your models as features extractors (cut final part) and build a siamese network (which compute features in parallel from two submodels), merge features obtained and add top layer which will classify from these new set of features (just retrain final part, freeze your models's layers). With this approach, you can retrain a new model which will keep both models's logic without having to retrain a full network. This approach is better than the first if retrain a model from scratch is too constraining.
Play around models's weights : you can access to weights of models and create a third by taking mean of weigths's layers. In my opinion, you can try this but I highly doubt of this approach, instead take the best model and use it.


Replace column in a dataframe with another column based on index

Try this:
A.loc[A.for_analysis & (A.index.isin(B.index)) , content_preprocessed] = B.content_preprocessed


Welcome to DataScience.SE! I'd never heard of this problem so I looked it up. It is explained on the third slide of this presentation by Geoff Hinton:

  More things that make it hard to recognize objects

  • Changes in viewpoint cause changes in images that standard learning
  methods cannot cope with.

  –  Information hops between input dimensions (i.e. pixels)

  • Imagine a medical database in which the age of a patient sometimes
  hops to the input dimension that normally codes for weight!

  –  To apply machine learning we would first want to eliminate this
  dimension-hopping.

In other words, it is about conceptual features migrating or hopping from one input feature dimension to another while still representing the same thing. One would like to be able to capture or extract the essence of the feature while being invariant to which input dimension it is encoded on.


Can you reuse observations from your train data in your final test data?

The Out of Bag(OOB) error in XG Boost is considered as a Validation error(test error). During Hyper-parameter selection in XG boost, OOB error is measured for different hyperparameters. In XG Boost algorithm, the decision trees are constructed by a random sample of observations and the successive decision trees are built by giving more importance to the observation which the previous trees are misclassified. As this XG boost keeps increasing the number of decisions trees, the validation error decreases and saturates at some point.

In your case, you can consider this OOB error as test error instead of creating a separate test set.

The following plot explains this more accurately


Is this Neo4j comparison to RDBMS execution time correct?

Looking at this document called Anatomy of Facebook I note that the median is 100. Looking at the cumulative function plot I can bet that the average is higher, near 200. So 50 seems to not be the best number here. However I think that this is not the main issue here.

The main issue is the lack of information on how the database was used.

It seems reasonable that a data storage designed specially for graph structures to be more efficient than traditional RDBMs. However, even if the RDBMs are not in the latest trends as a data storage of choice, these systems evolved continuously in a race with the data set dimensions. There are various types of possible designs, various ways of indexing data, improvements related with concurrency and so on.

To conclude I think that regarding reproducibility, the study lack a proper description of how the database schema was designed. I do not expect that a database to dominate on such king of interrogations, however I would expect that with a well-tuned design the differences to not be such massive.


Why do RNNs usually have fewer hidden layers than CNNs?

CNNs and RNNs feature extraction methods:

CNNs tend to extract spatial features. Suppose, we have a total of 10 convolution layers stacked on top of each other. The kernel of the 1st layer will extract features from the input. This feature map is then used as an input for the next convolution layer which then again produces a feature map from its input feature map.

Likewise, features are extracted level-by-level from the input image. If the input is a small image of 32 * 32 pixels, then we will definitely require fewer convolution layers. A bigger image of 256 * 256 will have comparatively higher complexity of features.

RNNs are temporal feature extractors as they hold a memory of the past layer activations. They extract features like an NN, but RNNs remember the extracted features across timesteps. RNNs could also remember features extracted via convolution layers. Since they hold a kind-of memory, they persist in temporally/time features.

In case of electrocardiogram classification:

On the basis of the papers you read, it seems that,

  ECG data could be easily classified using temporal features with the help of RNNs. Temporal features are helping the model to classify the ECGs correctly. Hence, the usage of RNNs is less complex.

The CNNs are more complex because,

  The feature extraction methods used by CNNs lead to such features
  which are not powerful enough to uniquely recognize ECGs. Hence, the larger number of convolution layers is required to extract those minor features for better classification.

At last,

  A strong feature provides less complexity to the model whereas a
  weaker feature needs to be extracted with complex layers.

Is this because RNNs/LSTMs are harder to train if they are deeper (due to gradient vanishing problems) or because RNNs/LSTMs tend to overfit sequential data fast?

This could be taken as a thinking perspective. LSTM/RNNs are prone to overfitting in which one of the reasons could be vanishing gradient problem as mentioned by @Ismael EL ATIFI in the comments.

I thank @Ismael EL ATIFI for the corrections.


What is the way to modify a neural network classifier to deal with sample points from outside of the label set?

This question is a very common question.
How to handle samples belong to categories that are not included in the data set. There is no optimal way to handle this:

Creating a separate class is possible but not recommended. First, you need images to start with, second this class will have a very large variation within the class, a property that is not favorable which makes designing a powerful classifier to handle the other classes more challenging and usually results in a low quality classifier.

I prefer to handle this by measuring the confidence. If you are expecting to have anomalies/ outliers,  make sure to use classifier that gives both prediction and probabilities ( confidence).

CNN is good enough , you can only accept predictions if the confidence is a higher than a threshold, or the difference between the highest confidence ( label) and the next confidence (label) is higher than a threshold. Most of the probability based classifies can give you confidence values.

Some other classifiers can be changed to give confidence, for example, you can analyze the training data used to give a label for a leave in decision tree, these samples belonging to the same leave can be clustered to get some centroids. In prediction you can predict the label and measure the shortest distance between your target ( testing sample) and each of the centroids belonging to the same leave you end with, that shortest distance can be used as a measure of confidence and can be compared to a threshold to accept / reject the prediction.

Please note that this problem is still under research and there is no clear answer to it. It needs very deep understanding to each classifier to modify it / or add to it to measure confidence and to understand what  that measure means


Should the input data be normalized using keras pre-trained models

According to Very Deep Convolutional Networks for Large-Scale Image Recognition, which is the paper that first presented VGG: "...The only pre-processing we do is subtracting the mean RGB value, computed on the training set, from each pixel."- (Karen Simonyan, Andrew Zisserman).
I am certain that Keras provides a preprocess function based on the above mentioned principle.
Checkout: tensorflow.keras.applications.vgg16.preprocess_input
If it is in the TF version, its certainly in the other version.


When I add regularization like L1,L2 , do I need more epochs to properly train my model?

The convergence time is sensitive to the data you have and a random seed. Specifically, the convergence time is linear in expectation in all three cases. SGDClassifier uses the stochastic gradient descent for optimization. Since L1 loss is only subdifferential, the L1 penalty causes the algorithm to converge noticeably slower.
Comparing with or without the L2 penalty, it is not clear what algorithm is faster. The loss function is differential. The L2 penalty may be faster in the underdetermined case. In the example below, I consider the gradient descent instead of the stochastic linear descent and regular regression to simplify the argument. Say, we aim to solve y = Xb + e, where we observe y and X only. We set the loss function to be f(b) = 0.5||y - Xb||^2. Without regularization, the solution is sol1 =(X^TX)^{-1}X^Ty and with L2 regularization, the solution is sol2 = (X^TX + lambda I)^{-1}X^Ty. In the latter case, we can guarantee that the matrix to invert is not close to singular, and, therefore, the faster convergence is expected.
In short, on average, I would expect the following number of iterations requires from smallest to largest ON AVERAGE:

L2 penalty
No penalty (potentially, with a close tie with L2 penalty)
L1 penalty

You observe the opposite order. It should be very specific to your data or random seed.


Kmeans on mixed dataset with high level for categ

There are plenty of methods, variations of k-means for the case of mixed dataset: k-modes, k-protoypes etc.

It has been discussed already.


As a general rule of thumb you should avoid doing different things between your train and test dataset. As a second general rule of thumb you rarely want to use knn for missing value imputation.
One efficient way to deal with missing value in your case would be to use a model that can handle missing values, like a tree model. (decision tree, random forest, xgboost...).


How to use GAN for unsupervised feature extraction from images?

Typically to extract features, you can use the top layer of the network before the output. The intuition is that these features are linearly separable because the top layer is just a logistic regression.

For GANs, you can use the features from the discriminator. These features are supposed to give a probability if the input came from the training dataset, "real images". In Radford's DCGAN paper, they use all the convolutional layers of the discriminator and run a max pooling layer extract features for CIFAR-10.

  To evaluate the quality of the representations learned by DCGANs for supervised tasks,
  we train on Imagenet-1k and then use the discriminator’s convolutional features from all layers,
  maxpooling each layers representation to produce a 4 × 4 spatial grid. These features are then
  flattened and concatenated to form a 28672 dimensional vector and a regularized linear L2-SVM
  classifier is trained on top of them.


Some things that could be happening:

You do not have enough parameters in your hidden Layer to learn. Try something like [67, 512, 5] or expand deeper with something like [67, 1024, 256, 5]. The idea is that your hidden layer may be too small to capture interactions between the attributes given the amount of training data you have.

A rule of thumb that I have seen is:

Nh=Ns/(α∗(Ni+No))

Nh = number of hidden neurons.
Ni = number of input neurons.
No = number of output neurons.
Ns = number of samples in training data set.
α = an arbitrary scaling factor usually 2-10.

Which in your case gets you to like 5000 neurons, but since you are oversampling your classes, I would also suggest under-sampling instead, and that gets you maybe to around 1024, which is fairly standard. The [67, 512, 5] configuration is pure gut reaction.

Learning rate is too high. Try starting with lower LR (like 10e-5). Sometimes this works with Adam.
Weight Initialization : double check you are using Xavier.


Could one algorithm fetch keywords from texts of different natural languages?

There are a few ways to deal with this issue.  Python has a package called NLTK which contains stop word lists for several languages (including English and Norwegian).  You can simply use this package, it's usage is as follows:

>>> from nltk.corpus import stopwords
>>> stop = stopwords.words('english')
>>> sentence = "this is a foo bar sentence"
>>> print [i for i in sentence.split() if i not in stop]
  ['foo', 'bar', 'sentence']

Alternatively, a method for automatically suppressing stop words is called tf-idf;  tf-idf is commonly used in search engines so that the most important words are promoted to the forefront.  In your case, I would suspect you'd want to have IDF scores for both English and Norwegian and apply only the appropriate one on a language to language basis.

http://en.wikipedia.org/wiki/Tf%E2%80%93idf


How to reshape ndarray to calculate the mean squared error?

The issue is not the shape of your data. If you would like to calculate any kind of score for either your train or test predictions you need to compare the true labels to the respective predictions (either for the training data or the test data but not mixing these two):

y_train_pred = model.predict(X_train)
mae_train = mean_absolute_error(y_train, y_train_pred)

And correspondingly for your test data:

y_test_pred = model.predict(X_test)
mae_test = mean_absolute_error(y_test, y_test_pred)

(for the example I am assuming that your model object has been assigned to the variable model and used the predict() syntax of SKLearn here)


Can Sequence to sequence models be used to convert code from one programming language to another?

Yes, it is possible to convert code from one programming language to another using sequence-to-sequence neural networks. Sequence-to-sequence can learn to translate anything from anything if there are consistent patterns and enough paired examples.

However, it would not be efficient because the models would only indirectly learn the semantics (i.e., the meaning of the code) through the observed string literals (i.e., the characters that appear). It would be more efficient to map one observed sequence of string literals to semantics, then use the latent semantics representation to generate a new observed sequence of string literals.

There are a variety of ways of doing that depending on much you handicap yourself. If you give the model access to abstract syntax tree (AST), it becomes much easier. Then a tree-based Long Short-Term Memory (LSTM) could learn the semantic space.

Overall, this is a field of study that tries to build a Neural Network Transpiler.


The solutions for sequences with different length are:

1- padding, but you do not want to.

2-truncate the sequences, so if you have for instance 3 sequences of different lengths you truncate the sequences so that they have all the length of the shortest one.

3- using timesteps of 1, but in this case your model will not learn the dependencies between different timesteps

4- using online learning with batch size = 1.

For the fourth solution this is an example in keras to create a random time-length batches of training data: Training an RNN with examples of different lengths in Keras. So you can have different batches with different lengths, and since you are using online learning with batch_size = 1 this can be a good solution. The fourth solution can also have advantages and disadvantages. For online learning the training is slower, but usually you need less epochs compared to sgd with batch size > 1 to have a good model. If you use stateful lstm, this could lead you to an unstable model, not always, but can happen.


How to predict correct text from incorrect text with machine learning?

It's unclear from your example how exactly you intend to predict if all you have are pairs of terms.
If you're assuming that you have all possible correct and incorrect pairs, then you can simply use your data as a hash table where each incorrect term is the key of a correct term.
Otherwise, assuming you have all the correct terms but you want to be able to handle slightly changed versions of the incorrect terms.
You can define a similarity measure (function) and use that to find the closest incorrect term, in order to return the correct term for it.
If you don't have all the correct words, then it's a problem.
You should look for a vocabulary of 'legal' terms (or names in your case) and use that as your dictionary.
There are some libraries in python that can do that for you, but as far as I know these are all based on regular dictionaries, like the ones used by LibreOffice for example (PyEnchant) is one example for that.
If you're looking for a dictionary for names, you should perhaps try looking for a specific dataset that consists of names.

Otherwise your data should be in a different format, which will have more information in it, such as context for example.

One way to 'predict' correct words based on incorrect words is constructing a LM (Language Model) for each (correct and incorrect) and based on probabilities returning the ones with maximal likelihood, like a basic translation model. But this will only make sense if you have a lot of text (not unigrams) and each word appears several times, otherwise you don't really have a way to construct the LMs.


To answer your initial question, yes you can use silhouette score with different clustering methods. You could also use the Davies-Bouldin Index or the Dunn Index.
Regarding over-fitting, (this is my personal suggestion) but you could train the model n times on different types of the same data to see if there clustering is the same even though the values are changed. Short example: If you have to cluster 5 apples and 6 oranges, the cluster should be the same for 10 apples and 12 oranges. You can find a bit more detail on this here: https://datascience.stackexchange.com/a/20292/103857
For your third query:
Calculate distances between data points, as appropriate to your problem.
Then plot your data points in two dimensions instead of fifteen, preserving distances as far as possible. This is probably the key aspect of your question. Read up on multidimensional scaling (MDS) for this. Finally, color your points according to cluster membership.
(source for third query: https://stats.stackexchange.com/a/173823)
Regarding pca, its subjective. PCA works well with high correlation. If your dimensions are like apples and oranges then your directly effecting your models performance, so do keep that in check. A bit of eda would help before you dive into that.


How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

To answer your specific questions:

AdditiveAttention() and Attention() layers, are (loosely but not exactly) based on Bahdanau and Luong's attentions, respectively.

They use post-2018 semantics of queries, values and keys. To map the semantics to the Bahdanau or Luong's paper, you can consider the 'query' to be the last decoder hidden state. The 'values' will be the set of the encoder outputs - all the hidden states of the encoder. The 'query' 'attends' to all the 'values'

If you run thru' the library code you will see that the query is first expanded over the time axis and then there is a dense layer that determines the weights of w1, w2. These weights are applied to the expanded query and the values and then they are added and another weight 'v' is finally applied. A softmax of this is taken to return the attention weights and then these attention weights are multiplied with the 'values' and added to return the context. This is Bahdanau's additive logic

However, while analysing tf.keras.layers.Attention Github code to better understand how to use the same, the first line I could come across was - "This class is suitable for Dense or CNN networks, and not for RNN networks". Since you are using RNN, I would use this layer with caution.  In general, all these ready-to-use layers are mostly for self-attention and if you want to create a transformer like model where you are doing away with the RNN altogether and want too use only attention to represent the sequence, you can consider these classes.

If you still want to use the same, you can go ahead and try the following:

    ##Input 1 = the last decoder hidden state: stminus1
    ##Input 2 = All hidden states of the encoder: lstm_out
    ##Apply Bahdanau additive attention and give me the
    ##output = context
    context = tf.keras.layers.AdditiveAttention()([stminus1, lstm_out])

You can now use the context additionally to strengthen the predictions.
However I would strongly recommend you write your own attention layer in less than half a dozen lines of code. See for e.g.: https://stackoverflow.com/questions/63060083/create-an-lstm-layer-with-attention-in-keras-for-multi-label-text-classification/64853996#64853996


None of the methods you described may classify a dataset alone whereas both can be used to transform your data into another domain in an unsupervised fashion.

PCA projects your data onto n-orthogonal components. A trained encoder (first component of the autoencoder) can project your data onto a latent space.

Both of those representations can be used in conjuction with a classifier, such as a decision tree, to form a classification pipeline.

There is no globally best solution, try them both for your specific problem. PCA is, in my opinion, way faster to try out using sklearn instead of building your own autoencoder using PyTorch or Keras. Give PCA a first go and if the results are poor, try using an autoencoder.


NoSQL vs SQL backend for semi structured data

I would use SQL and create a set of structured fields that are common across all applications (name, school, years of experience, job they are applying for, etc.) with a field containing the raw application for you 'semi-structured' part of the data. You can always do something fancy with the raw application field, but if you want to get some summary statistics quickly then SQL is the proper route.

In general, thinking carefully about your schema up front will pay big dividends in the future. I would recommend NoSQL only if you really have no idea how to structure things and are pressed for time. Even getting a list of the keys in a collection requires MapReduce.


There might not be an entire book about unsupervised techniques, given the greater effectiveness of supervised learning techniques. However, there are small books and selected chapters from longer books that could be relevant:

"Unsupervised Machine Learning in Python" which includes Gaussian Mixture Models
"Python Machine Learning" which has a section on Latent Dirichlet allocation (LDA)
"Modeling Techniques in Predictive Analytics with Python and R: A Guide to Data Science" which has a chapter on Market Basket Analysis
"R for Marketing Research and Analytics" which covers Confirmatory Factor Analysis and Structural Equation Modeling


What is difference between Standard Normal Distribution and Mean Normalization approaches to feature-scaling?

The terms standardization and normalization are often used interchangeably. However, strictly speaking they do refer to distinct feature transformations.
Normalization
Normalization, also called feature scaling usually means scaling the data between 0 and 1. There are many approaches that can be used to achieve this. One common way is by
$x' = \frac{x - x_{min}}{x_{max} - x_{min}}$
Standardization
Standardization transforms the feature to have a mean 0 and a standard deviation of 1. This is also called z-scoring and can be achieved by
$x_i' = \frac{x_i - \bar{x}}{s}$
where $\bar{x}$ is the mean of the feature and $s$ is the standard deviation of the feature.


What is stored in heap structure in the following example?

You would put triplets into your heap, storing

(distance, i, j)

where distance is the distance, i and j are the indices.

When you "pop" the topmost (smallest distance) element from the heap using the appropriate algorithm, the second smallest will now become the top.

Python has a module heapq for building min-heaps. Simply add tuples with heapq.heappush, and remove the smallest with heapq.heappop. To convert an array into a heap, use heapq.heapify (faster than calling heappush for every element).

Note that putting n^2 values into a heap yields runtime O(n^2 log n^2) which is fairly expensive.


Is a correlation matrix meaningful for a binary classification task?

Well correlation, namely Pearson coefficient, is built for continuous data. Thus when applied to binary/categorical data, you will obtain measure of a relationship which does not have to be correct and/or precise.

There are quite a few answers on stats exchange covering this topic - this or this for example.


TL;DR
Use Stacked Cross-Validation instead of traditional K-Fold Cross-Validation.
Stacked Cross-Validation
In Sckit-learn, this is called TimeSeriesSplit (docs).
The ideas that instead of randomly shuffling all your data points and losing their order, like you suggested, you split them in order (or in batches).
Belows is a picture of traditional K-Fold vs Stacked K-Fold. They both have K=4 (four iterations). But in the traditional K-Fold all the data is used all the time, whereas in the Stacked K-Fold only the past data is used for training and the now data for testing.

Strategy

Split your data in a Stacked K-Fold fashion.
At every iteration, store your model and measure its performance against the test set.
After all iterations are over, pick the stored model with the highest performance (might not even be the last one).

Note: This is a common approach in training neural nets (batching with validation sets).


How to implement gridsearchCV for onevsrestclassifier of LogisticRegression classifier?

When you use nested estimators with grid search you can scope the parameters with __ as a separator. In this case the LogisticRegression model is stored as an attribute named estimator inside the OneVsRestClassifier model:
from sklearn.model_selection import GridSearchCV
from sklearn.linear_model import LogisticRegression
from sklearn.multiclass import OneVsRestClassifier

tuned_parameters = [{'estimator__C': [100, 10, 1, 0.1, 0.01, 0.001, 0.0001]}]

# Find Optimal C by grid search

log_reg_clf = OneVsRestClassifier(LogisticRegression())

logistic_gs = GridSearchCV(log_reg_clf, tuned_parameters,scoring = 'f1_micro', cv=3)

logistic_gs.fit(x_train_bow, y_train)
print(logistic_gs.best_estimator_)


Boosted tree regression loss function when data has occasionally very large values to predict?

I am working almost on the same problem these days:

I have tried two options using XGB Regression with different objective functions including:

Using a linear regression objectiive function ("reg:linear" or "reg:squarederror") and transformed the target to the log space
Using the gamma objective function ("reg:gamma"), which is useful for a skewed target with gamma distribution, e.g., insurance claims severity. In this case, I did not transform my target to the log space.

You can try these two cases and see which one performs better. However, in my case, option 1 performed better than option 2 (around 15-20%).

Also, you can try "reg:squaredlogerror"


How can I create .nii (nifti) file from 3D Numpy array

You can this using nibabel:
import nibabel as nb

ni_img = nib.Nifti1Image(numpy_array, affine=np.eye(4))
nib.save(ni_img, "dicom_volume_image.nii")


what does `a factor of K-fold` mean in GPU-based training?

The author here refers to a comparison between older and most recent implementations of deep learning algorithms regarding execution time (accelerating processing speed). It just means that "they used GPUs and achieved x40 better execution time". It has nothing to do with K-Folds.


Understand a data
Find the most suitable metrics
handle missing values
normalize, scale etc your variables
feature selecting
use cross validation(kfold for example)
try oversampling or undersampling
trying other models


Feature importance after PCA (or other dimensionality reduction methods)

Most of the PCA methods return the linear transformation matrix which allows to convert from Component to Variable and viceversa.

What is impossible to do is to assign a feature importance from component to variable (unless you are willing to accept the assumption that "feature importance" could be assigned linearly).

If you accept the idea between parenthesis, you could have a linear equation system $Ax = b$ where $A$ is the component-to-variable matrix (of size components-by-variables), $b$ is the feature importance and $x$ is the result (the variable importance).

This idea I am giving to you is what I think could work, you are likely not going to find it in any research or similar.


Can one property name be used twice in the same branch of a DecisionTreeRegressor?

Good job looking at the tree and understanding what has happened.

There is no problem splitting on the same feature multiple times. A continuous feature has many split points available. The tree continues to subset and refine. The split criteria shows what will be the "best" greedy split at this point. If a feature is income, perhaps the best split is \$100,000. Then on the high side, there is another split for \$10,000,000 since those people behave differently from the \$100,100 income people.

Even a categorical variable may split again. For example black and blonde hair go left, all other hair color go right. Later splitting black and blonde is the best split available.

I saw a research project where scikit learn code was adjusted to give split priority to features that were already used to reduce the number of features in the model which may happen in trees on spurious interactions due to greediness. It worked well.


how to extract substrings from a dataframe column

In your specific case, when you want to find something between two different markers, you can use the .split(marker) method as follows:

In [1]: s = "{:fbbbhbhbh{,:50k: ghgjlj45llj,ljhjlhlhj,clause :59:jhjhhjxjhj65@,j
   ...: jjhjhd :70"

In [2]: s.split("50k:")[1].split("59:")[0]
Out[2]: ' ghgjlj45llj,ljhjlhlhj,clause '

If you want to get rid of the spaces at the end, just throw in another method at the end: strip():

In [3]: s.split("50k:")[1].split("59:")[0].strip()
Out[3]: 'ghgjlj45llj,ljhjlhlhj,clause'

You can add that to a function as you did with your own code, and put the results into a Pandas Dataframe.

def my_parser(s, marker1, marker2):
    """Extract strings between markers"""
    base = s.split(marker1)[1].split(marker2)
    part1 = base[0].strip()
    part2 = base[1].strip()
    return part1, part2

Now use the function to iterate over all thew string you have

results = []    # to collect results of each example
m1 = ...        # put whatever markers you need here
m2 = ...
for line in lines:
    results.append(my_parser(line, m1, m2))

# Create a dataframe from a list of lists (i.e. from_records). Each inner list becomes a row
df = pd.DataFrame.from_records(results)

Edit:

Given you have your strings in a DataFrame already and just want to iterate over them, you could do the following, assuming you have my_col, containing the strings:

for line in df.my_col:
    results.append(my_parser(line, m1, m2))    # results is a list as above

So that is what you said you wanted to extract, but it will maybe not generalise well. For example, if there are multiple of those markers in your sentence, you might get unexpected results, or at least only the first occurrence of what you want to extract (if there can be many in your other text examples).


Smoothing or averaging plots to better represent trends and their variations

You could take averages piecewise. Ideally, is like sliding a window of size k on the trend, taking the average of these points (it could be 3 or 5, for example).

Alternatively, you can use exponential smoothing functions, such as savgol filter available in scipy, LOESS smoothing, or Holt-Winters smoothing. The statsmodels.tsa submodule made many smoothing techniques available.


What should be the labels for subword tokens in BERT for NER task?

Method 2 is the correct one.

Leave the actual label of the word only in the first sub-token, and the other sub-tokens will have a dummy label (which in this case is 'X'). The important thing is that when calculating the loss (e.g., CELoss) and metrics (e.g., F1), this 'X' labels on the sub-tokens are not taken into account.

This is also the reason why we don't use method 1 is that otherwise, we would be introducing more labels of the type [B-count] and affecting the support number for such a class (which would make a test set no longer comparable with other models that do not increase the number of labels for such class).


How to correctly infer vectors in Gensim doc2vec?

I would use the first approach, given that both train and test are known, there is no need of generalization i.e. you don't expect unseen vectors.

In order to avoid the problem you mentioned, you have to find the most similar vector to a vector in test considering  only vectors in train. For example:

train = [v1, v2, v3]
test = [v4, v5]
most_similar = {}
for vector in test:
most_similar[vector] = v1
  for vector2 in train:
   if similarity(vector2, vector) > similarity(most_similar[vector], vector):
     most_similar[vector] = vector2

In the end in most_similar you have the most similar vector in train for each vector in test.


Thank you for your reply.
By accident I found out that MicrosoftML (https://msdn.microsoft.com/en-us/microsoft-r/microsoftml-introduction) offers just what I need - by using the featurizeText functions offered within Microsoft R client and R server.
Maybe it helps others - I found their example  and could it translate to my data (featurizeText() help under Microsoft ML).

trainReviews <- data.frame(review = c(
        "This is great",
        "I hate it",
        "Love it",
        "Do not like it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I kind of hate it",
        "I do like it",
        "I really hate it",
        "It is very good",
        "I hate it a bunch",
        "I love it a bunch",
        "I hate it",
        "I like it very much",
        "I hate it very much.",
        "I really do love it",
        "I really do hate it",
        "Love it!",
        "Hate it!",
        "I love it",
        "I hate it",
        "I love it",
        "I hate it",
        "I love it"),
     like = c(TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE, FALSE, TRUE, FALSE, TRUE,
        FALSE, TRUE, FALSE, TRUE), stringsAsFactors = FALSE
    )

    testReviews <- data.frame(review = c(
        "This is great",
        "I hate it",
        "Love it",
        "Really like it",
        "I hate it",
        "I like it a lot",
        "I love it",
        "I do like it",
        "I really hate it",
        "I love it"), stringsAsFactors = FALSE)

outModel <- rxLogisticRegression(like ~ reviewTran, data = trainReviews,
    mlTransforms = list(featurizeText(vars = c(reviewTran = "review"),
    stopwordsRemover = stopwordsDefault(), keepPunctuations = FALSE)))
# 'hate' and 'love' have non-zero weights summary(outModel)


Classify sensor data (multivariate time series) with Python's scikit-learn decision tree

For usage you need to flatten the 2D raw sensor data into 1D features. Below code demonstrates the basics.

What kind of feature engineering to apply for best predictive effect depends entirely on the nature of your sensors and problem. There are no details about this in the question or data provided.

Feature Engineering

The overall process is:

Look for patterns in the data (Exploratory Data Analysis)
Attempt to create a new feature which describes this pattern
Evaluate the new set of features using cross-validation
Analyze the samples that your classifier got wrong (Error Analysis)
Repeat from 1) until performance is good enough

Here are some things you should try:

Plot the raw sensor data from a few samples of the positive and negative class.
Plot the distributions (histogram) for each class of each raw sensor values across the entire dataset.
Try to standardize the data. For each time-series of sensor data, remove the mean and divide by the standard deviation for each sample.
Try some standard statistical summarizations on each time-series. Max, min, mean, std, skew, kurtosis. Unlikely to be better than something tailored to the patterns you see, but sometimes performs OK.

Focus first on uni-variate features per sensor. The decision-tree will be good at combining these together.

Fitting to classifier

import numpy
import pandas
from sklearn.ensemble import RandomForestClassifier

def get_sensor_data():

    timesteps = 10
    times = numpy.linspace(0.1, 1.0, timesteps)
    df = pandas.DataFrame({
        'time': times,
        'sensor1': numpy.random.random(timesteps),
        'sensor2': numpy.random.random(timesteps),
        'sensor3': numpy.random.random(timesteps),
        'sensor4': numpy.random.random(timesteps),
    })

    return df

samples = [ get_sensor_data() for _ in range(100) ]
labels = [ int(numpy.random.random() > 0.5) for _ in range(100) ]
assert len(samples) == len(labels)

print('sample from CSV file:\n', samples[0], '\nlabel', labels[0], '\n')

def to_features(data):
    # remove time column
    feature_columns = list(set(data.columns) - set(['time']))
    # TODO: do smarter feature engineering here
    sensor_values = data[feature_columns].values
    # Note: the features must be 1D for scikit-learn classifiers
    features = sensor_values.flatten()
    assert len(features.shape) == 1, features.shape
    return features

features = numpy.stack([ to_features(d) for d in samples ])

assert features.shape[0] == len(samples)
print('Features:', features.shape, '\n', features[0])

# XXX: do train/test splits etc
est = RandomForestClassifier(n_estimators=10, min_samples_leaf=0.01)
est.fit(features, labels)

Example output

sample from CSV file:
    time   sensor1   sensor2   sensor3   sensor4
0   0.1  0.820667  0.346542  0.625512  0.774050
1   0.2  0.821934  0.241652  0.485608  0.188131
2   0.3  0.264697  0.780841  0.137018  0.117096
3   0.4  0.464143  0.457126  0.972894  0.600710
4   0.5  0.530302  0.027401  0.876191  0.563788
5   0.6  0.598231  0.291814  0.588032  0.143753
6   0.7  0.627435  0.036549  0.276131  0.311099
7   0.8  0.527908  0.197046  0.580293  0.123796
8   0.9  0.068682  0.880533  0.956394  0.787993
9   1.0  0.244478  0.306716  0.586049  0.373013
label 1

Features: (100, 40)
 [0.82066682 0.62551234 0.77405    0.34654243 0.82193414 0.48560828
 0.18813108 0.24165186 0.26469686 0.1370181  0.11709553 0.78084136
 0.46414318 0.97289382 0.60070974 0.45712632 0.53030219 0.8761905
 0.5637877  0.02740072 0.59823073 0.58803188 0.14375282 0.29181434
 0.62743516 0.27613083 0.31109894 0.03654882 0.52790773 0.58029298
 0.1237963  0.19704597 0.06868206 0.95639405 0.78799333 0.88053276
 0.24447754 0.5860489  0.37301339 0.30671624]

```


Heuristic argument for Weight decay and regularization

Perhaps I can answer the first and second one:

(1)According to (93) on his book:
$w \rightarrow w=\left(1-\frac{\eta \lambda}{n}\right) w -\frac{\eta}{m}
  \sum_x \frac{\partial C_x}{\partial w}$,
there is an item $(1-\frac{\eta \lambda}{n})$on the weight, so weights will shrink every epoch. As weights were random chose at the beginning, it seems reasonable that the changes(the item $\frac{\eta}{m}\sum_x\frac{\partial C_x}{\partial w}$) will be chaotic on first epoch. Maybe that is why the weight decay will dominate the first epoch.
And there is another explanation here, which may be helpful.

(2)At each epoch, the item $(1-\frac{\eta \lambda}{n})$ will be multiplied $\frac{n}{m} $times, where $n$ is the number of training data, $m$ is the size of minibatch. So after each epoch, the weight will decay by a factor of$(1-\frac{\eta \lambda}{n})^\frac{n}{m}$. According to the definition of $e$:$e=\lim_{n\to \infty}(1+\frac{1}{n})^n$, lets change the format a little bit:

$(1-\frac{\eta \lambda}{n})^\frac{n}{m}=[(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}}]^{-\frac{\eta \lambda}{m}}$

As you can see, when $\eta \lambda \ll n$,  $(1-\frac{\eta \lambda}{n})^{-\frac{n}{\eta \lambda}} \approx e$.Then we can come to the conclusion that provided $ηλ≪n$ the weights will decay by a factor of $exp(−ηλ/m)$ per epoch.

(3)I have been through this for two days and have no idea about it,but I think it should be related to the formula(85):

$\begin{eqnarray} C = -\frac{1}{n} \sum_{xj} \left[ y_j \ln a^L_j+(1-y_j) \ln
(1-a^L_j)\right] + \frac{\lambda}{2n} \sum_w w^2.
\end{eqnarray}$

where the weights were summed on last item.


Finding user similarities within informal data sets

As you accept vague answers:
Sranford NLP tools are strong for this kind of stuff. NER, POS Tagger maybe, Parsers, etc. Now for machine learning itself, I would try looking up at WEKA, it's got a lot of filtering, classifiers and clustering methods, including StringToWordVector filters, that are, in my opinion, fundamental to text classification. Mostly, the tags you should be looking for are Text Categorization, Natural Language Processing and even Sentiment Analysis if you will.


First steps when analyzing a company's data

The thing you need to understand as completely as possible is how they expect a data analysis to enable them to achieve their objective. They are a business, so their overall objective is likely related to maximising profit. However, there will be a more immediate objective underneath that heading. To maximise profit you can either reduce costs or increase sales. In turn, to increase sales you can increase the number of customers or increase the amount of sales to each customer etc.

The question then turns on how you can use data science to perform one those objectives.

For example, questions that can almost be answered with data science could be 'how do I better identify potential customers?' or 'how do increase existing custmers' spend?' These are still very high level questions, but they are the sort of questions that you need to have in mind as you start to do your descriptive stats etc.

Bear in mind that this is an iterative process and it is completely normal to start off in a fuzzy sort of area. At this stage it is almost the case that having a question in mind is a McGuffin - it will kick things off, but it may not be the question you end up answering.

The CRISP-DM process is a process that has been built for data mining that discusses how to iteratively use results from analyses and models to increase your understanding of the customer's situation, and hence drive the development of a better business objective for use in a data science project.


ROC curve for different hyperparameters of `RandomForestClassifier`?

Seems like you already tuned your algorithm based on the mean_test_score. You also could tune the AUC, which is the area under the ROC curve. Tuning this can even provide better mean_test_score as I show in my blog post: http://philipppro.github.io/Tuning_random_forest/


Models to deal with huge possible outputs

There is something called "sampled softmax" (e.g. tensorflow's implementation), which simply partitions the output space and at each training step only takes into account one of the partitions (see section 3 from this article to learn the math). Sampled softmax is only meant to speed up training for very large output spaces; at inference time you use normal softmax.

Also, if your output is of hierarchical nature, you can use hierarchical softmax. See this blog post for a review of different methods to handle very large categorical space.