Has anybody created interesting ConvNets on 3D movie data?

The 2015 Thumos challenge was about action recognition in video collected from youtube - totalling 13k videos (430 hrs or 45 million frames) and 101 'action classes'. The video is of activities like people brushing their teeth, playing basketball, and surfing, and some video clips included multiple activities.

The video is untrimmed, so it's closer to a real-world setting than video edited down to exclusively a single action of interest. The winning and second place entries for action classification included CNNs in various guises (after a quick scan I couldn't find any entries that didn't).

Another use you might be interested in is video captioning (not classification): this paper incorporates CNNs to describe in words what is taking place in each video (examples from p9 of the article, I love the descriptions at bottom right):


static and dynamic data in clinical trials

What you are referring to is called a multi-input model and can be esaily built in most deel learning frameworks. The idea is to have both types of data as separate inputs, then use specific layers depending their types (recurrent layers to sequence data, CNN to images, and so on...) to later on concatenate them together.

If you can use Keras, there is the functional Api which is specialy well suited for the task at hand. An example of code (based the example given in the documentation) for your problem could be:

from keras.layers import Input, Embedding, LSTM, Dense, merge
from keras.models import Model

# headline input: meant to receive sequences of 100 integers, between 1 and 10000.
# note that we can name any layer by passing it a "name" argument.
main_input = Input(shape=(100,), dtype='int32', name='main_input')

# this embedding layer will encode the input sequence
# into a sequence of dense 512-dimensional vectors.
x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)

# a LSTM will transform the vector sequence into a single vector,
# containing information about the entire sequence
lstm_out = LSTM(32)(x)

#At this point, we feed into the model our auxiliary input data by concatenating it with the LSTM output:

auxiliary_input = Input(shape=(5,), name='aux_input')
x = merge([lstm_out, auxiliary_input], mode='concat')

# we stack a deep fully-connected network on top
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)

# and finally we add the main logistic regression layer
main_output = Dense(1, activation='sigmoid', name='main_output')(x)

#This defines a model with two inputss:
model = Model(input=[main_input, auxiliary_input], output=main_output)

#Then compite and train
model.compile(optimizer='rmsprop', loss='binary_crossentropy')
model.fit([headline_data, additional_data], labels,
          nb_epoch=50, batch_size=32)

In your case, the dynamic data would be the headline_input and your static data the auxiliary_input. The model will take both, apply the recurrent layer to the former and concatenate them to pass the union through the dense layers.

Of course, many of these parameters will depend on your data, but at least this example will give you an idea on how to build such model.

There is also this interesting project conditionall RNN which is also meant for this purposes. Worth taking a look.


There is another choice which popular recently: docker(https://www.docker.com). Docker is a container and let you create/maintain a working environment very easily and fast.

install essential tools for data science in python

https://registry.hub.docker.com/u/ceshine/python-datascience/

use r language to do data science

https://github.com/rocker-org/rocker

Hope that would help you.


BERT word embedings for finding word definition

BERT generates contextualized word embeddings, which means that BERTprovides the most accurate embeddings when a word is in a sentence(context).
For each of the words within the sentence, BERT will generate a vector of numbers. In your case, you will have a good representation of the word "bank".
So if you have a sentence for all the other words that you are trying to match with "bank", all you need to do is to extract the embeddings for them, then compare them with a similarity matric (i.e. cosine similarity). this way you are ranking them to see which ones are most correlated to "bank".


Data binning - Why we need to transform Categorical Variables?

Yes. If you need the information of your categorical variables then you need to represent them by numbers as most of machine learning algorithms are math-based.


Is this a good practice of feature engineering?

1) Yes, it makes sense. Trying to create features manually will help the learners (i.e. models) to graspe more information from the raw data because the raw data is not always in a form that is amenable to learning, but you can always construct features from it that are. The feature you are adding are based on one feature. This is common. However, your learner, logistic regression, is sensitive to multi-collinearity. You need to be careful on which feature and how many features to add. Otherwise, you model may overfit.

2) Since you are using a logistic regression, you can always use AIC or perform a statistical significance test, like chi-square test (testing the goodness of fit) before adding new structure, to decide whether the distribution of the response really is different with and without this structure. This is particularly useful when your data is scarce. Another way is to add penalty term to your model. For example, logistic lasso regression.

3) Keep adding new features is not always a good idea. Be careful with the curse of high-dimensionality. When adding new feature, you are actually adding a new dimension on your data.  Naively, one might think that gathering
more features never hurts, since at worst they provide
no new information about the class. But in fact their benefits
may be outweighed by the curse of dimensionality. I hope Useful things to know about machine learning session6 could help.


predict with Multinomial Logistic Regression

That's not the error I get and I'm thinking that you left out some code. I get:

predict(test$finalModel, newdata = dses, "probs")
Error in as.data.frame(newdata) : object 'dses' not found

I don't know why I see this so much, but you should avoid using the finalModel object for prediction. train is doing some things that the model from multinom sin't aware of (such as expanding dummy variables). Try using

predict(test, newdata = dses, "probs")

and never use predict(test$finalModel) again...

Max


What do each of the three SVM classes in R represent?

Go to the kaggle site that you shared.
Click on Data on the left. Click on datadict.txt. This is known as data dictionary, a file that help us understand the dataset.

We can see

23: outcome
- what eventually happened to the horse?
- possible values:
1 = lived
2 = died
3 = was euthanized

Those are the meaning of the $3$ classes. You might also like to print out hodata$outcome or do str(hodata$outcome).


Data augmentation when using flow_from_directory in CNN

Augmentations often rely on the nature of your data. Imagine if an the result of an augmentation would be logical in your context.

For example let's say you have a cats vs dogs dataset. The images here can be flipped left to right. On the other hand in the MNIST dataset it makes no sense to flip the images at all (what good would it be feeding a flipped '3' to your model). Some augmentations might actually confuse the model (e.g. an upside down '9' might look like a '6').

Furthermore, if augmetation is making your model worse, try smaller augmentations (e.g. stick to affine transformations with small ranges - $[-5%, +5%]$ rotation/translation/scaling).


How many layers should I replace in transfer learning CNN

To build on the previous answer:
In transfer learning, the goal is to use a pre-trained model and tweak the model to then specialise it to suit a certain task.
So, what we do is, as SrJ has eluded to, keep the main model's architecture in tact. So this would be the 6 CNN layers (and possibly the three linear layers, if they were also involved in pre-training).
After a model has been pre-trained, what we do is add additional layers to the model so that it is suited for our task. So in this case, the least you would do is have a final output softmax layer, which produces a probability distribution over the authors.
In between the final output layer and the original model's architecture, you can add more layers if it is appropriate.
When training this model with your task-specific data (this stage is called fine-tuning). We freeze our original model's architecture. This essentially means that the parameters within the models original layers will not change to prevent possible loss in generalisation performance. We only allow the additional layer's parameters to change during fine-tuning.
Overall message is to not replace layers, always add onto the existing model to tailor the model more to your classification task.


Random Forest, Type - Regression, Calculation of Importance Example

MSE is measure of error of the overall regression model, $\frac{1}{n}\sum\|y_i-\hat y_i\|^2$.

For an important variable, if it is replaced with random noise, you would imagine MSE with the faulty data to increase. IncMSE (Incremental MSE) for a particular variable is how much the MSE will increase if the variable is completely randomized.

This is usually computed on the out-of-bag data.

Node purity is a measure of how homogeneous a node is. An example of node purity is information entropy, i.e. $-p_1\log p_1-p_0\log p_0$ if there are two classes.
For regression models, node impurity is usually taken as the variance in a node.

Everytime you split a node, you do it to make the new nodes homogeneous, hence the purity increases.

IncPurity of a variable is weighted average of incremental purity because of each split by this variable was used to split, with the node population as the weight.


What's the name of data points that trained model gets on input in production?

I think you're talking about production data.  This is typically real data that you generate or acquire somehow, and it's the data you expect your trained model to classify or use to make predictions.

You mention that this is after initial training.  I'm assuming you've already used test data to measure your model's accuracy, and this is in fact now in production.


R Script to generate random dataset in 2d space

None of the algorithms you mention are good with data that has uniform distribution.

size <- 20             #length of random number vectors
set.seed(1)
x <- runif(size)          # generate samples from uniform distribution (0.0, 1.0)
y <-runif(size)
df <-data.frame(x,y)

# other distributions: rpois, rmvnorm, rnbinom, rbinom, rbeta, rchisq, rexp, rgamma, rlogis, rstab, rt, rgeom, rhyper, rwilcox, rweibull.

See this page for tutorial on generating random samples from distributions.

For specific set of randomized data sets that are 'hard' for these methods (e.r. linearly inseparable n-classes XOR patterns), see this blog post (incl. R code): http://tjo-en.hatenablog.com/entry/2014/01/06/234155.


Why is spam detection a classification problem and not a class modelling problem

Regarding the case of spam vs ham, you are right that the spam category has common features (words), whereas the ham category could have multiple sub-categories, each with distinct feature sets. However, these distinct features can also be used to label an instance as "ham". Eg: If spam messages don't usually talk about sports scores, then the occurrence of the word "score", could be used to classify a message as "ham", even when it co-occurs with typical spam words such as "ad", "offer", etc. However, the one-class classifier cannot make use of such a mechanism.

In a similar manner, specific features will help to identify a news article as belonging to "sports", "religion", etc, thereby making them easier to label as "non-political". The miscellaneous class would be more diverse, however.

One-class SVMs are useful in cases where you want to detect novel instances, the kind of which you have not seen before, and hence cannot characterize in advance. Binary classification can be used when the understanding is that the available labeled dataset covers the typical kind of examples you would see while using the model for prediction.


Invariance Property of Vowpal Wabbit Updates - Explaination

Often different data samples have different weighting ( eg the costs of misclassification error  for one group of data  is higher than for other classes).
Most error metrics are of the form $\sum_i e_i$ where e_i is the loss ( eg squared error) on data point $i$. Therefore weightings of the form $\sum_i w_i e_i$ are equivalent to duplicating the data w_i times (eg for w_i integer).

One simple case is if you have repeated data - rather than keeping all the duplicated data points, you just "weight" your one repeated sample by the number of instances.

Now whilst this is easy to do in a batch setting, it is hard in vowpal wabbits online big data setting: given that you have a large data set, you do not just want to represent the data n times to deal with the weighting ( because it increases your computational load). Similarly, just multiplying the gradient vector by the weighting - which is correct in batch gradient descent - will cause big problems for stochastic/online gradient descent: essentially you shoot off in one direction ( think of large integer weights) then you shoot off in the other - causing significant instability. SGD essentially relies on all the errors to be of roughly the same order ( so that the learning rate can be set appropriately). So what they propose is to ensure that the update for training sample x_i with weight n is equivalent to presenting training sample x_i n times consecutively.

The idea being that presenting it consecutively reduces the problem because the error gradient (for that single example $x_i$) reduces for each consecutive presentation and update (as you get closer & closer to the minimum for that specific example). In other words the consecutive updates provides a kind of feedback control.

To me it sounds like you would still have instabilities (you get to zero error on x_i, then you get to zero error on x_i+1,...). the learning rate will need to be adjusted to take into account the size of the weights.


What are the merges and vocab files used for in BERT-based models?

The vocab file contains a mapping from vocabulary strings and indices used for embedding lookup in the model.
The merges say how to split the input string into subword units. The algorithm is as follows: At the beginning of merging, a word split into characters and then you greedily search for neighboring symbols that can be merged (i.e., are in a list of allowed merges), you stop when there is nothing to merge and this is the subword segmentation.


What does it mean to take the "average" of two decision trees by 'voting'

I think that you are mixing together two different things - random forests
for regression and for classification. Regression means to predict a
continuous value (number). Random forest can construct multiple regression
trees, each of which makes a prediction about the number.  In that case,
it is simple to understand. The numerical predictions are averaged to give
a robust prediction of the true number value.

However, I think that you are asking about classification - predicting a
nominal value (also called categorical or factor). In this case, each
decision tree predicts a category.  Usually, it does not make sense to
talk about averaging categories. Instead, the multiple decision trees
"vote" - that is one counts how many times each category was predicted
and takes the category that received the most votes as the prediction.
There is no averaging, only counting.

Here is a simple example.

Data

V1   V2    V3    Class
A     C     E      X
A     C     F      X
B     C     F      Y
B     D     F      Y
B     D     E      X

Decision Tree 1 uses only feature V1:
If V1 = A, predict X, otherwise predict Y

Decision Tree 2 uses only feature V2:
If V2 = C, predict X, otherwise predict Y

Decision Tree 3 uses only feature V3:
If V3 = E, predict X, otherwise predict Y

Now we want to predict the class of a new point (A, C, F):
 - Decision Tree 1 sees V1 = A and predicts Class=X
 - Decision Tree 2 sees V2 = C and predicts Class=X
 - Decision Tree 3 sees V3 = F and predicts Class=Y
There were two votes for X and one vote for Y,  so the forest predicts X,
the class that received the  majority of the votes.


Machine learning with constraints on features

To be honest I didn't understand the problem very well, but it looks like genetic learning could be a good direction to study.
Essentially a genetic algorithm works by iteratively trying many combinations of values for some features, evaluating "something" based on the features, selecting the best combinations and starting again with a combination of the best features so far. The goal is to eventually converge to an optimal combination of values.


In the scikit-learn implementation of Spectral clustering and DBSCAN you do not need to precompute the distances, you should input the sample coordinates for all id_1 ... id_n.  Here is a simplification of the documented example comparison of clustering algorithms:

import numpy as np
from sklearn import cluster
from sklearn.preprocessing import StandardScaler

## Prepare the data
X = np.random.rand(1500, 2)
# When reading from a file of the form: `id_n coord_x coord_y`
# you will need this call instead:
# X = np.loadtxt('coords.csv', usecols=(1, 2))
X = StandardScaler().fit_transform(X)

## Instantiate the algorithms
spectral = cluster.SpectralClustering(n_clusters=2,
                                      eigen_solver='arpack',
                                      affinity="nearest_neighbors")
dbscan = cluster.DBSCAN(eps=.2)

## Use the algorithms
spectral_labels = spectral.fit_predict(X)
dbscan_labels = dbscan.fit_predict(X)


I don't think it is wise. Your intention to do validation on your real data is correct. But the way you have it now your model will be prevented from training on data that is from the same distribution as what you actually want to predict.

It is best to first split the images into training and validation sets, then do data augmentation on the training set.


Help with reusing glove word embedding pretrained model

I was stuck in a similar problem while working with glove. Assuming that you have a dataset in text form, from which you want to collect the topmost 100000 words, you'll have to make a list of those words. In the glove file, each embedding is on a separate line, with each line starting with the word itself and then the embedding. You'll have to write a code to compare your list of words with the words in glove file and extract the lines which make a hit. Have a look here for example code.


F1_score(average='micro') is equal to calculating accuracy for multiclasification

In classification tasks for which every test case is guaranteed to be assigned to exactly one class, micro-F is equivalent to accuracy.

The above answer is from:
https://stackoverflow.com/questions/37358496/is-f1-micro-the-same-as-accuracy

More detailed explanation:
https://simonhessner.de/why-are-precision-recall-and-f1-score-equal-when-using-micro-averaging-in-a-multi-class-problem/


Use both differencing and normalization in time series modeling to make it stationary?

Normalization does not stationarize a time series, as by definition, a non-stationary process has time-variant unconditional joint probability distributions - this implies that the mean and variance changes over time.

To see this for yourself, try computing the mean and standard deviation then normalizing the first half of the time series. Keep these estimates and transform the remaining half, and the resulting time series is still non-stationary.

Differencing $K$ times is a common method of stationarizing time series if it is integrated of order K, $I(K)$, and usually works in practice. However, note that stationarity implies $I(0)$ but the converse is not necessarily true.


Why is it necessary to overfit a yolov2 on a subset of images before model training?

As far as I know, overfit a small sample of training data doesn't help predict performance. Normally, overfit a batch of training data is a debugging method.

It's often very hard to detect logical errors in a neural network. One easy way to test is if our neural network can overfit a small batch of training data, we can reasonably assume that it has no critical bug and start training. The reason is that a neural network can even fit a random labeling of training data, if it cannot fit a small batch, it's almost guaranteed that there's a bug in your network.

Keep in mind that this is only one test to perform, there might be some other subtle bugs that prevent you from getting a good result.


Estimating and/or determining the amount of data sufficient enough to train a model

There are a few different perspectives from which to determine the amount of data you need. Those include:

Project complexity: Each parameter that your model has to consider in order to perform its task increases the amount of data that it will need for training.
Training method: As your models is forced to understand a greater number of interlinking parameters, the resulting complexity forces a change in the way that it is trained.
Labeling needs: Depending on the task you’re doing, data points can be annotated in different ways.
Diversity of input: In cases where your model’s input won’t be highly controlled, more data will be necessary to help your model function in that unpredictable environment.

You could begin working on your model with the data you have, then add more data when you feel it becomes necessary. Ideally, this will give you a very clear picture of your data needs, and they'll likely fit into one of the above categories.

This article covers training data needs in-depth, so it might be helpful. It also includes examples of dataset sizes for a variety of language-based and NLP projects. Unfortunately, there's not a lot of consistency among projects; some of the projects (text language identification) used 3000 training and 1000 test samples, while others (sentiment analysis) used datasets with over 10,000 samples. The differences here can likely be attributed to availability of data (tweet data is much easier to collect than South African language texts, for example).


Why do we choose principal components based on maximum variance explained?

do we always have to choose principal components based on maximum
  variance explained?

Yes. "Maximum variance explained" is closely related to the main objective as follows.

Our main objective is: for a limited budget K dimensions, what information $\mbox{a}=(a_1,...,a_K)$ to keep from original data $\mbox{x}=(x_1,...,x_D)$ ($D \gg K$) in order to be able to reconstruct $\mbox{x}$ from $\mbox{a}$ as close as possible?

If we only allow rotation and scaling of original data, i.e. $a_k := \mbox{x}.\mbox{v}_k$ for unknown set of vectors $V_K=\{\mbox{v}_k|\mbox{v}_k \in \mathbb{R}^D, 1 \leq k \leq K\}$, and define the reconstruction error as
$$loss(\mbox{x},V_K):=\left \| \mbox{x}-\underbrace{\sum_{k=1}^{K}\overbrace{(\mbox{x}.\mbox{v}_k)}^{a_k}\mbox{v}_k}_{\hat{\mbox{x}}} \right \|^2,$$
the solution $V^*_K$ that minimizes this error is PCA. For first dimension, PCA keeps the projection of data on vector $\mbox{v}^*_1$ in the direction of largest data variance, namely $a^*_1$. For second dimension, it keeps the projection on vector $\mbox{v}^*_2$ in the direction of second largest data variance, namely $a^*_2$, and so on.

In other words, when we try to find a K-vector set $V_K$ that minimizes $loss(X,V_K)=\frac{1}{N}\sum_{n=1}^{N}loss(\mbox{x}_n,V_K)$, the solution
 $V^*_K$ includes $\mbox{v}^*_k$ that is in the direction of $k\mbox{-th}$ largest data variance.

Note that "ratio of variance explained" is a measure from statistics. Using the previous notations, it is defined as:

$$\mbox{R}(X,V_K):=1 - \frac{loss(X,V_K)}{Var(X)}$$

Since variance of original data $Var(X)$ is independent of solution, minimum of $loss(X,V_K)$ is equivalent to maximum of $\mbox{R}(X,V_K)$. For example, if $K=2$, then $V^*_2=\{\mbox{v}^*_1, \mbox{v}^*_2\}$ minimizes $loss(X,V_2)$ and equivalently maximizes $\mbox{R}(X,V_2)$. Ideally, if original data $X$ can be perfectly reconstructed from $V_K$, then $R(X, V_K)$ would be $1$.

  Does maximum variance means most information about my data in higher
  dimension is captured into lower dimension?

Yes. If we agree that "keep as much information as possible" is equivalent to "be able to reconstruct the data as close as possible", then our objective $min_{V_K}loss(X,V_K)$ formalizes "keep as much information as possible",  and its solution is "maximum variance".


Calculate weighted mean for two columns and hundreds of rows?

Given your data is in DataFrame called df, you can simply do this:

mu = sum(df$Probability * df$SubjectA * df$salary)


How to implement custom loss function that has more parameters with XGBClassifier in scikit-learn?

def focal_loss(alpha, gamma):
    def custom_loss(y_pred, y_true):
        a,g = alpha, gamma
        def fl(x,t):
            p = 1/(1+np.exp(-x))
            return -( a*t + (1-a)*(1-t) ) * (( 1 - ( t*p + (1-t)*(1-p)) )**g) * ( t*np.log(p)+(1-t)*np.log(1-p) )
        partial_fl = lambda x: fl(x, y_true)
        grad = derivative(partial_fl, y_pred, n=1, dx=1e-6)
        hess = derivative(partial_fl, y_pred, n=2, dx=1e-6)
        return grad, hess
    return custom_loss

xgb = xgb.XGBClassifier(objective=focal_loss(alpha=0.25, gamma=1))

Using Python Closures!!


to generate consistent encoding for words in Keras using tf.keras.preprocessing.text.one_hot

Keras' one_hot function has many limitations. The biggest issue is that the function does not actually do one hot encoding, it does the hashing trick.
One possible fix is to use keras' hashing_trick function. It allows the hashing function to specified. If you pick a stable hashing function like md5, then the values will be consistent across runs.
Here is an example:
from tensorflow.keras.preprocessing.text import hashing_trick

corpus = ['nice app']
text_hashed = [hashing_trick(text=words, n=10_000, hash_function='md5') for words in corpus]
assert text_hashed == [[9146, 6067]]


B.Tech Project for final year of College

Do machine learning on microcontrollers or other edge devices (link1, link2, link3, link4, link5). This is a hot topic, new emerging technologies, and it will make your profs happy from both electronics and IT departments.

There was a recent news article (link) where a team from the Fraunhofer IMC did a handwritten digits recognition on an Arduino Uno. They claim that they not only run a trained neural network (which would be easy) but actually train the network on Arduino.

However, they don't show their code. On their license page (link) they only say some blah-blah-blah about discussing possibilities with partners and customers and boast 30 years of experience in the development of microelectronic circuits. This is lame. I am sure that this can be done by an undergraduate student without 30 years of experience and that student will write a blog post about it.

If you develop a machine learning code that would do such tasks on an Arduino or even something smaller, like ATTINY85 that would be cool. Your model should be able to be trained on a microcontroller given the restrictions of its memory and computational capacity. Then upload your code on a github or a blog, put your link here. If your code is good, it will get lots of likes, you will showcase your abilities to your employers, and will put those Fraunhofer dinos who hide their "great discoveries" to shame. Besides, it is very useful because of emerging technologies with IoT, 5G, etc.

By the way, it does not have to be a neural network. Other models, like SVM, KRR would also be great. Also, it does not have to be Arduino. Other microcontrollers, like PIC or ST are also OK.


How to do batch inner product in Tensorflow?

There is no native .dot_product method. However, a dot product between two vectors is just element-wise multiply summed, so the following example works:

import tensorflow as tf

# Arbitrarity, we'll use placeholders and allow batch size to vary,
# but fix vector dimensions.
# You can change this as you see fit
a = tf.placeholder(tf.float32, shape=(None, 3))
b = tf.placeholder(tf.float32, shape=(None, 3))

c = tf.reduce_sum( tf.multiply( a, b ), 1, keep_dims=True )

with tf.Session() as session:
    print( c.eval(
        feed_dict={ a: [[1,2,3],[4,5,6]], b: [[2,3,4],[5,6,7]] }
    ) )

The output is:

[[ 20.]
 [ 92.]]


Research in high-dimensional statistics vs. machine learning?

This makes me think right off the bat of Gaussian processes.

The GP was initially developed by geostatisticians in the 1970s as a model for the distribution of resources over a geographic area; this process is known as kriging (or also Gaussian process regression). But more recently GPs have been used pretty widely in the movement/control/robotics area to model movements and trajectories of robotic limbs etc. over time.

Statistically, GPs are treated as priors over vectors of infinite dimension, such that the distribution of any subset of the values of a vector is Gaussian distributed. There's an online book if you want to read more!


Reinforcement Learning - How are these state values in MRP calculated?

As it is such a small MRP, it is possible to solve it quickly and analytically using simultaneous equations based on the Bellman equation:

$$v(s) = \sum_{r,s'} p(r,s'|s)(r + v(s'))$$

and substituting in each state in turn. Using the variables $a,b,c,d,e$ to represent $v(A), v(B), v(C), v(D), v(E)$ makes this easier to read:

$a = \frac{1}{2}(0 + b)$
$b = \frac{1}{2}(a+c)$
$c = \frac{1}{2}(b+d)$
$d = \frac{1}{2}(c+e)$
$e = \frac{1}{2}(d+1)$

It takes multiple substitutions to resolve any individual value, at least to start with. But it is possible to resolve these terms by repeated substitution of how many times $a$ they are, and then proceeding to the next equation getting each variable:

$a = \frac{1}{2}(0 + b) \therefore b = 2a$

$b = \frac{1}{2}(a+c) \rightarrow 2a = \frac{1}{2}(a + c) \therefore c = 3a$

$c = \frac{1}{2}(b+d) \rightarrow 3a = \frac{1}{2}(2a + d) \therefore d = 4a$

$d = \frac{1}{2}(c+e) \rightarrow 4a = \frac{1}{2}(3a + e) \therefore e = 5a$

$e = \frac{1}{2}(d+1) \rightarrow 5a = \frac{1}{2}(4a+1) \therefore a = \frac{1}{6}$

That last one resolves everything, and gives the expected answer. You can also see a very strong pattern here, which should intuitively hold for any length of one dimensional random walk with equal $p=0.5$ - there is a proof for that more general case that would void the need to work the substitutions in detail for each length, but that is more complex.

One other simple intuition: Each state value in turn is the mean of the two state values either side of it, without exception. That means taking any three values next to each other, they should be co-linear; the two higher and lower values with the mean between them. As this holds on all values with overlap, plotting the values in order should place all points all on the same line. That in turn means that there is a simple linear relationship between position in the random walk and the state value.


This is a classic case of recommendation problem.
There are a couple of steps in which a recommendation is made:

Candidate Generation - What to show to a customer?
Candidate Ranking - How to show the items to the customer?
Personalisation - How to show the items to each customer?

Step 1 and 2 are general that means they talk about the overall trend and customer base, meaning we perform these steps by saying in general what would happen.

Step 3 is more personalised, where the ranking (order) in which the items are shown can be different for different customers.

For starters you should experiment with Singular Value Decomposition and Matrix Factorisation for recommendation.

If you have already tried your hand at these, then you should go with word2vec or doc2vec for building the step 1 and 2 and some kind of neural network at the end that ties to it in the end.


Multi layer back propagation Neural network for classification

Suppose that you need to classify something in K classes, where K > 2. In this case the most often setup I use is one hot encoding. You will have K output columns, and in the training set you will set all values to 0, except the one which has the category index, which could have value 1. Thus, for each training data set instance you will have all outputs with values 0 or 1, all outputs sum to 1 for each instance.

This looks like a probability, which reminds me of a technique used often to connect some outputs which are modeled as probability. This is called softmax function, more details on Wikipedia. This will allow you to put some constraints on the output values (it is basically a logistic function generalization) so that the output values will be modeled as probabilities.

Finally, with or without softmax you can use the output as a discriminant function to select the proper category.

Another final thought would be to avoid to encode you variables in a connected way. For example you can have the binary representation of the category index. This would induce to the learner an artificial connection between some outputs which are arbitrary. The one hot encoding has the advantage that is neutral to how labels are indexed.


Location Invariance not achieved in CNN in spite of 99% test accuracy

As far as I know, the usp of CNN is the fact that location of a pattern doesnt matter.

This is not true. Some options used in CNN architecture, such as max pooling, or strided convolutions, can add a moderate amount of translation invariance. However this will not cover larger translations - anything that is a significant percentage of the image width/height.

CNNs also support translation equivariance, which is where image textures and motifs that appear repeatedly but in different locations (lines, corners, curves) are learned efficiently and appear in the feature maps. This is probably closer to the USP of CNNs, that they learn the "representation language" of a signal, such as photograph, where it is consistent across the dimensions that are being convolved.

The cause of your problem is therefore very likely that your training set does not include enough images with the more central translation, or enough variations with the separate images.

To cover large translations you could look at one or more of:

Image pre-processing. In your supplied examples it looks very feasible to centre the digits.
Data augmentation. Generate translated versions of your training data to cover expected range of input translations in test.
Network architecture. A RNN or RNN/CNN hybrid could probably consume the images using smaller overlapped tiles of the images in sequence, and be trained to output the captcha string at the end. You might also be able to do this without a RNN, but that would require labelling each tile for the separate digits.


What are the advantages of HDF compared to alternative formats?

Perhaps a good way to paraphrase the question is, what are the advantages compared to alternative formats?

The main alternatives are, I think: a database, text files, or another packed/binary format.

The database options to consider are probably a columnar store or NoSQL, or for small self-contained datasets SQLite.  The main advantage of the database is the ability to work with data much larger than memory, to have random or indexed access, and to add/append/modify data quickly.  The main *dis*advantage is that it is much slower than HDF, for problems in which the entire dataset needs to be read in and processed.  Another disadvantage is that, with the exception of embedded-style databases like SQLite, a database is a system (requiring admnistration, setup, maintenance, etc) rather than a simple self-contained data store.

The text file format options are XML/JSON/CSV.  They are cross-platform/language/toolkit, and are a good archival format due to the ability to be self-describing (or obvious :).  If uncompressed, they are huge (10x-100x HDF), but if compressed, they can be fairly space-efficient (compressed XML is about the same as HDF).  The main disadvantage here is again speed: parsing text is much, much slower than HDF.

The other binary formats (npy/npz numpy files, blz blaze files, protocol buffers, Avro, ...) have very similar properties to HDF, except they are less widely supported (may be limited to just one platform: numpy) and may have specific other limitations.  They typically do not offer a compelling advantage.

HDF is a good complement to databases, it may make sense to run a query to produce a roughly memory-sized dataset and then cache it in HDF if the same data would be used more than once.  If you have a dataset which is fixed, and usually processed as a whole, storing it as a collection of appropriately sized HDF files is not a bad option.  If you have a dataset which is updated often, staging some of it as HDF files periodically might still be helpful.

To summarize, HDF is a good format for data which is read (or written) typically as a whole; it is the lingua franca or common/preferred interchange format for many applications due to wide support and compatibility, decent as an archival format, and very fast.

P.S. To give this some practical context, my most recent experience comparing HDF to alternatives, a certain small (much less than memory-sized) dataset took 2 seconds to read as HDF (and most of this is probably overhead from Pandas); ~1 minute to read from JSON; and 1 hour to write to database.  Certainly the database write could be sped up, but you'd better have a good DBA!  This is how it works out of the box.


Algorithmically extract seasonality in time series data

You can try FBProphet for your timeseries analysis. You will be able to check the seasonality, stationarity on your data.


How to conclude the generality of any classification methods?

You're experiencing an unfortunately common issue with the current state of system/model evaluation. In addition to evaluating on different datasets, authors often leave out important details, such as the procedure for hyperparameter tuning, detailed evaluation metrics (i.e. true positives, false negatives, etc. in addition to F-score), and ablation analyses. In cases like these, we cannot conclude that one method is necessarily better than the others or state-of-the-art.
The best way to estimate the generality of each method when the literature has so far failed to do so is to implement each yourself and do a fair comparative evaluation. You would evaluate all methods on the same dataset with the same pre-processing steps and hyperparameter tuning procedure and, if possible, introduce additional evaluation datasets. It can also be very enlightening to perform an ablation analysis in which you iteratively remove certain components of the methods and re-evaluate to see how much of a performance hit you take.
Doing the above and communicating it (via a publication, blog post, or whatever) will not only help you, but everyone else working in the area.


Data Mining Gear/Goods Websites for Specific Prices

I would look into the Python package BeautifulSoup. It parses HTML documents into a tree structure and allows for all kind of filtering and manipulation of the tree. I've taken a look at the Gear Patrol website and I would advice these steps:

Scrape the front page looking for articles. By taking a look using a web inspector like the one in Chrome (Ctrl+Shift+I) you can look at the HTML structure. Looking there we can see that the links to the articles are in:

body > div:content > div:home-this-week > div:posts hfeed > article > div:body-wrap

Now you look for all article tags and create a list of the corresponding URLs. You can use the link of the pagination to retrieve more article links or use their URL structure to make an educated guess where more articles are.

Now that we have a list of URLs, iterate over them and parse these pages in BeautifulSoup to retrieve the necessary information in a similar fashion. I advice taking breaks of a few seconds in between requests for new pages to not get banned from their server and not overly disrupt it. There are a lot of tutorials available how to do webscraping using BeautifulSoup and this site looks like an easy victim.


What is the differences between normal equation and gradient descent for polynomial regression

You want to solve $X \times \theta = Y$. Actually, you have to find the parameters $\theta$. To find it, you should multiply both sides by $X^{-1}$ but it may have not an inverse. Consequently, you multiply each side by $X^t$ due to the fact that $X^tX$ has the inverse. After that, you multiply each side by $(X^tX)^{-1}$ which leads to a closed form equation for finding $\theta$. It should be mentioned that $X^tX$ does not always have an inverse. If you want to be sure it has it, you have to exploit data samples which are more than the number of features and they should not be linearly dependent otherwise the constructed matrix will not inverse and you will not be able to use it.


Grid search model isn't recognized as fitted for Graphviz

Try using Tree_opt.best_estimator_.  The grid search object passes fit and score functions through to the best estimator when refit=True, but other (esp. more specific) methods need to be called directly from best estimator object.


What exactly is the linear layer in the transformer model?

In the original Transformer article, these linear layers are just matrix multiplications.
As described in the paragraph you referred to in your question $W^Q$ is a matrix of dimensions $d_{model} \times d_k$, that is, it is a fully connected layer with $d_k$ units.
In practical implementations, these have the optional addition of a bias vector. You can see their actual definition in the fairseq code.


Are Intersection over Union (IoU) scores preferring larger objects?

There are a lot of reasons for this behaviour:

training data: In your training data, there could be more larger objects than smaller objects, so the model didn't have enough information about smaller objects to learn to segment them well.
training procedure: Depending on your training procedure (e.g. loss function), your model will focus more on some type of objects (in your case, larger objects ...)
model architecture: Every model has some advantages, so some models are better in recognizing objects of various sizes. E.g.  convolutional and pooling layers can increase a receptive field of neurons so that they could recognize objects of various sizes more easily. Also, some model architectures are specialized in recognizing models of varying sizes.

Possible directions:

training data: Know your data! :) Inspect the dataset to see what's in. The things to look for: are all classes represented well, are labels (ground truth) good, sizes of objects, ...
training procedure: Try with various loss functions (cross entropy, dice loss, cross entropy + dice loss, focal loss, ...). Also, dataset augmentation techniques are important (e.g. with dataset augmentation you can create objects of various sizes ...).
model architecture: Resnet, Inception, InceptionResnet - those are the backbones that you might want to use for your backbone as they have proved to be good for semantic segmentation.


Additive vs Multiplicative model in Time Series Data

I want to know which model between additive and multiplicative best suits the above data.

It is hard to tell just by looking at it.

A multiplicative decomposition roughly corresponds to an additive decomposition of the logarithms.
The additive decomposition is the most appropriate if the magnitude of the seasonal fluctuations, or the variation around the trend-cycle, does not vary with the level of the time series. When the variation in the seasonal pattern, or the variation around the trend-cycle, appears to be proportional to the level of the time series, then a multiplicative decomposition is more appropriate. Multiplicative decompositions are common with economic time series.

An alternative to using a multiplicative decomposition is to first transform the data until the variation in the series appears to be stable over time, then use an additive decomposition. So, basically you need to check for heteroskedasticity, eliminate that if it is there by transformations and do an additive decomposition of the transformed series.

Most common transformations are log or square root of the series and are special cases of Power transform.

Reference:
Forecasting principles and practice


Clustering based on distance between points

I think for HAC (Hierachical Aglomeritive Clustering) it's always helpful to obtain the linkage matrix first which can give you some insight on how the clusters are formed iteratively. Besides that scipy also provides a dendrogram method for you to visualize the cluster formation, which can help you avoid treating the clustering process as a "black box".

import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import dendrogram, linkage

# generate the linkage matrix
X = locations_in_RI[['Latitude', 'Longitude']].values
Z = linkage(X,
            method='complete',  # dissimilarity metric: max distance across all pairs of
                                # records between two clusters
            metric='euclidean'
    )                           # you can peek into the Z matrix to see how clusters are
                                # merged at each iteration of the algorithm

# calculate full dendrogram and visualize it
plt.figure(figsize=(30, 10))
dendrogram(Z)
plt.show()

# retrive clusters with `max_d`
from scipy.cluster.hierarchy import fcluster
max_d = 25       # I assume that your `Latitude` and `Longitude` columns are both in
                 # units of miles
clusters = fcluster(Z, max_d, criterion='distance')

The clusters is an array of cluster ids, which is what you want.

There is a very helpful (yet kinda long) post on HAC worth reading.


Stratified selection based on the y response creates a bias in information (Berkson's bias)?

It would be better to frame your problem as survival analysis, rather than classification. The goal of survival analysis is predict the duration until an event happens.


Optimizing vector values for maximum correlation

This is one approach you can follow:

Setup a linear regression system, with each match as a row, and each feature corresponding to a team.
The unknown feature coefficient for each team is the team 'strength' that we try to determine.
The feature values will be one of 0, 1, or -1, depending on whether the team did not play, was the column team, or the row team respectively in that match.
The regression target will be the score differential (column team score - row team score) in that match.

Eg: For the result matrix above, the system would be:

$$
(-1) * x_1 + (1) * x_2 + (0) * x_3 = 3\\
(-1) * x_1 + (0) * x_2 + (1) * x_3 = 6\\
(0) * x_1 + (0-1) * x_2 + (1) * x_3 = 3
$$

One solution to the above system is: $x_1=-6; x_2=-3; x_3=0$
Multiple solutions are possible, which can be viewed as translations of each other (adding a constant to all team strengths).
If there are 'n' teams, then there are only 'n-1' linearly independent columns in the regression. (In R, one of the coefficients comes out as NA. This can be treated as 0, or dropped from the regression, effectively making it 0).


You can use plotly in python to get this done.

import plotly.graph_objects as go
import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([1, 3, 2, 3, 1])

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, name="linear",
                    line_shape='linear'))
fig.add_trace(go.Scatter(x=x, y=y + 5, name="spline",
                    text=["tweak line smoothness<br>with 'smoothing' in line object"],
                    hoverinfo='text+name',
                    line_shape='spline'))
fig.add_trace(go.Scatter(x=x, y=y + 10, name="vhv",
                    line_shape='vhv'))
fig.add_trace(go.Scatter(x=x, y=y + 15, name="hvh",
                    line_shape='hvh'))
fig.add_trace(go.Scatter(x=x, y=y + 20, name="vh",
                    line_shape='vh'))
fig.add_trace(go.Scatter(x=x, y=y + 25, name="hv",
                    line_shape='hv'))

fig.update_traces(hoverinfo='text+name', mode='lines+markers')
fig.update_layout(legend=dict(y=0.5, traceorder='reversed', font_size=16))

fig.show()

https://plot.ly/python/line-charts/


PCA for dimensionality reduction with simultaneous clustering

Your task is achieved by Subspace Clustering


# of iterations in Restricted Boltzmann Machine (RBM)

When you are unsure about how something (e.g. RBMs) should really be implemented, it is often useful to look at the code of others. G. Hinton himself has published a MATLAB-script (here) which demonstrates the training of an RBM. There, you can see that for each mini-batch, he does the positive phase, then the negative phase, and finally updates the weights - and that's it. So he doesn't iterate between the visible and hidden states.

However, this is not the full truth: for the weight updates we need to know the probability $p(v,h)$. This is very complicated to calculate, as it would contain a sum over all possible states of the RBM. There is a "mathematical trick" called Gibbs sampling: it allows us to iterate back and forth between visible and hidden units to calculate this probability $p(v,h)$.

But: for the result to be correct, we have to iterate forever, which is not really practical. So what Hinton proposed is to iterate for only 1 step instead (this is $CD_1$), so he only goes back-and-forth once.

But, you can also iterate any number of times $k$, which is denoted by $CD_k$.
While for Hinton's $CD_1$, you would do

visible --> hidden --> visible

For a $CD_3$, you would iterate from visible to hidden and back, three times:

visible --> hidden --> visible --> hidden --> visible --> hidden --> visible

And just to be clear: you run this iteration for every data point in the training set. Actually, you usually make so-called mini-batches of maybe 10 data points, which you run at the same time. Then you calculate the average weight update from this batch. But then, you do this iteration again, and again, and again, until you have finished your training.


What tools are available for programming language parsing for ML?

NLP stands for Natural Language Processing. Programming language source code are synthetic (or unnatural) languages. Thus, NLP tools are not useful for processing programming language source code.

Understanding programming language source code is done by the compiler or interpreter. Compilers and interpreters perform many functions, including lexical analysis, parsing, and semantic analysis.

For Python language, language analysis is relatively straightforward since the main Python implementation, named CPython, compiles Python programs into intermediate bytecode, which is executed by the virtual machine. The dis module supports the analysis of CPython bytecode by disassembling it.


NLP: find the best preposition for connecting parts of a sentence

The traditional approach for this kind of problem would be an n-gram language model. The language model is trained on a large corpus, then it's reasonably simple to calculate the most likely missing tokens for any incomplete sentence. SRILM was one of the most common toolkits, but there are probably many other libraries.


How to calculate a specific time complexity of inverse calculation of matrix?

If you need complexity of this calculation in big O notation - it is: $$O(n^3)$$
Why? Because matrix inverse needs $$O(n^3)$$ operations, and it is biggest complexity here.

Multiplication matrix by its transpose is $$O(n^2p)$$ (Because for computing every value in the resulting matrix of size NxN you have to compute p multiplications).
Matrix transpose is $$O(np)$$ But you can ignore any complexities lesser than $$O(n^3)$$ (like matrix sum, transpose, or multiplying by constant) because their order of growth much lesser than order of growth of inverse operation.

Computational_complexity_of_mathematical_operations


bagging vs. pasting in ensemble learning

Actually I think that you are mostly correct, except that in my understanding "with/without replacement" applies only to selecting one subset, not across subsets. This means that if we have a training set of instances $T=\{t_1,..,t_N\}$:

With bagging, a particular sample can contain duplicate instances, in other words it is not a subset of $T$ but a multiset where an instance $t_i$ may occur several times. Of course the probability to pick the same instance $t_i$ $n$ times decreases quickly when $n$ increases, depending on the size $N$.
With pasting a sample is a subset of $T$ and cannot contain the same instance $t_i$ twice. However another sample is drawn again from the full set $T$, which means that it is possible for an instance to be selected in several different samples.

In theory, the size of a sample can be higher than $N$ with bagging but not with pasting.
Note that I'm referring to multiset for the sake of clarity but formally $T$ itself is not a set since in theory it may contain the same instance twice.


Clustering Crime Data which has {latitute, longitude, crime-type} tuples

There is no meaningful way to combine "type" with a distance in meters.

My suggestion is that you:

split the data set by type.
cluster each type, with DBSCAN, haversine distance, and the same minpts/eps values for each crime type
compare the resulting clusters for similarity and differences


What is the range of values of the expected percentile ranking?

What is $t$?

It means observed $r_{ui}$ in the one-week test set (page 6-left).

  Is my understanding of the metric correct?

First two examples are correct. Assuming user-item relation $r_{ui}^t$ is constant $a$ for all items in the test set, and predicted ranks are uniform across $[0, 1]$, then, the third one would be:

$$\overline{\text{rank}} = \frac{\sum_{u,i} r^t_{ui} \text{rank}_{ui}}{\sum_{u,i} r^t_{ui}}=\frac{\sum_{u,i} a \text{ rank}_{ui}}{\sum_{u,i} a}=\frac{1}{|I|}\sum_{u,i} \text{ rank}_{ui}=\frac{1}{|I|}\frac{|I|}{2}=\frac{1}{2}$$
This makes sense. Items are identical to the user, therefore no model can do better than random guessing, since there is no observed preference to help the model favor one item over the other. Of course, another assumption here is that training (4 weeks) and test (next week) sets are from the same distribution.


Recognition human in images through HOG descriptor and SVM classifier performs poorly

You are using the training set that opencv is giving you which it doesn't correspond to the kind of images you are using. The data you are using comes from getDefaultPeopleDetector and the kind of images that the default detector uses are pictures of many people, not a female model from a fashion ecommerce.

If you want to distinguish between models and garments you can try to train your own classifier with HOG or other features.

Another path you can take is to detect if there is a face or not. You could use haar cascades for that.


Difference between training and test data distributions

A good way to measure the difference between two probabilistic distributions is Kullbak-Liebler. You have to take into account that the distribution has integrate to one. Also you have to take into account that it's not a distance because it's not symmetric. KL(A,B) not equal to KL(B,A)


Azure Data Studio opens Jupyter Notebook / Lab HTML

Changing the file ~/.config/mimeapps.list with settings to
text/html=firefox.desktop
fixed the issue.
This issue is related to this one where other solutions are also provided (set a default browser for Jupyter).


Merging Training and Validation Sets for better accuracy

After you have finished with the model building process (in which it is assumed that you have used your test set once and only once for assessing the performance of your final model on unseen data), and before deploying your model, both common sense and standard practice say that you should re-train it on all the available data, including the portion that, until then, had been put aside as test. Leaving out available data is a luxury which normally we cannot afford; and, provided that there are no issues with your model building process, and your test set is qualitatively similar to your training one (an assumption implicitly always present), there is nothing to worry about.
Qualitatively speaking, this approach is similar with what we do with cross validation, where afterwards we routinely re-train the model using all the available data.
The following Cross Validated threads might be useful; although they address the cross-validation issue, the rationale is similar -  at the end, use all the data:

Training on the full dataset after cross-validation?
How to choose a predictive model after k-fold cross-validation?


You can run 2 xgboost binary classifiers

1 classifier  classifies if sample is (good or very good)
2 classifier classfies if sample is very good
if both true on unseen data classify as very good
if only 1st one true, second false  classify as good
 both false=> classify as bad


The Udacity Data Analyst Nanodegree gives a very gentle introduction using Python and R for mostly exploratory Data Analysis which I think is what you are looking for. (It's not tailored towards business analysis. The courses are free and you can just take the ones you think are interesting. In your case, I would skip the ones about data visualization with javascript and data wrangling with MongoDB.) THey provide a bunch of resources to go from there. Of course there are plenty of other online courses.

If you are more into books you could check out Tukey's Exploratory Data Analysis. A classic that is still highly relevant.

In terms of tools, the most commonly used are R and Python and I would focus on them for now.


In Orange, is it possible to analyze or visualize real time, streaming data?

"Realtime" is a very big word, in many regards you would only call a system realtime when there is very little latency in a system, almost no latency. It would require specialized hardware and os, for a discussion of python and realtime see https://stackoverflow.com/questions/7079864/real-time-operating-via-python.

If you describe the requirement as "continously (or frequently) updated", it looks possible to me to do some mining on streamed data with orange. I am adding my 5 ct, as I have just started looking into orange.

You would need to create a widget that opens your input stream and propagates it in chunks to the subsequent widgets in the flow, i.e. when you want to build something like a "tail -f" on a file source, reads character by character until there is a linefeed in the stream. In that case it would release a signal with the line as data to the output.

Orange widgets are receiving their inputs as signals, "widgets receive inputs at runtime with the designated handler method (specified in the OWWidget.inputs class member)." This next widget would update on that signal and propagate on its own outputs. So the continued live analysis propagation of streaming data through orange widgets seems in general possible to me.

How close to or far from real time this would be depends on many factors like load, computing power, and memory. The choice of mining algorithms might also be limited by such factors.


using simple autoencoder for feature selection

Autoencoders normally aren't linear models. If you make them linear (i.e. you create a shallow Autoencoder with linear activations) then you get exactly a PCA result. The power of Neural Networks is their non-linearity, if you want to stick with linearity go for PCA imho.

Keep a Train-Validation-Test set split, and try different configurations of hyperparams checking their performance on Validation data. Alternatively there are many libraries, such as hyperopt, that let you implement more sophisticated Bayesian hyperparameter searches, but unless you want to be published at a conference or win some competition it's a bit overkill. If you're still interested, the internet is plenty of tutorials like this one.


Generally, you will find datasets being distributed in CSV format for their simplicity and human readable format that you could ingest in any programming language with just the packages that the language is shipped with. Usually, tabular data is exported in CSV format and that is one of the reasons why MNIST dataset is not provided in CSV format.

Here is the quote from LeCun's website for storing the dataset in idx format.

  The data is stored in a very simple file format designed for storing
  vectors and multidimensional matrices.

In terms of performance, binary file formats fairs better compared to text file formats like CSV or rich text format like ODS.

Following are some of the binary file formats that are widely used.

Avro format
Parquet format
Optimized Row Columnar (ORC)
Protocol Buffers (protobuf)

These file formats support data compression, stores data type metadata to serialize and de-serialize data effeciently.


How to analyze which site has most numbers

The most obvious way of visualizing this is to have the number of computers on the Y-axis and the size of the dots representing the percentages. The categories (or sites in your case) can be represented on the X-axis. The image below shows an example where the Y-axis represents a continuous value (can be mapped to number of computers in your case), the X-axis represents a discrete value (can be mapped to sites in your case), and the size of the dots represents another attribute (like percentage in your case). I have used the R package ggplot2 for this.


You have to tell Corpus what kind of source you are using.  Try:

Corpus(VectorSource(d1$Yes))


Where to find statistically relevant documentation of common Python packages?

I think the documentation, at least for your example, is not too obscure. It doesn't tell you what kind of interpolation it uses because it doesn't use any: resample is a deferred operation. In order for it to work you have to call it in conjunction with a function that performs the interpolation, e.g. series.resample(frequency).mean() or series.resample(frequency).interpolation('cubic') (as can be seen from the examples in the documentation). Sometimes, it can be helpful to also look into the release notes for additional information.

Scipy and Numpy are sometimes not very detailed, but they contain at least a decent number of references with more information. (There is also a section on Matlab versus Numpy here. If you are familiar with dplyr in R , then the pandas cheat sheet can help as a quick comparison between R and pandas for typical data wrangling operations.)

I'm not aware of a "second more detailed documentation" that encompasses the entire libraries. You probably have to fall back to books (which, of course, aren't complete or always up to date).


What happens if GBM parameters (e.g., learning rate) vary as the training progresses?

Learning rate decay is implemented in, e.g., XGBoost and LightGBM as callbacks.  (XGBoost used to allow the learning_rate parameter to be a list, but that was deprecated in favor of callbacks.)  Similar functionality for other hyperparameters should be possible in the same way.
xgboost.callback.reset_learning_rate
LightGBM callback discussion
I played around with these ideas (for learning rate and tree depth) a while back, but didn't get improved performance.  But you should try it out; if you do see significant gains, it'd be great to add it as an answer here.


Handling a regularly increasing feature set

In an ideal world, you retain all of your historical data, and do indeed run a new model with the new feature extracted retroactively from historical data. I'd argue that the computing resource spent on this is quite useful actually. Is it really a problem?

Yes, it's a widely accepted technique to build an ensemble of classifiers and combine their results. You can build a new model in parallel just on new features and average in its prediction. This should add value, but, you will never capture interaction between the new and old features this way, since they will never appear together in a classifier.


Tensorflow: how to look up and average a different amount of embedding vectors per training instance, with multiple training instances per minibatch?

Use tf.gather().

Single instance case

In the example below, we selected a variable number of embedding vectors from the matrix embedding. The selection indexing vector user can be of variable length. Then we calculate the average embedding.

with tf.Graph().as_default():
    embedding = tf.placeholder(shape=[10,3], dtype=tf.float32)
    user = tf.placeholder(shape=None, dtype=tf.int32)
    selected = tf.gather(embedding, user)
    average = tf.reduce_mean(selected, axis=0)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        embedding_ = np.random.randn(10,3)
        user_ = [1,3,5]
        print(sess.run(average, feed_dict={embedding:embedding_, user:user_}))
        print(np.mean(embedding_[user_], axis=0))

Multiple instances in a mini-batch

You may manually specify the first vector in embedding to be a zero vector, and patch the above selection vector with 0s. For example

with tf.Graph().as_default():
    embedding = tf.placeholder(shape=[10,3], dtype=tf.float32)
    user = tf.placeholder(shape=[None, None], dtype=tf.int32)
    selected = tf.gather(embedding, user)
    non_zero_count =  tf.cast(tf.count_nonzero(user, axis=1), tf.float32)
    embedding_sum = tf.reduce_sum(selected, axis=1)
    average = embedding_sum / tf.expand_dims(non_zero_count, axis=1)

    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        embedding_ = np.concatenate([np.zeros((1,3)),np.random.randn(9,3)], axis=0)
        user_ = [[3,5,7,0], [1,2,0,0]]
        print(sess.run(average, feed_dict={embedding:embedding_, user:user_}))
        print(np.sum([embedding_[i] for i in user_], axis=1) / np.atleast_2d(np.count_nonzero(user_, axis=1)).T)

You can use tf.gather() like this even if the embedding is a trainable variable instead of a placeholder.


Chi square distribution for feature selection

There are different ways for feature selection. A very good read in machinelearningmastery, to recap:

Univariate Selection.
Recursive Feature Elimination.
Principal Component Analysis.
Feature Importance.

Chi-Squared test For Feature Selection goes under the Univariate Selection method for non-negative features. My favorite explanation of chi-Squared in one photo taken from this blogpost is:

As you can see scikit-learn has an implementation for feature_selection using chi2 (perhaps according to scipy.stats.chisquare) as was shown very briefly in the above-mentioned blog post.

If you want a more thorough explanation and details how test ranks features based on statistics according to chi2 distribution and p-value etc., and also how to build your own chi2 class for feature selection in Python see this great post. Obviously one can read about the basics of chi2 distribution and test in wikipedia.


Predict Best Branch Locations For a Company

This is a broad question, what I offer is one of many possible approaches; I will use R workflow (sf package to be exact).

The first step is to generate a grid covering your area of interest - city, country, continent, whatever :) Consider the approach described in this answer on StackOverflow. The size of the grid cells will depend on your company product - the catchment area of a corner shop is different from a catchment area of a strip mall.

The next step is link your store locations to grid cells - sf::st_intersects will be your friend here, and aggregate your desired metric (sales volume, new customer acquisition, or what not) per grid cell.

The third step is to find as much demographic and economic data as possible for administrative regions of your country (counties, districts, census areas...) - this will be country specific, so I cannot be your guide you here - and again link them to your grid.

The last step is creating a regression for desired output in a grid cell depending on the economic and demographic indicators.

Once you have a formula you can apply it to all the empty grid cells and pick the 10 most promising.

Edit [2019-01-22]: I have extended the suggested workflow into a blog post, you can view it at
http://www.jla-data.net/eng/branch-network-optimization-in-r/


Is the activation function the only difference between logistic regression and perceptron?

TL;DR: Yes and No; they're both similar decision function models but there's more to each model than their main formulation

One could use the logit function as the activation function of a perceptron and consider the output a probability.  Yet, that value would likely need a probability calibration.

As with most ML models several things are very similar from model to model, on the other hand varying tiny parameters can result in a different model.  Let's take on both sides:

Logit Regression and Perceprton similarities

The logit function is used in logit regression for its properties of being an S-curve, by default valued between 0 and 1.  The sign activation function in the perceptron is also shaped like an "S-curve" (with very rough edges - so mathematically not an S-curve by its definition but with similar properties) valued "between" -1 and 1.  Another activation function often used with perceptron is hyperbolic tangent (tanh), which is another S-curve - very similar to the sign function but with a rounded shape (and also valued between -1 and 1).  We can say that tanh is similar to sign because:

$$
\texttt{sign}(x) \approx \texttt{tanh}(kx) \qquad \text{for k >> 0}\\
or\\
\texttt{sign}(x) = \texttt{tanh}(kx) \qquad k \to \infty
$$

So it makes sense to compare tanh with logit as an analogy to compare sign with logit.

Now the logit function is (the one used in logistic regression that is - not the statistical $ln(x/(1- x))$ one):

$$
\texttt{logit}(x) = \frac{L}{1 + e^{-k(x - b)}}
$$

Where $L$, $k$ and $b$ are parameters that we can steer. (Since $b$ is a bias term, an unknown constant, it does not matter if we write $- b$ or $b$)

And the hyperbolic tangent is:

$$
\texttt{tanh}(x) = \frac{e^{2x} - 1}{e^{2x} + 1}
$$

But wait, if we set the parameters of the logit as $L = 1$, $k = 1$ and $b = 0$, then:

$$
2 \cdot \texttt{logit}(2x) - 1 = 2 \cdot \frac{1}{1 + e^{-2x}} -1
= \frac{2e^{2x}}{e^{2x} + 1} - 1
= \frac{2e^{2x}}{e^{2x} + 1} - \frac{e^{2x} + 1}{e^{2x} + 1}
= \frac{2e^{2x} - e^{2x} - 1}{e^{2x} + 1}
= \frac{e^{2x} - 1}{e^{2x} + 1} = \texttt{tanh}(x)
$$

So tanh, which is more-or-less a rounded sign, is a scaled and shifted special case of logit.  Not only the model differ by the activation function alone but the activation functions are very similar to each other.

Logit Regression and Perceprton differences

Log probabilities (this is probably the major difference and the most important one)

We looked at the logit function above but in reality the logit regression takes the logarithm of the logit instead of plain values for the probabilities. Or, more exactly, in statistics the logit is defined as the natural logarithm of what in most ML implementations is defined as a logit decision function. This is very different from the perceptron which performs the output directly from the activation function.

Regularization

Both models (logit regression and perceptron) are often trained with some form of gradient descent optimizer (be it SGD, have it momentum, or even something else). This training will optimize all parameters ($w$ in your representation or $k$ in mine for logit and the weights in a perceptron) but the model itself will also be given a bunch of hyper-parameters.  And in terms of hyper-parameters things do differ:

The perceptron will be trained by optimizing the weights (including a bias) and can be regularized with $L_2$ or $L_1$ (or a combination of both). One may or may not add a bias term.

The actual model of the logistic regression will optimize the logarithm but there's more: most implementations will include a scaling hyper-parameter ($C$) which will multiply the log probabilities. One can use $C$ to regularize the model apart from $L_1$ and/or $L_2$.

Multi-class

The perceptron is always a binary classifier.  One sets an output threshold and that works as the decision function (for the sign or tanh functions the threshold is often $0$). For multi-class classification one must build One-Vs-Rest or One-Vs-One groups of models.

Logistic Regression can be used as a binary classifier and in this case can be used for multi-class classification with One-Vs-Rest and One-Vs_one methods. But, there exist a formulation of logistic regression for direct multi-class classification:

$$
\texttt{multinomial logit}(x|y) = \frac{e^{-k_y(x - b)}}{\sum^{K}_{k=0} e^{-c(x - b)}} \qquad \text{for y in }\{0,1,\dots\}
$$

i.e. whether $x$ belongs to class $y$. This is performed for each class and the result can be passed to a softmax function.  This multi-class formulation can be performed because logit regression deals with log probabilities instead of direct inputs (contrary to a perceptron).


How to study the effect of eps in sklearn.cluster.DBSCAN?

The meaning of $\epsilon$ is that of the neighbourhood size. The neighbourhood of a point $p$, denoted by $N_{\epsilon}(p)$, is defined as the $N_{\epsilon}(p) = \{q \in D | dist(p,q) \leq \epsilon \}$. Here $D$ is a database of $n$ objects (points) and $q$ a query point.
So what you Professor probably wants you to do is to evaluate goodness of clustering algorithm (DBSCAN) based on its input Parameters Epsilon. This goodness of fit you can measure for example with Silhouette score


Is there a ready-made classifier to determine if webpage is an article or not?

I'm going on the assumption that you "have a set of URLs" that you need to pre-filter prior to doing something else with.  Based on this assumption you could use urlparse and some string patterns to filter out X percentage of the URLs in your set that are most likely non-articles.
import re as regex
from urllib.parse import urlparse

urls =['https://candid.org/privacy-policy',
       'https://happify.com/health/privacy-policy',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://www.ndtv.com/rss']

patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement', 'rss-feeds',
            'newsfeeds', 'rss']
for url in urls:
    split_url = urlparse(url)
    possible_article = [pattern for pattern in patterns if regex.findall(pattern, split_url.path)]
    if not possible_article:
        print(f'Possible article: {url}')
        # output
        Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html

You could potentially expand the regular expression above to flag other URLs that meet common article URLs that can date strings or common keywords, such news or wirestory.
import re as regex
from urllib.parse import urlparse

urls =['https://www.ndtv.com/rss',
       'https://candid.org/privacy-policy',
       'https://www.sciencedaily.com/newsfeeds.htm',
       'https://happify.com/health/privacy-policy',
       'https://www.nasa.gov/content/nasa-rss-feeds',
       'https://www.bbc.com/news/technology-55675826',
       'https://www.abstract.com/legal/customer-terms-of-service',
       'https://pesa.org.au/membership/terms-of-service-and-privacy-statement',
       'https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html',
       'https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet',
       'https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed']

non_article_patterns = ['privacy-policy', 'customer-terms-of-service', 'terms-of-service-and-privacy-statement',
                        'rss-feeds', 'newsfeeds', 'rss']

known_article_patterns = ['\d{4}\/\d{2}\/\d{2}', 'news', 'wireStory']

for url in urls:
    split_url = urlparse(url)
    non_article = [pattern for pattern in non_article_patterns if regex.findall(pattern, split_url.path)]
    if non_article:
        pass
    else:
        possible_article = [pattern for pattern in known_article_patterns if regex.findall(pattern, split_url.path)]
        if possible_article:
            print(f'Possible article: {url}')
            Possible article: https://www.bbc.com/news/technology-55675826
            Possible article: https://www.cnn.com/2021/01/14/economy/unemployment-benefits-coronavirus/index.html
            Possible article: https://www.cnet.com/news/samsungs-galaxy-s21-upgrades-likely-wont-spell-an-end-to-galaxy-fe-or-note-lines-yet
            Possible article: https://abcnews.go.com/Politics/wireStory/trump-leave-washington-morning-bidens-inauguration-75278801?cid=clicksource_4380645_6_heads_hero_live_headlines_hed

BUT as Erwan pointing out in his comment it's not clear "how you define the difference between article and not article", so my answer is only part of a potential solution to your problem of URL classification.
It's hard to make any solid recommendation for any "out-of-the-box" solution that can meet an undefined use case.
Here are some links that might help you:

paper: Fast webpage classification using URL features
paper: A Comprehensive Study of Features and Algorithms for URL-Based
Topic Classification
paper: Web Page Classification: Features and Algorithms
commercial: Website Categorization API

I'm still of the mindset that your use case will likely require a multi-prong approach and will need to be refined as more URLs are classified as either "an article or non-article."


To extract the skills required for the job given the job description

Assuming you already have the raw text, you can do the followings.

Create train data:

You need to create set words and bigrams labeled as skills. There might be some available lists to help you out. Otherwise, generate your list using resources such as wordnet and thesaurus. You can also start with a short list based on the data you have and then expand it using word2vec or similar word embedding techniques. For example, we start with a list that contains coding as one of the skills. Then, we query the word2vec pretrained model for closest words/bigrams. There is a fair chance you will end up with programming, software coding, and computer programming.

Another approach would be to cluster the words and bigrams in your dataset using word embedding techniques. Then, look into your clusters to see which ones contain the skill set.

Note that any word/expression not on your list will not be considered as required skills. Therefore, you may need to expand your list after a few trials.

Detecting the Skills

Tokenize your raw text into words and expressions
Remove stop words
Encode your tokens using an embedding (Word2Vec, FastText, etc)
Use the list from the previous step to add labels to your data (anything on the list is True, other as False)
Train a binary classifier (Naive Bayes classifier should be good enough)
Evaluate your model, feature set, and labels. If needed, refine and repeat.


How to prefer no choice instead of bad choice with sklearn decision tree

TL;DR
You can train your classifier as usual and threshold on the prediction probability.
Scikit-learn
You won't get that out of the box, but you can use this simple class to do just that!
from sklearn.tree import DecisionTreeClassifier
from numpy import argmax, max

class MyClassifier(DecisionTreeClassifier):

    unknown_class = 'unknown'

    def __init__(self, no_class_threshold=0.75, **kwargs):
        self.no_class_threshold = no_class_threshold
        super().__init__(**kwargs)

    def predict(self, X):
        preds = self.predict_proba(X)
        y_pred = [self.classes_[i] if v > self.no_class_threshold else self.unknown_class
                  for i, v in zip(argmax(preds, axis=1), max(preds, axis=1))]
        return y_pred

Just use this class as you were using the DecisionTreeClassifier before. You can pass all the parameters from DecisionTreeClassifier plus an extra one: no_class_threshold.
The no_class_threshold works in the following way:
IF
    prediction_probability > no_class_threshold
THEN
    output predicted class
ELSE
    output 'unknown'

Changing classifier
You might not want to stay with the DecisionsTreeClassfier, and instead, you might want to experiment with other classifiers. You can do this by inheriting from another Scikit-learn class. Simple as that!
For example:
from sklearn.svm import SVC

class MyClassifier(SVC):
    [...]

Choosing the no_class_threshold
You can:

Choose it manually, to a degree that you feel confident
Use hyper-parameter tuning to learn it, if you have data points labelled as 'unknown'.

Example
Here is a script you can run to test all the above.
from numpy import argmax, max
from sklearn.svm import SVC

class MyClassifier(SVC):
    unknown_class = 'unknown'

    def __init__(self, no_class_threshold=0.75, **kwargs):
        self.no_class_threshold = no_class_threshold
        super().__init__(**kwargs)

    def predict(self, X):
        preds = self.predict_proba(X)
        y_pred = [self.classes_[i] if v > self.no_class_threshold else self.unknown_class
                  for i, v in zip(argmax(preds, axis=1), max(preds, axis=1))]
        return y_pred

if __name__ == '__main__':
    X = [[1, 1, 1],
         [1, 0, 1],
         [0, 0, 0],
         [0, 1, 0],
         [0, 1, 1]]
    y = ['class1', 'class1', 'class0', 'class0', 'class0']

    clf = MyClassifier(no_class_threshold=0.55, probability=True, C=1)

    clf.fit(X, y)

    X2 = [[1, 1, 0],
          [0, 1, 1],
          [0, 0, 1]]
    pred = clf.predict(X2)
    print(pred)


predicted value is more conentrated than actual with DNN

It seems like your model hasn't learnt much and may almost always be predicting the mean of the target distribution. Please check if the mean of the actual target variable resides in the 1260-90 range.
This may happen if your features are not correlated to the target.
Also, by looking at your graph, the MAE is certainly larger than 5%.
You want to collect more informative features for tour model.


Compare ratings of players over different leagues

An existing way to do this is to use modified Elo ratings.

Video game servers sometimes use a similar scoring system called Glicko or Glicko-2 which might be better for your purposes.

My understanding is that you want to understand the rating of each player relative to every other player and that each in-league rating is essentially the rating of each player in that league relative to each other player in that league. If this is true then there are two ways I can think of to help you work out the hypothetical rating of player X in league L2 based on their performance in league L1:

Generate global ratings based on one of the methods mentioned above and use them to insert the new player in the right place in the ranking
Calculate am in group rating based on the methodologies described above and then calculate the new players ranking by stimulating the score he/she would accumulate if they played each of the players in L2 that they could be compared to in L1 and achieved the expected outcome (i.e. beat all the players they were ranked better than and were beaten by all the players they were ranked worse than)

You might also be interested to check whether the rank correlation between leagues is strong based on players that are ranked in both leagues.


How to fix class imbalance in training sample?

Actually what they mentioned is right. The idea of oversampling is right and is one of, in general, Resampling methods to cope with such problem. Resampling can be done through oversampling the minorities or undersampling the majorities. You may have a look at SMOTE algorithm as a well-stablished method of resampling.
But about your main question: No it's not only about the consistency of distributions between test and train set. It is a bit more.
As you mentioned about metrics, Just imagine accuracy score. If I have a binary classification problem with 2 classes, one 90% of the population and the other class 10%, then with no need of Machine Learning I can say my prediction is always the majority class and I have 90% accuracy! So it just does not work regardless of the consistency between train-test distributions. In such cases you may pay more attention to Precision and Recall. Usually you would like to have a classifier which minimizes the mean (usually harmonic mean) of Precision and Recall i.e. the error rate is where FP and FN are fairly small and close to each other.
Harmonic mean is used instead of arithmetic mean because it supports the condition that those errors are as equal as possible. For instance if Precision is $1$ and Recall is $0$ the arithmetic mean is $0.5$ which is not illustrating the reality inside the results. But harmonic mean is $0$ which says however one of the metrics is good the other one is super bad so in general the result is not good.
But there are situations in practice in which you DO NOT want to keep the errors equal. Why? See the example bellow:
An Additional Point
This is not exactly about your question but may help understanding.

In practice you may sacrifice an error to optimize the other one. For instance diagnosis of HIV might be a case (I am just mocking-up an example). It is highly imbalanced classification as, of course, the number of people with no HIV is dramatically higher than the ones who are carrier. Now let's look at errors:

False Positive: Person does not have HIV but test says they have.
False Negative: Person does have HIV but test says they don't.

If we assume that wrongly telling someone that he got HIV simply leads to another test, we may take much care about not wrongly say someone he is not a carrier as it may result in propagating the virus. Here your algorithm should be sensitive on False Negative and punishes it much more than False Positive i.e. according to the figure above, you may end up with higher rate of False Positive.
Same happens when you want to automatically recognize people faces with a camera to let them enter in an ultra-security site. You don't mind if the door is not opened once for someone who has permission (False Negative) but I'm sure you don't want to let a stranger in! (False Positive)
Hope it helped.


How to deal with big output values after classification layer during training?

As you stated in your question, those numbers go into a softmax function. Another name for softmax is normalized exponential function. Softmax normalizes numbers where the sum is constrained to be 1 and each value becomes the probability of categorical membership.
In the specific case of [23.4, -21254.3, 32123.4], applying the softmax transforms them to [0, 0, 1]. The numbers become either 0 or 1 because of numerical underflow.


The task of predicting how many years a person has left to live is called survival analysis. Survival analysis is a type of time to event analysis. Thus, survival analysis needs a special type of loss function. An appropriate loss function would avoid predictions like 50 years left when the current age is 70. A common survival analysis model is Cox regression model which can be adapted to neural networks.

If survival analysis is used, the current age can be inputted in the model directly in a single input node.


How reliable are model performance reportings?

The question of the measured test error of a classification model is reliable, hence if the test error on unknown set $T_1$ is the same as on $T_2$ is hard two answer. It depends on the following factors:

How many digits of the error are reported?
How many samples have $T_1$ and $T_2$? The more digits are reported, the more samples you need. As a rule of thumb, make sure that any change in the reported error means at least 3 samples have changed. So if you use accuracy and report two decimal places (e.g. 12.34%), then 0.01% must be bigger than 3 => $3 < \frac{0.01}{100} \cdot |T_1| \Leftrightarrow 30000 < |T_1|$
The distribution must be similar. The simplest part is the distribution of classes. The more difficult part is how the features look like.

For other forms of error analysis, you might want to look into my Master's thesis Analysis and Optimization of Convolutional Neural Network Architectures


Cannot fig out error in my gradient function implementation in python

Check your theta dimensions.

Most likely, your X dimensions indicate you have 2458 training samples per iteration with each having 31 features. Hence, your theta should be a matrix of shape (31, 1).

With X having shape (2458, 31) and if theta has shape (31, 1), X*theta will have dimensions (2458, 1), same as y and as expected. Now, y-theta has the same dimensions as that of y or theta. And so does exp(y-theta)

X_T has shape (31, 2458) and hence, d_theta = - X_T*exp(y-theta) will have shape (31, 1), same as our initial assumed theta shape and now, you can subtract d_theta from theta.


Online learning w/ feature weighting/adjusting

Yes, this can be done in Python. Scikit-Learn has a few online learning algorithms available, of which you can derive the feature importances. Look at the following webpage under 6.1.3. Incremental learning:

http://scikit-learn.org/stable/modules/scaling_strategies.html


How to interpret a specific feature importance?

You can use one of 2 approaches:

The 1st is unsupervised:

Use PCA algorithm to extract the feature vectors best representing the dataset variance. The PCA algorithm extract new features which each of them is a linear combination out of the other features (independent from the label) when the 1st feature it extracts is most important feature and last one is the least important. Then you can retrieve the weights of each "Beta" value at the most important feature. Here is an example for that:
https://stackoverflow.com/a/34692511/6677037

Another approach is the supervised:

using the labels, which you should use carefully and not choose the features based on the test set.
With these methods you can see the most important features using "Xi Square" , "mutual information gain" etc. Then you remove the least important features.
here is the easiest way to do that:
https://hub.packtpub.com/4-ways-implement-feature-selection-python-machine-learning/

good luck.


Word2Vec is not a combination of two models, rather both are variants of word2vec. Similarly doc2vec has Distributed Memory(DM) model and Distributed Bag of words (DBOW) model. Based on the context words and the target word, these variants arised.

Note: the name of the model maybe confusing

Distriubted Bag of words is similar to Skip-gram model
Distributed Memory is similar to Continuous bag of words model


remove special character in a List or String

Regular expressions can be used to create a simple tokenizer and normalizer:
from __future__ import annotations
import re

def tokens(text: str) -> list(str):
    "List all the word tokens in a text."
    return re.findall('[\w]+', text.lower())

assert tokens("To be, or not to be, that is the question:") == ['to', 'be', 'or', 'not', 'to', 'be', 'that', 'is', 'the', 'question']

Otherwise, use an established library like spaCy to generate a list of tokens.


Turning R scripts into a "data science tool"

You can refer below the answer to the similar question asked.

https://stackoverflow.com/questions/14096520/compile-r-script-into-standalone-exe-file

Alternatively,
You can use R-shiny app for it. You can build application which can be deployed on website also. See below link for the examples of application made by R-shiny.

https://shiny.rstudio.com/gallery/

This way client will not be seeing your code at all and will be working on it just as a tool.


First you should define your goal for the prediction. Do you want to predict also a time series output? Then the best thing to do is to use any machine learning model that is sequence focused (e.g. is LSTM), and with the time series that you have as input. Also study the time scale of your model. Study if it should be hour, minutes or any other scale.


You’ve asked your question a while ago, but maybe it helps someone anyway.

In the mllib guide they say to refer to the PrefixSpan paper by Jian Pei and his team, there’s another version fully available here

There, they explain that a sequence is composed of elements which are composed of items:
For example “a sequence (a (abc)(ac)d(cf)) has five elements: a , (abc), (ac) , d and (cf)  where items a and c appear more
than once respectively in different elements.” (taken from the paper)

That's why you need a list of lists.

And note that items within an element shouldn’t follow any particular order (in the paper they are assumed to be sorted alphabetically).  So if you are dealing with time series I suppose that the events arriving at the same time (so no particular order) would form the items of an element.


How can I get an objective measure of song similarity?

Two songs are two separated documents which have characteristics that make them similar or non-similar. There are plenty of techniques to determine similarness between documents:

How to compute the similarity between two text documents


Which classification model to use on large, high-dimensional dataset?

I don't really have experience with such a massive dataset, but my first thought would be to explore the instances in order to see if so many are needed. I would start with an ablation experiment, trying various sizes of training data with a simple method (random forest seems a good idea) in order to observe the evolution of the performance w.r.t size of the training data. It's likely that the performance reaches a plateau at some point, and it would be useful to know this point.

It might also make sense to study if the data contains duplicates or near duplicates. You can't remove duplicates directly because the distribution matters, but it might be possible to replace them by assigning weights to instances. I'm not expert in this but there are methods for instance selection.


Help with approaching this problem correctly

This problem sounds like an optimization problem , where the goal to find the best solution under constraints. In this case, the solution space would minimize travel times while meeting staff constraints.
Often the most difficult aspect is encoding the raw data into a format that works for the optimization solver. Generally, try a simplest encoding that could possible work. For example, use distance as a proxy for travel times.
PuLP is a linear programming package for Python that could solve the problem once it is properly encoded.


LSTMs: what is $W_x$ & $U_z$ in $φ(W_x + U_z + b)$?

If you look deeper in LSTMs or GRUs, we observe that the gates(input, output, cell or forget based on the RNN) are calculated using an equation like you specified.

Example, according to deep learning tutorial of lstm, it=sigma(Wi xt + Ui ht-1 + bi) In this, h is the hidden state vector and x is the input state vector as specified and W and U are the corresponding weights for the input gate it. SImilarly, there are gates for output and forget. So in the paper, they recall a gist of RNNs and sum it up as a general equation. It is a common computational block in RNNs despite their minor differences.

Refer Colah's blog or wildml, I think they are one of the best to understand RNNs.


R has {drake} which makes it easy to make reproducible data pipelines. Does Python have a similar package?

Sklearn has pipeline. If you have fit and transform attributes iteratively, you can make them pipeline by Pipeline class in sklearn.pipeline.

Read the docs:

https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html

Additionally you can save and load a pipeline object by joblib.dump and joblib.load.


Help with deep learning for motorbike inspection

Data

If you are falling short of data, you obviously need to create a custom dataset i.e by labelling each part of a motorcycle. If you can collect some images, we can use a pre-trained classifier along with image augmentation technique. You can Google's Dataset Search.

To annotate your dataset, you can find tools here and here. If you need to download a number of images from Google Image Search, try this tool.

Model Architecture

Now, regarding the model, we can use any object localization algorithms to detect various parts of a motorcycle. Next, these individual parts would be fed to different models which will classify the respective faults. For detection, we can use

Faster RCNN and Mask RCNN ( semantic segmentation )
YOLO ( Lower inference time )


Do statistical test with orange between data

The Box Plot gives a T-Test for two-group comparisons and Anova for > 2 group comparisons.

The Sieve Diagram gives a Chi2 test.


Identifying software and version range in a sentence

If all your sentences are in the following format(atleast that what I assume), you can parse it with regex instead.

`([A-Z][a-z]+)\s+(\d+\.\d+)(?:\s+before\s+)(\d+\.\d+)`

Using this pattern with the given sentence finds all the software and the versions in it. re.findall(pattern, sentence) will give you a list of tuples like this
[('Firefox', '2.4', '2.5'), ('Safari', '6.0', '6.8')]


Since you do not have labelled data, you'll have to use a lot of domain knowledge. Also, predicting a class (poor, bad, normal, good, perfect) might be easier than predicting a score (0-100). The approach you could use is:

Apply clustering on your dataset.
Try to find the optimal number of clusters for your dataset using the Elbow Method. Suppose you get 3 optimal clusters.
Now comes the part for applying domain knowledge. Try to analyse the clusters by picking up individual examples from the cluster and using domain knowledge identify which category (here, good, bad, average) you can classify them into.
After analysing some examples from the clusters, you can assign the label you get to the entire clusters as the examples within a cluster are similar.

The above method makes some assumptions but can be an interesting experiment for your dataset.