Why can't I specify the correct NumPy size?

Keras needs you to pass one more dimension than it says in the error message: the batch dimension.
I.e.
If you have a model that requires each sample to have an input shape of (4,) and you have 1000 training samples you need to feed it with an array of (1000, 4).
In your case since you want to feed it with just one sample you need to pass a shape of (1, 4). Try self.model.fit(np.array([[1, 1, 1, 1]])) (notice the one extra bracket with makes the array's shape be (1, 4)) .


Obtaining a dataset is an important part of defining a ML problem but it's not the only one. Typically this involves the following steps:

Define the goal of the problem. Example: predict AZT level of tolerance among AIDS patients.
Obtain appropriate data for the problem.
Design the formal setting of the experiment:

what kind of problem is it (e.g. classification)
what is the target variable and what are the features in the data
how to evaluate the quality of the results (performance measure, e.g.f1-score)
Experimental setup: ML method(s), use of cross-validation etc.


How to compare a sentence with a paragraph and get its probability in terms of correctness?

This is a complex problem related to Natural Language Understanding (NLU). The key part in such a system is certainly textual entailment, but it could also use techniques such as Question Answering and summarization.

I'm not aware of any direct model or tool to carry out this task exactly, so I think you will have to study the literature in these fields.


I think you will have to experiment; there isn't generally a one-activation-fits-all for hierarchical models. Give the ones you mentioned a try, but perhaps starting with sigmoid and tanh and also a LeakyReLU.

The reason I'd perhaps leave out a normal ReLU at first is because that too would potential trim out information before the final model, as it simply removes negative weights to zero. A good final model should be able to correct for this, but I think its unnecessary to force it to.


Diminishing returns in language identification data set size?

LIDE: Language Identification from Text Documents benchmarked the difference between using character and word n-gram information for natural language identification.

Word-length algorithm for language identification of under-resourced languages proposed new algorithms for languages with limited available data.


converting a Tensorflow model for Tensorflow.js

You can build from source and use the summarize_graph tool which is distributed as a part of Tensorflow graph_transforms tools. Detailed docs are available here. Quick install:

First, make sure you have bazel build system installed (for MacOS):

brew install bazel

Next get Tensorflow sources, build and run summarize_graph:

git clone git@github.com:tensorflow/tensorflow.git
cd tensorflow
bazel build tensorflow/tools/graph_transforms:summarize_graph
bazel-bin/tensorflow/tools/graph_transforms/summarize_graph --in_graph=path/to/model_graphdef.pb

Note that summarize_graph requires GraphDef format as input.

If during the build attempt bazel can't find Xcode, configure it:

bazel clean --expunge
sudo xcode-select -s /Applications/Xcode.app/Contents/Developer
sudo xcodebuild -license
bazel clean --expunge


lookup and fill some value from one dataframe to another

I would recommend "pivoting" the first dataframe, then filtering for the IDs you actually care about.
Something like this:
useful_ids = [
    'A01',
    'A03',
    'A04',
    'A05',
]
df2 = df1.pivot(index='ID', columns='Mode')
df2 = df2.filter(items=useful_ids, axis='index')


How to handle fixed values for variables in pre-processing

By definition, if these columns or features contain a constant value and yet the output variables change, then they are not influencing the output and likely can be ignored.
A more formal test is to determine how much of the variance between a model that uses that feature is attributable to that feature.
A simple example to illustrate this principle is to look up examples of PCA. In those examples, the technique tries and identifies feature that drive the most variance.


(500,) stands for a array with 500 values,
array (500,20), stands for an array which contains 500 list and each list containing 20 elements.
Examples:
array(1,2,3,4) ----------------------> (4,)

array([1,2],[3,4],[5,6])-------------> (3,2)


Basic Time Series Classification Examples

What you need to learn are RNNs and LSTMs. Here are some links I used to learn these:

CS231n RNN+LSTM lecture
Understanding LSTMs

Also I would suggest you to use Keras, a Tensorflow API. In my experience, it makes working with RNNs and LSTMs way easier, if you're a beginner.

I found these articles which seemed related to your problem:

Time series classification project by naveen sai on github
https://aboveintelligent.com/time-series-analysis-using-recurrent-neural-networks-lstm-33817fa4c47a
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/

Keywords to search: Time series classification tensorflow keras


How to correct mislabeled data in dataset?

Yes! This could be an excellent test case for your classification algorithm. With only 5% mislabeling a good algorithm will be easily able to identify "outliers" by having much worse predictions for these mislabeled records.

If you were able to at least identify "correct" records to generate a training set that would be even better but with 5% mislabeling even if not it won't be a problem. This also leads to the second part of the answer, that while it might be better to remove or correct the mislabeled records it might also not matter.

This is obviously based on the assumption that the 5% errors are somewhat randomly distributed over all classes.

Finally, you did not mention any hints/data/info that could identify mislabels. Obviously if you have information about those errors doing some pre-processing to identify and remove them based on analysis / rules generation would be best.


Initial values of memory and previous block output in LSTM?

Either you use a zero vector state for both (most commonly used), or you can use other approaches such as described in this article.

I have personally always used zero vectors.


Does a NB wrapper consider feature subset size?

Short Version

The difference is in the trade-off between complexity of model and how hard it will be to train.  The more states your variable can take, the much more data you'll need to train it.  It sounds like you're reducing the feature space through some front-end processing.

Example

Context:

Say you're using NBC to model if you go outside given the high temperature that day.  Let's say temperature is given to you as an int in Celsius which just happened to range between 0 and 40 degrees.  Over one year of running the experiment - recording the temperature and if you went outside - you'll have 365 data points.

Internal Representation:

The internal structure the NBC uses will be a matrix with 82 values (41 values for your one input variable * 2 because you have two states in your output class).  That means each bin will have an average of 365/82 = 8.9ish samples.  In practice, you'll probably see a few bins with lots more samples than this and a many bins with 0 samples.

A Pitfall

Say you saw 8 cases at a temperature of 5 C (all of which you stayed inside) and nothing at a temp of 3 or 4 C.  If, after the model is built, you 'ask it' what class a temp of 4 C should be in.  It won't know.  Intuitively, we'd say "stay inside", but the model won't.  One way to fix this is to bin the classes 0,1,2,... into larger groups of temperatures (i.e., class 0 for temp 0-3, class 1 for temp 4-7, etc).  The extreme case of this would be two temperature states "high" and "low".  The actual cut-off should depend more on the data observed, but one such scheme is converting temperatures within 0-20 to "low" and 21-40 is "high".

Discussion

This front end processing seems to be what you're talking about as the "wrapper" around the NBC.  The result of our "high" and "low" binning scheme will result in a much smaller model (2 input states and 2 output classes gives you a 2*2 = 4 number of classes.  This will be way easier to process and will take way less data to confidently train at the expense of complexity.  One example of a drawback of such a scheme is:  Say the actual pattern is that you love spring and fall weather and only go outside when it's not too hot or not too cold.  So your outside adventures are evenly split across the upper part of the "low" class and the lower part of the "high" class giving you a really useless model.  If this were the case, a 3-class bin scheme of "high", "medium", and "low" would be best.


Euclidean distance for more than 2 datapoints

The Euclidean distance between 2 vectors

$\mathbf{u}$: $(u_1, u_2, ... , u_n)$ and $\mathbf{v}$: $(v_1, v_2, ... , v_n)$

is simply

$\sqrt{(u_1-v_1)^2+(u_2-v_2)^2+...+(u_n-v_n)^2}$


Is the LSTM share the same model parameter in each block?

I am confused about the parameters: $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$.  Are those parameters trained by data(e.g., back-propagation)?

Yes. They do get trained by backpropagation. There are caveats here. Variants to backprop too exist when training a sequence-to-sequence to models. Check out BPTT for example.

  My second question is that, In the Figure, We have the three block of LSTM 1, 2, and 3 (mark by blue color). Do them share the same $U^i, U^f, U^o , U^g$ and $W^i, W^f, W^o , W^g$. ?  Or each block have their own $U^*$ and $W^*$?

Each block has the same set of weights shared among them.

An following mini example demo would be like in Keras.

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.layers import Embedding
from keras.layers import LSTM
model = Sequential()
model.add(LSTM(128, input_shape=(None, 100)))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])

creates an simple LSTM model. If you look at model summary to count the params you got.

model.summary

You will get

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm_3 (LSTM)                (None, 128)               117248
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 129
=================================================================
Total params: 117,377
Trainable params: 117,377
Non-trainable params: 0
_________________________________________________________________

And the total of 117,377 params you see is obtained from (((128*128)+(128*100)) * 4) + (4*128). Here W is of size 128*100 and U is 128*128 and there are 4 of them resp. and last 4*128 goes for the biases.


Why augmenting the training data with binary attributes works better for our dataset?

Your third model has the most capacity of the three.  When you have the values $x$ with special negative value $\xi$  and indicators $\mathbf{1}_0$ and $\mathbf{1}_{\xi}$, the linear model can fit
$$ \alpha x + \beta \mathbf1_0 + \gamma \mathbf1_{\xi}$$
When $x>0$ you just get $\alpha x$, when $x=0$ you just get $\beta$, and when $x=\xi$ you get $\alpha \xi + \gamma$;
so your model can get the correct (linear) signal out of the positive values and arbitrary values for the two special values.


How to make a CNN predict a continuous value?

A common way to do this is flatten your output after your last convolution layer, and pass it through a fully connected layer.


Regression performance varies hugely on shuffling training and testing data

I can't add comments yet, so that's why I'm making a post.
It is natural that for a time series data (I assume that's the kind of data you have if you don't want to shuffle) without shuffling you get worse results. Imagine this toy example. You want to predict sales of a grocery store where your data is sales of the store for every day. If you shuffle you can get dates in your training set that are later than dates in test set. So for example if you have 2nd of March in your training set, it is easy to predict the sales for 1st of March if it's in the test set. However, this situation is not feasible in real life - you don't have data about future when making the prediction.


Difference of Activation Functions in Neural Networks in general

A similar question was asked on CV: Comprehensive list of activation functions in neural networks with pros/cons.

I copy below one of the answers:

  One such a list, though not much exhaustive:
  http://cs231n.github.io/neural-networks-1/

  Commonly used activation functions

  Every activation function (or non-linearity) takes a single number
  and performs a certain fixed mathematical operation on it. There are
  several activation functions you may encounter in practice:

           Left: Sigmoid non-linearity
  squashes real numbers to range between [0,1] Right: The tanh
  non-linearity squashes real numbers to range between [-1,1].

  Sigmoid. The sigmoid non-linearity has the mathematical form $\sigma(x) = 1 / (1 + e^{-x})$ and is shown in the image above on
  the left. As alluded to in the previous section, it takes a
  real-valued number and "squashes" it into range between 0 and 1. In
  particular, large negative numbers become 0 and large positive numbers
  become 1. The sigmoid function has seen frequent use historically
  since it has a nice interpretation as the firing rate of a neuron:
  from not firing at all (0) to fully-saturated firing at an assumed
  maximum frequency (1). In practice, the sigmoid non-linearity has
  recently fallen out of favor and it is rarely ever used. It has two
  major drawbacks:

  Sigmoids saturate and kill gradients. A very undesirable property of the sigmoid neuron is that when the neuron's activation
  saturates at either tail of 0 or 1, the gradient at these regions is
  almost zero. Recall that during backpropagation, this (local) gradient
  will be multiplied to the gradient of this gate's output for the whole
  objective. Therefore, if the local gradient is very small, it will
  effectively "kill" the gradient and almost no signal will flow through
  the neuron to its weights and recursively to its data. Additionally,
  one must pay extra caution when initializing the weights of sigmoid
  neurons to prevent saturation. For example, if the initial weights are
  too large then most neurons would become saturated and the network
  will barely learn.
  Sigmoid outputs are not zero-centered. This is undesirable since neurons in later layers of processing in a Neural Network (more on
  this soon) would be receiving data that is not zero-centered. This has
  implications on the dynamics during gradient descent, because if the
  data coming into a neuron is always positive (e.g. $x > 0$
  elementwise in $f = w^Tx + b$)), then the gradient on the weights
  $w$ will during backpropagation become either all be positive, or
  all negative (depending on the gradient of the whole expression
  $f$). This could introduce undesirable zig-zagging dynamics in the
  gradient updates for the weights. However, notice that once these
  gradients are added up across a batch of data the final update for the
  weights can have variable signs, somewhat mitigating this issue.
  Therefore, this is an inconvenience but it has less severe
  consequences compared to the saturated activation problem above.

  Tanh. The tanh non-linearity is shown on the image above on the right. It squashes a real-valued number to the range [-1, 1]. Like the
  sigmoid neuron, its activations saturate, but unlike the sigmoid
  neuron its output is zero-centered. Therefore, in practice the tanh
  non-linearity is always preferred to the sigmoid nonlinearity. Also
  note that the tanh neuron is simply a scaled sigmoid neuron, in
  particular the following holds: $ \tanh(x) = 2 \sigma(2x) -1  $.

           Left: Rectified Linear
  Unit (ReLU) activation function, which is zero when x &lt 0 and then
  linear with slope 1 when x &gt 0. Right: A plot from Krizhevsky
  et al. (pdf) paper indicating the 6x improvement in convergence
  with the ReLU unit compared to the tanh unit.

  ReLU. The Rectified Linear Unit has become very popular in the last few years. It computes the function $f(x) = \max(0, x)$. In
  other words, the activation is simply thresholded at zero (see image
  above on the left). There are several pros and cons to using the
  ReLUs:

  (+) It was found to greatly accelerate (e.g. a factor of 6 in Krizhevsky et
  al.) the
  convergence of stochastic gradient descent compared to the
  sigmoid/tanh functions. It is argued that this is due to its linear,
  non-saturating form.
  (+) Compared to tanh/sigmoid neurons that involve expensive operations (exponentials, etc.), the ReLU can be implemented by simply
  thresholding a matrix of activations at zero.
  (-) Unfortunately, ReLU units can be fragile during training and can "die". For example, a large gradient flowing through a ReLU neuron
  could cause the weights to update in such a way that the neuron will
  never activate on any datapoint again. If this happens, then the
  gradient flowing through the unit will forever be zero from that point
  on. That is, the ReLU units can irreversibly die during training since
  they can get knocked off the data manifold. For example, you may find
  that as much as 40% of your network can be "dead" (i.e. neurons that
  never activate across the entire training dataset) if the learning
  rate is set too high. With a proper setting of the learning rate this
  is less frequently an issue.

  Leaky ReLU. Leaky ReLUs are one attempt to fix the "dying ReLU" problem. Instead of the function being zero when x < 0, a leaky ReLU will instead have a small negative slope (of 0.01, or so). That is, the function computes $f(x) = \mathbb{1}(x < 0) (\alpha x) + \mathbb{1}(x>=0) (x) $ where $\alpha$ is a small constant. Some people report success with this form of activation function, but the results are not always consistent. The slope in the negative region can also be made into a parameter of each neuron, as seen in PReLU neurons, introduced in Delving Deep into Rectifiers, by Kaiming He et al., 2015. However, the consistency of the benefit across tasks is presently unclear.

  Maxout. Other types of units have been proposed that do not have the functional form $f(w^Tx + b)$ where a non-linearity is applied
  on the dot product between the weights and the data. One relatively
  popular choice is the Maxout neuron (introduced recently by
  Goodfellow et
  al.) that
  generalizes the ReLU and its leaky version. The Maxout neuron computes
  the function $\max(w_1^Tx+b_1, w_2^Tx + b_2)$. Notice that both
  ReLU and Leaky ReLU are a special case of this form (for example, for
  ReLU we have $w_1, b_1 = 0$). The Maxout neuron therefore enjoys
  all the benefits of a ReLU unit (linear regime of operation, no
  saturation) and does not have its drawbacks (dying ReLU). However,
  unlike the ReLU neurons it doubles the number of parameters for every
  single neuron, leading to a high total number of parameters.

  This concludes our discussion of the most common types of neurons and
  their activation functions. As a last comment, it is very rare to mix
  and match different types of neurons in the same network, even though
  there is no fundamental problem with doing so.

  TLDR: "What neuron type should I use?" Use the ReLU non-linearity, be careful with your learning rates and possibly
  monitor the fraction of "dead" units in a network. If this concerns
  you, give Leaky ReLU or Maxout a try. Never use sigmoid. Try tanh, but
  expect it to work worse than ReLU/Maxout.

  License:
  The MIT License (MIT)

  Copyright (c) 2015 Andrej Karpathy

  Permission is hereby granted, free of charge, to any person obtaining
  a copy of this software and associated documentation files (the
  "Software"), to deal in the Software without restriction, including
  without limitation the rights to use, copy, modify, merge, publish,
  distribute, sublicense, and/or sell copies of the Software, and to
  permit persons to whom the Software is furnished to do so, subject to
  the following conditions:

  The above copyright notice and this permission notice shall be
  included in all copies or substantial portions of the Software.

  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
  EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
  IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
  CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
  TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
  SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.*


model.predict keeps predicting the same, wrong, class

Figured it out.  Need to normalize the image I'm going to predict to $[-1,1]$ instead of what-I-assume-is-the-standard $[0,1]$.  It seems this is model-dependent.  The project I'm working off of uses transfer learning and imports this model.
It's not immediately clear to me though just how this model needs inputs normalized to $[-1,1]$.


Here is nice implementation of mixed type data in R-

https://dpmartin42.github.io/posts/r/cluster-mixed-types

This question right here-
K-Means clustering for mixed numeric and categorical data

and a Discussion Thread of Kaggle-

https://www.kaggle.com/general/19741

There are ways, to either map your categorical data to numeric type and then you can go about the business as usual, or choose similarity measures which works for categorical data type, in this case you have options to choose from counting frequencies etc  .


Real world example of Generative model & Discriminative model

A generative model is able to generate instances from a given distribution. So let's try to get our model to generate instance from the distribution of all hand written digits. This includes numbers 0 to 9 and for each of these numbers there are a very large set of possible way of writing the same number. Take the number 4 from the MNIST dataset, which are comprised of handwritten numbers.

You can quickly see how many different ways each of these numbers can be written yet still represent the same value. This is the distribution we want to capture using our generative model. The model works by training a generator to effectively try and cheat a discriminator that has been trained to distinguish between real and artificial instances from the desired distribution. When the discriminator can no longer distinguish between the artificial and real instances, we can say the generative model can successfully produce examples which match this distribution.

In other words, once the generator starts producing realistic digits, we can say that the generator is producing instances which match the instances captured in the MNIST dataset.

This idea of matching a distribution can be applied to  wide range of applications where you want to draw from a distribution. Some for demonstration purposes of the techniques without any real useful application and some very useful applications which are currently being used for a number of tasks. Here is a short lists of some examples:

Generating artificial faces
Text to image generation
Producing fake voices or noises
Image denoising
Changing geometries within an image
Coloring images
MRI image reconstruction

Most importantly, to generate new Pokemon.

In theory, given sufficient data, any distribution (anything in the world, including the world itself) can be modeled by a generative model from which we can draw instances. So any time you think of a simulation in the far future some generative model will be able to observe multiple real examples and then produce a large number of simulated examples. For example, simulating car crashes, a generative model can visualize multiple real crashes, then we can use the generative model to produce millions of similar instances for testing purposes without having to buy more cars. Of course GANs are not that powerful, they require a lot of data to learn very constraint distributions. However, through innovation anything is possible, until then we can continue to generate images of handwritten numbers and bedroom scenes.


Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

Word of warning from a former airline Revenue Management analyst: you might be barking up the wrong tree with this approach. Apologies for the wall of text that follows, but this data is a lot more complex and noisy than might appear at first glance, so wanted to provide a short description of how it's generated; forewarned is forearmed.

Airline fares have two components to them: all the actual fares (complete with fare rules and what have you) that an airline has available for a certain route, most of which are published the Airline Tariff Publishing Company (a few special-use ones are not, but those are the exception rather than the rule) and the actual inventory management performed by the airline on a day-to-day basis.

Fares can be submitted to ATPCO four times a day, at set intervals, and when airlines do so, it will usually consist of a mixture of additions, deletions, and modifications of existing fares. When an airline initiates a pricing action (assuming their competitors aren't trying to make their own moves here), they usually have to wait until the next update to see if their competitors follow/respond. The converse goes when a competitor initiates a pricing action, as the airline has to wait until the next update before they can respond.

Now, this is all well and good with respect to fares, but the problem is that, because this is all getting published in ATPCO, fares are the next best thing to public information... all your competitors get to see what you've got in your arsenal, so attempts to obfuscate are not unheard of, such as publishing fares that will never actually be assigned any inventory, listing all the fares as day-of-departure, etc.

In many ways, the secret sauce comes down to the actual inventory allocation, i.e. how many seats on each flight will you be willing to sell for a given fare, and this information is not publicly available. You can get some glimpses by scraping web info, but the potential combinations of departure time/date and fare rules are quite numerous and may quickly escalate beyond your ability to easily keep track of.

Typically an airline will only be willing to sell a handful of seats for a very low fare and the people who snag those have to book quite far in advance lest the fare rules lock them out, or other travelers simply beat them to the punch. The airline will be willing to sell a few more seats for a higher fare, and so on and so forth. They will be quite happy to sell all of the seats for the highest fare they've got published, but this is not usually feasible.

What you're seeing with fares getting higher the closer you get to the day of departure is simply the natural process of having the cheap seats get booked farther out, while the remaining inventory gradually gets more expensive. Of course, there are some caveats here. The RM process is actively managed and human intervention is quite common as the RM team generally strives to meet its revenue goals and maximize revenue on each flight. As such, flights that fill up quickly may be "tightened up" by closing out low fares. Flights that are booking slowly may be "loosened up" by allocating more seats to lower fares.

There is a constant interplay and competition between airlines in this area, but you are not very likely to capture the actual dynamics just from scraping fares. Don't get me wrong, we had such tools at our disposal, and, despite their limitations, they were quite valuable, but they were just one data source that fed into the decision-making process. You'd need access to the hundreds, if not thousands of operational decisions made by RM teams on a daily basis, as well as state-of-the-world information as they see it at the time. If you cannot find an airline partner to work with in order to get this data, you might need to consider alternate data sources.

I'd recommend looking into getting access to O&D fare data from the Official Airline Guide (or one of their competitors) and try to use that for your analysis. It's sample-based (about 10% of all tickets sold) and aggregated at a higher level than would be ideal so careful route selection is imperative (I'd recommend something with plenty of airlines, flying non-stop multiple times a day, with large aircraft), but you may be able to get a better picture of what was actually sold (average fare) and how much of it was sold (load factor), vs. merely what is available for sale at a given point in time. Using that information you might be in better position to at least explore the outcomes of the airlines' pricing strategy, and make your inferences from there.


I suggest you use facebook's faiss. It is a library for similarity search, which can be used to compute kNN on large vector collections.
From facebook's own numbers:

With approximate indexing, a brute-force k-nearest-neighbor graph (k = 10) on 128D CNN descriptors of 95 million images of the YFCC100M data set with 10-intersection of 0.8 can be constructed in 35 minutes on four Maxwell Titan X GPUs, including index construction time.
Billion-vector k-nearest-neighbor graphs are now easily within reach. One can make a brute-force k-NN graph (k = 10) of the Deep1B data set with 10-intersection of 0.65 in under 12 hours on four Maxwell Titan X GPUs, or 0.8 in under 12 hours on eight Pascal P100-PCIe GPUs. Lower-quality graphs can be produced in under 5 hours on the Titan X configuration.


What is the difference between classification and regression?

For the sake of illustration, let's imagine that you're trying to predict the amount of gas in the tank of your car.

A classification problem statement of this question would be whether you have gas in your car (yes or no).

A regression statement of this problem would predict the level of gas in your car (anywhere between completely full or completely empty) and could take any value.

The output of a classification model can be one of n options, where n is the number of classes (and/or the probability associated with each class).

The output of a regression model is a (possibly bounded) continuous value.


ImportError: DLL load failed: The specified module could not be found

Use the following command to install tensorflow (even if you have already installed it)

For me it worked on win7 64bit and python 3.6.5

  conda install tensorflow keras


Whether Interaction terms should be included in Linear Regression analysis?

Just to expand on @Ankita Talwar answer and give some slightly more formal intuition you can write a linear model with to regressors and their interaction as follows:
$$ y = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1 x_2$$
where $x_1 x_2$ is the interaction term.
Now refactoring you can see that the interaction can be absorbed into the coefficient for $x1$ making it depend on $x_2$: $ y = w_0 + v_1(x_2) x_1 + w_2 x_2$, where now $v_1(x_2) = w_1 + w_2 x_2$ is a function that depends on $x_2$ (alternatively you can view the interaction as modifying $x_2$ coefficient).
So, when you add an interaction term you allow the coefficient value of one variable to vary depending on the value of the other variable. This operation only touches the coefficients not the variables themselves (so it doesn't say anything about them being collinear).
I guess your question might come from the term $x_1 x_2$ in the linear model, noticing that this term depends on $x_1$ (or $x_2$). If that is the case, please notice that $x_1 x_2$ would be collinear with, let say, $x_1$ if $x_2$ were a constant and not a variable (and viceversa). When both are variables (and of course provided the original variables are not linearly related) the interaction term is not collinear with any of the two variables.
However you might encounter some collinearity if one variable has a much smaller scale than the other one, in which case the interaction is mainly driven by the variable with the larger scale. This problem can be solved by standardizing your variables (which is anyway a good idea in regression problems).
Hope this clarifies a bit.


I suggest the use of Hidden Markov Models, with two possible states: (1) high levels and (0) low levels.

This technique might be helpful to decode your signal. Probably you would need a specific HMM for each codification.

If noise is an issue an FIR filter with a Blackman-Harris window function would allow you to isolate the frequency you're concerned with.


Given the definition for mutual information

$$I(X;Y) = \sum_{y \in Y} \sum_{x \in X}
  p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
  \right) },$$

it follows from rearrangement of the summands

$$I(Y; X) =
  \sum_{x \in X} \sum_{y \in Y}
  p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)}
  \right) }.$$

Hence $I(X; Y) = I(Y; X)$.

Edit

  "$I(X;Y)$ measures the average reduction in uncertainty of $X$ that results from knowing $Y$"

If you interpret $H(X) - H(X|Y)$ where $H(X)$ is the marginal entropy of $X$ and $H(X|Y)$ the conditional entropy of $X$ given $Y$ as the reduction in uncertainty, then it is equal to $H(Y) - H(Y|X)$ and $I(X; Y)$ (see Wikipedia).


Why do we have multiple neurons in the output layer of a neural network?

If you want multiple things out of your network you need multiple output nodes. In the case of multiclass classification you want multiple outputs, one for each class. These represent the probability distribution over the different classes. For binary you can get away with only one output because the other class has 1-P as the probability. You could say the same for multiclass that you need one less, however this is commonly parameterized with a softmax which needs all values for the denominator.

Other cases might be for example one output per pixel, for example for image segmentation.


How to extract a subset from a data frame?

There are several ways

df[df['A']>=3]

Maybe using query

df.query('A >=3')

Or using the .loc

df.loc[df['A'] >= 3]

This question is posted on several places.


how to interpret predictions from model?

Alright so I rewrote some parts of your model such that it makes more sense for a classification problem. The first and most obvious reason your network was not working is due to the number of output nodes you selected. For a classification task the number of output nodes should be the same as the number of classes in your data. In this case we have 5 kinds of flowers, thus 5 labels which I reassigned to $y \in \{0, 1, 2, 3, 4\}$, thus we will have 5 output nodes.

So let's go through the code. First we bring the data into the notebook using the code you wrote.

from os import listdir
import cv2

daisy_path = "flowers/daisy/"
dandelion_path = "flowers/dandelion/"
rose_path = "flowers/rose/"
sunflower_path = "flowers/sunflower/"
tulip_path = "flowers/tulip/"

def iter_images(images,directory,size,label):
    try:
        for i in range(len(images)):
            img = cv2.imread(directory + images[i])
            img = cv2.resize(img,size)
            img_data.append(img)
            labels.append(label)
    except:
        pass

img_data = []
labels = []
size = 64,64
iter_images(listdir(daisy_path),daisy_path,size,0)
iter_images(listdir(dandelion_path),dandelion_path,size,1)
iter_images(listdir(rose_path),rose_path,size,2)
iter_images(listdir(sunflower_path),sunflower_path,size,3)
iter_images(listdir(tulip_path),tulip_path,size,4)

We can visualize the data to get a better idea of the distribution of the classes.

import matplotlib.pyplot as plt
%matplotlib inline

n_classes = 5

training_counts = [None] * n_classes
testing_counts = [None] * n_classes
for i in range(n_classes):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(n_classes)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(n_classes)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1,2,3,4))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

We will now transform the data and the labels to matrices.

import numpy as np
data = np.array(img_data)
data.shape
data = data.astype('float32') / 255.0
labels = np.asarray(labels)

Then we will split the data.. Notice that you do not need to shuffle the data yourself since sklearn can do it for you.

from sklearn.model_selection import train_test_split

# Split the data
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, shuffle= True)

Let's construct our model. I changed the last layer to use the softmax activation function. This will allow the outputs of the network to sum up to a total probability of 1. This is the usual activation function to use for classification tasks.

from keras.models import Sequential
from keras.layers import Dense,Flatten,Convolution2D,MaxPool2D
from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K

model = Sequential()
model.add(Convolution2D(32, (3,3),input_shape=(64, 64, 3),activation='relu'))
model.add(MaxPool2D(pool_size=(2,2)))
model.add(Flatten())
model.add(Dense(128,activation='relu'))
model.add(Dense(5,activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Then we can train our network. This will result in about 60% accuracy on the test set. This is pretty good considering the baseline for this task is 20%.

batch_size = 128
epochs = 10
model.fit(x_train, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test_binary))

After the model is trained you can predict instances using. Don't forget that the network needs to take the same shape in. Thus we must maintain the dimensionality of the matrix, that's why I use the [0:1].

print('Predict the classes: ')
prediction = model.predict_classes(x_test[0:1])
print('Predicted class: ', prediction)
print('Real class:  ', y_test[0:1])

This gives

  Predict the classes:  1/1 [==============================] - 0s
  6ms/step Predicted class:  [4] Real class:   [4]

Some suggestions

The model you are currently using is the one that is most common for MNIST. However, that data only has a single channel thus we don't need as many layers. You can increase the performance by increasing the complexity of your model. Or by reducing the complexity of your data, for example you can train using the grayscale equivalent of the images, thus reducing the problem to a single channel.


How to read the predicted label of a Neural Netowork with Cross Entropy Loss? Pytorch

If you're dealing with classification problem, then model.predict is supposed to give you logits.
outputs = net(images)
_, predicted = torch.max(outputs, 1)

for i in range(num_input):
    print(classes[predicted[i]])

if you have only one input then the predicted class would be as following:
classes[predicted[0]]
In your case:
prediction = F.softmax(prediction)
print(prediction)
output = model(input.unsqueeze(0))
_,pred = output.max(1)
print(pred)

# print class name
print(classes[pred[0]])  # <- should be added


How to interpret metrics of a model after scaling the data

I solved this issue with the help of @Franziska W. Thanks!

I currently reverse transform y_test and y_pred and then calculate the metrics as following:

# qt_y is a QuantileTransformer instance, gb is a GradientBoostingRegressor
y_pred = gb.predict(X_test)
inv_y_pred = qt_y.inverse_transform(y_pred.reshape(-1, 1))
inv_y_test = qt_y.inverse_transform(y_test.reshape(-1, 1))

mse_test = mean_squared_error(inv_y_test, inv_y_pred)
print("RMSE on Test:", np.sqrt(mse_test))
print("MSE on Test:", mse_test)

mae_test = mean_absolute_error(inv_y_test, inv_y_pred)
print("MAE on Test:", mae_test)


Python: how to handle categorial values in dataset to build models

Yes, your approach seems to be right. The only thing that I would like to point out is that while it is desirable to convert categorical features to dummies but if you don't have enough memory then you can even consider to factorize your categorical variables. You can read about it more here.
Also, just make sure that after converting to dummies, the number of features are reasonable compared to the number of training samples.


Pandas is a python library that you will find very useful for these types of tasks.

Here is a stack overflow post that tells you how to do what you want to accomplish.

It boils down to three very pythonic lines with a groupby and transformation followed by a drop_duplicates:

import pandas
df = pandas.read_csv('csvfile.csv', header = 0)
df['Total'] = df.groupby(['Name', 'Age'])['Salary'].transform('sum')
df.drop_duplicates(take_last=True)


Relation between hyperparamters and training set for an object detection model

A major difference between the first and the second model you trained is the size of the data assuming that the model is not pretrained. Increased data, of course, needs increased epochs. According, the batch size must also increase.

Batch Size:

While training on the smaller dataset, a batch size of 10 yielded better results. The errors were averaged over 10 samples and then back-propagated through the model. But now for the larger dataset, the batch size remains the same and hence only little optimization occurs as the error is averaged over 10 samples only ( which is relatively smaller for a large dataset ).

Learning Rate:

For the larger dataset, the number of epochs is increased. The purpose of the learning rate is to scale the gradients of the loss with respect to the parameter. A smaller learning rate always helps as it prevents the overshooting of the minima of the loss function. I would insist you increase the learning rate so that the optimization does not diminish as we are having a larger number of epochs. Gradually decrease the learning rate, as the loss approaches its minima.

  If you are training a popular architecture ( like Inception, VGG, etc. ) and that too on datasets like ImageNet, COCO with little modifications, you should definitely read the research papers published on various problems as they would provide a better start to the training.


Scala vs Java if you're NOT going to use Spark?

This is a bit off topic for this SE, or maybe opinion-based, but, I work in this field and I'd recommend Scala.

No I would not characterize Scala as a "stats-oriented" Java. I'd describe it as what you get if you asked 3 people to design "Java 11" and then used all of their ideas at once.

Java 8 remains great, but Scala fully embraces just about all the good ideas from languages that you'd want, like type safety and closures and functional paradigms, and a more elaborate types/generics system, down to more syntactic sugar conveniences like case classes and lazy vals. Plus you get to run everything in the same JVM and interoperate with any Java libraries.

The price is complexity: understanding all of Scala is a lot more difficult than all of Java. And some bits of Scala maybe were a bridge too far. And, the tooling ecosystem isn't great IMHO in comparison to the standard Java tools. But there again you can for example use Maven instead of SBT. But you can mostly avoid the parts of Scala that are complex if you don't need them. Designing Scala libraries takes a lot of skill and know-how; just developing run-of-the-mill code in Scala does not.

From a productivity perspective, once you're used to Scala, you'll be more productive. If you're an experienced Java dev, I actually think you'll appreciate Scala. I have not liked any other JVM language, which tend to be sort of one-issue languages that change lots of stuff for marginal gains.

Certainly, for analytics, the existence of Spark argues for Scala. You'll use it eventually.


According to the notation page, it means the $i$-th column.

$W$ is a template matrix, where the closer $x$ is to $i$-column of $W$, the bigger the value of $h_i$.


Understanding the ||w|| = 1 constraint for SVMs

The constraint is interesting in itself as it prevent divergences (or other strange behavior) for the optimisation problem that is the calibration process. The value it is set to not that much. $||w|| = 1$ is a practical convention. For $||w||$ to be constrained to 2.53 you would have need to :

Have some practical interest for it to being set to 2.53, which there is none. $||w|| = 1$ might help simplify some theoretical calculations.
Reach a consensus between users that it would be good to use this value. It's not hard to see why the consensus was reached for 1 instead of 2.53.


What is the level of measurement / name of the scale of list-features?

With a list such as this list of authors, the only representation which truly capture the semantics is a (long) list of boolean features, one for each possible author in the set. Ideally one would also pre-process the whole set of authors in order to match the different ways a name can be written, e.g. "J. Doe" = "John Doe" = "Doe, John".


What is the difference between "expected return" and "expected reward" in the context of RL?

Is the "expected reward" actually $\mathcal{R}^a_{ss'}$ instead of $V^\pi(s)$?

In short, yes.

Although there is some context associated - $\mathcal{R}^a_{ss'}$ is in the context of specific action and state transition. You will also find $\mathcal{R}^a_{s}$ used for expected reward given only current state and action (which works fine, but moves around some terms in the Bellman equations).

"Return" may also be called "Utility".

RL suffers a bit from naming differences, however the meaning of reward is not one of them.

Notation differences also abound, and in Sutton & Barto Reinforcement Learning: An Introduction (2nd edition), you will find:

$R_t$ is a placeholder for reward received at time $t$, a random variable.
$G_t$ is a placeholder for return received after time $t$, and you can express the value equation as $v_{\pi}(s) = \mathbb{E}[G_t|S_t=s] = \mathbb{E}[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}|S_t=s]$
$r$ is a specific reward value
You won't see "expected reward" used directly in an equation from the book, as the notation in the revised book relies on summing over distribution of reward values.

In some RL contexts, such as control in continuous problems with function approximation, it is more convenient to work with maximising average reward, than maximising expected return. But this is not quite the same as "expected reward", due to differences in context (average reward includes averaging over the expected state distribution when following the policy)


The threshold you choose depends on the specifics of the problem you are trying to solve.  More specifically, it should be based on how you weigh false positives vs. false negatives, i.e. how bad each of these are relative to each other. You mention that you are trying to maximize recall on the positive class, but if that were true you would should just classify everything as a positive class, and get a recall of 1.0.  Based on the domain you are working in, you should decide how much a false positive 'costs' vs. how much a false negative 'costs'.  Once you decide this, you can find the threshold that minimizes the function

total cost = false negative count x FN cost + false positive count x FP cost


It is faster if you use the built-in functions of numpy (instead of reimplementing them yourself):

import numpy as np
from scipy.stats import entropy

np.apply_along_axis(func1d=entropy, axis=2, arr=matrix)


Which machine learning model should I learn for this problem?

You should use a Markov reward model to model your problem. All the possible words are the different states of your chain. The replacement process corresponds to the transitions of your Markov chain.

After defining all the properties of your chain (states, transitions, rewards, ...), you can train your model and get the best strategy for each current word.


Samples in a PCA form lines, what are the reasons?

In general (or common cases) you would find those lines when applying PCA, which are the result of finding those Principal Components Analysis, meaning the directions of maximum variance of your original datatset, i.e., the components which give you more information out of your data.
As explained with examples in scikit-learn docu: PCA finds orthogonal directions in the raw feature space that correspond to directions accounting for maximum variance.


Are there any tools for feature engineering?

Very interesting question (+1). While I am not aware of any software tools that currently offer comprehensive functionality for feature engineering, there is definitely a wide range of options in that regard. Currently, as far as I know, feature engineering is still largely a laborious and manual process (i.e., see this blog post). Speaking about the feature engineering subject domain, this excellent article by Jason Brownlee provides a rather comprehensive overview of the topic.

Ben Lorica, Chief Data Scientist and Director of Content Strategy for Data at O'Reilly Media Inc., has written a very nice article, describing the state-of-art (as of June 2014) approaches, methods, tools and startups in the area of automating (or, as he put it, streamlining) feature engineering.

I took a brief look at some startups that Ben has referenced and a product by Skytree indeed looks quite impressive, especially in regard to the subject of this question. Having said that, some of their claims sound really suspicious to me (i.e., "Skytree speeds up machine learning methods by up to 150x compared to open source options"). Continuing talking about commercial data science and machine learning offerings, I have to mention solutions by Microsoft, in particular their Azure Machine Learning Studio. This Web-based product is quite powerful and elegant and offers some feature engineering functionality (FEF). For an example of some simple FEF, see this nice video.

Returning to the question, I think that the simplest approach one can apply for automating feature engineering is to use corresponding IDEs. Since you (me, too) are interested in R language as a data science backend, I would suggest to check, in addition to RStudio, another similar open source IDE, called RKWard. One of the advantages of RKWard vs RStudio is that it supports writing plugins for the IDE, thus, enabling data scientists to automate feature engineering and streamline their R-based data analysis.

Finally, on the other side of the spectrum of feature engineering solutions we can find some research projects. The two most notable seem to be Stanford University's Columbus project, described in detail in the corresponding research paper, and Brainwash, described in this paper.


How to scale a variable when not knowing the maximum

If your model works in production, you should not retransform your scaler, you should transform new example like 150 is still the maximum value. (It will give you higher than 1, so its a bit problematic but possible solution is below)
However you can still label those example as outlier.

Possible solution for that case: If you have high number of outliers/leverages, you should consider tree ensembles and/or regularized models.

If your predictor is not on production, just add those examples to your train set and fit again since your sample in the first training would be different than the reality.


Is it possible to use the saved xgboost model (with one-hot encoding features) on unseen data (without one-hot encoding) for prediction?

A model is built on a specific set of features, which may include categorical features encoded using one-hot encoding.  If you have new data with additional categories, your model has no idea how to interpret the significance of those categories.  You should either map the new value to none of the 1-hot values identified in training, or to an 'other' value.

For example, say you trained on data that had color=[blue,green].  Your one-hot fields would have color_blue and color_green.  You could also have a field called color=other, that you might use to encode very infrequent values.  That's a data preparation choice. So for 'red', you could encode that as either:

color_green = 0
color_blue = 0

or

color_green = 0
color_blue = 0
color_other = 1

Using either of these techniques will work with xgboost, but as xgboost only accepts numeric inputs, you will have to choose one of these methods as a data pre-processing step.


Time Series Forecasting for Multiple Customers using one RNN

Here are a few rapidfire ideas:

Does the customers affect each other in any way? If that is the case, you need to feed the information of multiple customers at the same time to your RNN. If not, then I would advise creating a time series for each of your customers and feeding it separately to your RNN.
What is it you are trying to predict? Is it a percentage chance of something? Is it a number (e.g. volume of customer purchases). You need to create your outputs, and the inputs you will use to predict that output accordingly. You may also use multiple RNNs if some of your outputs have no relation to each other. For example, if you are trying to predict the volume of purchase for a customer and whether a customer will leave or not, you may want to create two seperate RNNs that will take different inputs (e.g. use the average purchase volume as input for the RNN that predicts the customer volume, and use the frequency of customer visits as input for the RNN that predicts whether the customer will leave or not
Is there huge discrepancy in values of your variables? (e.g. some customers only do 10 dollar purchases, but the others do 100,000, and you want to feed the information of both to your RNN) If that is the case, you may want to "normalize" your variables as well as the outputs (labels) you are trying to predict
Do you really need an RNN? If your customers' outputs are independent of the sequence of variables from the past, then drop the RNN and use something simpler like feedforward neural network or logistic regression
Are your customers' outputs affected by variables from a long time ago? If so, you may want to add a memory component, such as an LSTM, to your RNN
Does your customers have any periodic components? (e.g. the customer usually buys something 10 days after adding it to the shopping cart). This tells you the lookback (how many past days' data you need to feed to the RNN) you will need for your RNN

Hope it helps. I must admit however, that without getting a sense of the inputs and outputs it is difficult to answer this question.


Unsupervised Text Classification with Python: Kmeans

If you're questions are relatively short and the number of labels are not excessive, I would strongly suggest you take a look at zero-shot classification which can be used to classify data to categories without training a model.
Both multi-class and multi-label classification are possible.
Specifically I recommend using a pre-trained language model such as BART MNLI
You can try it out of the box with some of your questions and labels and see how it works: TRY!
From my experience I think that unsupervised approaches won't give the desired results. So if zero-shot classification is no option for you for whatever reason, I would suggest you try something like snorkel to programmatically build some training data.


Depending samples in ad ranking and click rate prediction

Let $X$ represent the event "a user clicks an ad" with advertiser X,
Let event $Y$ represent a new user ($\neg Y$ a returning user),

for any advertiser $X$ the probability of a user clicking an ad is:

$p(X) = p(X\wedge Y) + p(X\wedge \neg Y)$

$p(X) = p(X|Y)p(Y) + p(X| \neg Y)p(\neg Y)$

$p(X) = p(X|Y)p(Y) + p(X| \neg Y)(1 - p(Y))$

If it's possible to obtain the information "new user" (event $Y$) in the system, then obviously there should be two distinct rankings, one for $Y$ and one for $\neg Y$. Phrased in a ML context: $Y$ is a binary feature and the predicted best advertiser depends on it. In this scenario a different advertiser would be selected depending on whether the user is new or not, here A for returning users and B for new users.

If not possible, then the advertisers should be ranked according to their overall probability $p(X)$, which depends on the probability of a new user $Y$. Based only on your initial values 10% vs. 9%, this means selecting advertiser $A$ is optimal overall.

To illustrate how this works we can calculate the detail based on the values provided. First we calculate $p(Y)$:

$p(X) = p(X|Y)p(Y) + p(X| \neg Y) - p(X| \neg Y)p(Y)$

$p(X)-p(X| \neg Y) = (p(X|Y)- p(X| \neg Y))p(Y) $

$(p(X)-p(X| \neg Y)) / (p(X|Y)- p(X| \neg Y)) = p(Y) $

We have $p(A)=0.1$, $p(A|Y) = 0.08$ and $p(A| \neg Y) = 0.3$, so we can derive $p(Y)=0.91$.

What happens in this example is that the high value $p(A| \neg Y) = 0.3$ makes up for
the low $p(A|Y) = 0.08$ despite $p(Y)=0.91$:

A: $0.91*0.08+0.09*0.3 = 0.1$
B: $0.91*0.09+0.09*0.09 = 0.09$

Of course this would be different if $p(A| \neg Y)$ was 0.15 instead of 0.3 or if $p(A|  Y)$ was 0.11 instead of 0.09, etc.

[edit based on comments by OP]

Indeed it seems that a more accurate representation of the problem would take into account the probability that a user clicks again (and again, and again...) on an ad.

This reminds me of classical probability exercises such as the Russian roulette game, the tree of possible outcomes is somewhat similar:

user doesn't click on the ad at all: $n=0$
user clicks on the ad at least once

user never clicks again: $n=1$
user clicks on the ad at least twice

user never clicks again: $n=2$
user clicks on the ad at least 3 times

user never clicks again: $n=3$
.......

So the goal would be to represent as accurately as possible the random variable $n$ which represents how many times a user clicks on an ad, and to use the expected value of $n$ (that is, how many times a user clicks on average) to rank the advertisers.

I'm guessing that this can get pretty sophisticated depending on how we choose to represent the problem. I'm probably not able to do anything sophisticated so I'll stick to a simple representation where I assume that there are two probabilities involved:

initial probability $F=p(n\geq 1)$ for a new user to click
probability for a user to click again after having clicked $m$ times: $G=p(n\geq m+1|n\geq m)$ for $m>0$.

This representation has the advantage to be very close to the original probabilities in the question: new user clicks and returning user clicks.
Notice that the model assumes that the probability to click again is a constant, which is unlikely in reality. Also the model doesn't take into account any other features, such as for how long a user hasn't clicked. I'll  just assume that one estimate these probabilities from a training set for the sake of simplicity.

Now let's calculate the expected value [fair warning, I might get it wrong!]

$p(n=0) = 1 - F$
$p(n=1) = F \times (1-G)$
$p(n=2) = F \times G \times (1-G)$
$p(n=3) = F \times G  \times G \times (1-G)$
...
$p(n=i) = F \times G^{i-1} \times (1-G)$

Therefore the expected value is:

$\sum_{i\geq 0} (p(n=i) \times i) = \sum_{i\geq 1} (F \times G^{i-1} \times (1-G) \times i)$

[Ok so normally now we can plug in the values for advertisers A and B. The problem is that I just calculated the result for the first few values and it doesn't really work... I probably made a mistake somewhere, or maybe the values don't represent the same thing... Anyway I hope this gives you the idea, maybe I'll give it another try later]


How to tune XGBClassifier's each and every hyper-parameter "efficiently"?

Its kind of difficult to avoid a large processing time when you have many parameters and a large sample.

A better method than gridsearch is Bayesian Optimization, this library even has done examples for xgboost ( https://github.com/fmfn/BayesianOptimization )
A good trick as well is to take a reduced sample of your data set, if you have over 2 million samples, you might acquire very similar results with smaller subsets (given that they have a similar distribution).


'int' object is not iterable using panda python

for time_range in range (len(data['time'])):
start = [datetime.strptime(t,'%H:%M:%S') for t in time_range]

time_range is an integer due to using range(int). There is nothing to iterate using an int. It is not a container.


How can I use multiple features in basic sentiment analysis in scikit-learn?

You can one-hot encode the gender column and append it to your tfidf table.
gender = pd.get_dummies(training_data['gender-column'])
X = text_tfidf.join(gender)


Paper proving neural networks with more data are better in NLP than traditional methods

Here's a blog post providing a view to the history of the process with several popular references in it. In particular, consider the Natural Language Processing section and its conclusion:

  On NLP tasks that have a long enough history to graph, there seems to be no clear indication that deep learning performs above trendline.

Also, worth noting from the general conclusion of the article:

  Deep learning provides discontinuous jumps relative to previous machine learning or AI performance trendlines in image recognition and speech recognition; it doesn’t in strategy games or natural language processing, and machine translation and arcade games are ambiguous (machine translation because metrics differ; arcade games because there is no pre-deep-learning comparison.)


Is Faster RCNN the same thing as VGG-16, RESNET-50, etc... or not?

Image Classification Models are commonly referred as a combination of feature extraction and classification sub-modules. Where the total model excluding last layer is called feature extractor, and the last layer is called classifier.
Popular Image Classification Models are: Resnet, Xception, VGG, Inception, Densenet and Mobilenet.

Object Detection Models are more combination of different sub-models rather than single end to end connected models, as you mentioned it is more like an architecture. Object detection model contains a feature extraction model, region proposal network, classification and regression models. Here the above mentioned classification models (Resnet50, VGG, etc) excluding all dense layers are used as a feature extractors.
Some of the popular object detection models are: Region-based Convolutional Neural Networks(R-CNN), Spatial Pyramid Pooling(SPP-net), Fast R-CNN, Faster R-CNN, Regression-based object detectors, YOLO(You only Look Once),Single Shot Detector(SSD).


Is there any different between feature selection and pca? If there is could anyone please kindly explain for me please?

PCA reduces dimensionality by generating orthogonal (i.e. uncorrelated) "new" features from the original features. This can be very useful, e.g. when you do linear regression. So principle components "transform" the original features to orthogonal ones. You can see this as some kind of feature engineering. Usually PCA will reduce the number of features which can be very helpful. See Ch. 10.2 in ISL.
Feature selection basically means to choose features which perform well in some model (e.g. have high explanatory power) or exclude features which are noisy. This can be any kind of features (transformed or original features). So feature selection is a very broad term and may be linked to various different methods.


Can this problem be solved using deep learning?

The problem is a case for "deep learning", I think, and I would start with boosting. I have good experiance with catboost (in Python). It can handle factors very well ("catfeatures", encoded as dummies) and it is relatively fast. Also lightgbm might be a good option. Also xgboost is an option.

I think data-wise one hot encoding and scaling is the first thing to do. Things beyond that is a bit try and error. The most important part might be that you tune your hyperparameter well (during boosting).

Catboost (Python): https://catboost.ai/docs/concepts/python-usages-examples.html
LightGBM (Python): https://lightgbm.readthedocs.io/en/latest/Python-Intro.html

Happy coding!


Classification of articles fetched from the web as describing stock (trending) increase or decrease

You can use a deep learning network to generate sentiment analysis. Pretrained models can be found which have been trained using vectorized word lists which are commonly associated with positivity and negativity. However, these may not generalize well to your specific use case.

Due to the historic nature of stocks you can easily compile your own database from online sources. For example, you can construct instances in your dataset as follows. Tokenize all the words found in articles pertaining to Google for February 5th 2001, then the target label can be the stock value for February 7th 2001. You can choose the time lag as you please. If you do this for a large amount of instances, your dataset should be able to contain some notion of what words found in articles will cause a stock price to rise or lower.


One scaler for all features or one scaler per feature?

Sklearn scaler works on feature/column (and thats why you want)

Imagine if it did not. Than you would shift your mean and std in weird-determined-by distribution-of-the-whole-set-kind of way.


Why is my loss blowing up after adding regularization

When you use regularization, your loss will be larger because you add the regularization term. So, it is normal if your best loss without regularization is lower than your best loss with regularization.
However, adding regularization should not affect convergence in the long-term. Meaning that even though your overall loss might be larger, it should decrease after some epochs.
I see that it increases for the first few epochs. I would suggest running your model for longer and see how the loss behaves. If an epoch takes long, use a small chunk of your dataset to investigate the behaviour of the loss. It may be possible that for the first few epochs, the loss doesn't start converging quite yet.


How to calculate events per day in R including dates when no events occurred?

I made the assumption that:

Events (All) =   Events (Female)  + Events (Male)

I also manipulated the data so that there are more than one event per day.

As a result, df looks like:

  Sex Age       Date
1   2  36 2004-01-05
2   1  47 2004-01-05
3   1  26 2004-01-10
4   2  23 2004-01-10
5   1  50 2004-01-27
6   2  35 2004-01-27
7   1  35 2004-01-30
8   1  38 2004-02-30
9   2  29 2004-02-30

The following code should achieve the desired results.

library(tidyr)
df = read.csv([Path to dataset], stringsAsFactors = FALSE)

df %>% group_by(Date, Sex) %>%
  summarise(sex_count = n()) %>%
  spread(Sex,sex_count, fill=0) %>%
  rename( event_female = '1', event_male = '2') %>%
  mutate(event_all = event_female + event_male ) %>%
  select(event_all, event_female, event_male)

output:

 Date       event_all event_female event_male
  <chr>          <dbl>        <dbl>      <dbl>
1 2004-01-05         2            1          1
2 2004-01-10         2            1          1
3 2004-01-27         2            1          1
4 2004-01-30         1            1          0
5 2004-02-30         2            1          1
```


It does, it's an API called Keras, written on top of TensorFlow.
With that you can rapidly prototype any "standard" NN architecture, such as feedforward NN, RNN, or CNN. With it you can export the trained model with one line of code and save it for later use. As for visualization, Keras offers integration with pydot (GraphViz) to plot the whole NN as a graph for you to inspect.


Your question is not clear in a way there are two different Graph Clustering problems. One is having a dataset of different graphs and you would like to cluster similar graphs (in this case each object is a graph), and the other when you have a graph (e.g. a social network) and you would like to group similar nodes inside that graph (here each object is a node).

The first problem needs Graph Embedding which is to transfer graphs into a n-dimensional manifold and from there on, you have a classical clustering problem. For this case have a look at what Horst Bunke has done.

A more simplified approach would be feature extraction from networks and apply classical clustering methods. These features are statistical and topological network measures such as Clustering Coefficient, Assortativity Index, Average Shortest Path Length, Density, Diameter, Average Centrality, etc. This approach is more about the characteristics of real-world networks e.g. biological networks or social networks as it takes into account those characteristics which determine the real-world phenomena.

The second problem is usually called Community Detection in literature. There are several method for doing that based on Modularity Score, Information Theory, Topological Structure or Spectral Graph Theory.


How to show value of a classification model even though it doesn't get the desired performance?

Often customers don't understand the value behind the performance of machine learning models. 60% precision might sound not so good, since 50% is often related to nearly random for binary problems. However, it might well be a very difficult problem, and 60% might well be very good performance.

One way you can show your customer that your model is indeed performing well, is to compare its precision against a human classification.

You can:

Define a random baseline, and explain why it is 35% and not 50% for your problem.
Get a couple of humans from your client side to try the task manually.
Let your model perform the task and show the improvement.

More often than not, humans perform tasks more precisely than models (if in small scale). So, if your model performs better, hopefully, they will see the value your model is adding to their company.


Why should the data be shuffled for machine learning tasks

Shuffling data serves the purpose of reducing variance and making sure that models remain general and overfit less.

The obvious case where you'd shuffle your data is if your data is sorted by their class/target. Here, you will want to shuffle to make sure that your training/test/validation sets are representative of the overall distribution of the data.

For batch gradient descent, the same logic applies. The idea behind batch gradient descent is that by calculating the gradient on a single batch, you will usually get a fairly good estimate of the "true" gradient. That way, you save computation time by not having to calculate the "true" gradient over the entire dataset every time.

You want to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. Shuffling your data after each epoch ensures that you will not be "stuck" with too many bad batches.

In regular stochastic gradient descent, when each batch has size 1, you still want to shuffle your data after each epoch to keep your learning general. Indeed, if data point 17 is always used after data point 16, its own gradient will be biased with whatever updates data point 16 is making on the model. By shuffling your data, you ensure that each data point creates an "independent" change on the model, without being biased by the same points before them.


How to estimate the mutual information numerically?

Binning

One easy way to do such an estimate is to put the continuous values into bins and obtain a discrete problem. Split up the domains of $X$ and $Y$ into bins and count the number of points that fall within each bin to obtain a density. So, the calculation would be:

$$
    \sum_{b_x \in Bins_x}
    \sum_{b_y \in Bins_y}
    \frac {\#(b_x, b_y)} N \log \frac {\frac {\#(b_x, b_y)} N}
      {{\frac {\#b_x} N} {\frac {\#b_y} N}}
$$

where $\#(b_x b_y)$ is the number of samples where $X \in b_x$ and $Y \in b_y$, $\#b_y$ is the number of samples where $Y \in b_y$ and $\#b_x$ is the number of samples where $X \in b_x$.

Entropy and Density Estimation

Another method is to note that the mutual information can be represented as

$$
    I(X, Y) = H(X) + H(Y) - H(X,Y)
$$

so if you can estimate entropy you can estimate information. Now, in order to find the expectation of a function over your data you can just use the plugin estimator and do

$$
    E[g(X)] = {\frac 1 n} \sum_i g(X_i)
$$

The problem here is that we want to estimate the function $\log p(x)$ where $p$ depends on our data.

So one can use an estimate of the density of $p$ for each point and then use the plugin estimator. Kernel density estimators are one approach and nearest neighbor estimators are another. Both methods are non-parametric.


How to use inverse_transform in MinMaxScaler for pred answer in a matrix

From my understanding you are working on a regression task in which you have applied MainMaxScaler to your target variable y prior modeling.
If so you have two options:

As the error message suggests, you can reshape the output with array.reshape(-1, 1)

Scikit learn has implemented a class to work with transformations on target:

So just try
from sklearn.svm import SVR
from sklearn.compose import TransformedTargetRegressor
from sklearn.metrics import r2_score
from sklearn.preprocessing import MinMaxScaler

regressor = SVR(kernel = 'linear')

model = TransformedTargetRegressor(regressor= regressor,
                                        transformer = MinMaxScaler()
                                        ).fit(trainX,trainY)

pred = model.predict(testX)
print(pred)

Every time you make a model.predict(X) it will apply an inverse transformation so that your predictions are in the same scale as prior MinMaxScaler
EDIT:
Working example of transformation without using Scikit-learn
# array example is between 0 and 1
array = np.array([0.58439621, 0.81262134, 0.231262134, 0.191])
#scaled from 100 to 250
minimo = 100
maximo = 250
array * minimo + (maximo - minimo)

Returns:
array([208.439621 , 231.262134 , 173.1262134, 169.1])


When would one use Manhattan distance as opposed to Euclidean distance?

According to this interesting paper, Manhattan distance (L1 norm) may be preferable to Euclidean distance (L2 norm) for the case of high dimensional data.
The authors of the paper even go a step further and suggest to use Lk norm distances, with a fractional value of k, for very high dimensional data in order to improve the results of distance-based algorithms, like clustering.


How to plot multiple bar charts with different infill and outline?

Use (more information in the docs):
import matplotlib.pyplot as plt
import numpy as np

# example data
x = list(range(14))
y1 = np.arange(14)
y2 = np.arange(14)[::-1]
y3 = [8 for x in range(14)]

# Interesting part
fig, ax = plt.subplots()

ax.bar(x, y1, color="red",  edgecolor="None", linewidth=2, label='1')
ax.bar(x, y2, color="None", edgecolor="blue", linewidth=2, label='2')
ax.bar(x, y3, color="None", edgecolor="green", linewidth=2, label='3')
ax.legend()

plt.show()


How to find the count of consecutive same string values in a pandas dataframe?

Break col1 into sub-groups of consecutive strings. Extract first and last entry per sub-group.

Something like this:

df = pd.DataFrame({'col1':['A>G','C>T','C>T','G>T','C>T', 'A>G','A>G','A>G'],'col2':['TCT','ACA','TCA','TCA','GCT', 'ACT','CTG','ATG'], 'start':[1000,2000,3000,4000,5000,6000,10000,20000]})

df['subgroup'] = (df['col1'] != df['col1'].shift(1)).cumsum()

  col1 col2  start  subgroup
0  A>G  TCT   1000         1
1  C>T  ACA   2000         2
2  C>T  TCA   3000         2
3  G>T  TCA   4000         3
4  C>T  GCT   5000         4
5  A>G  ACT   6000         5
6  A>G  CTG  10000         5
7  A>G  ATG  20000         5

df.groupby('subgroup',as_index=False).apply(lambda x: (x['col1'].head(1),
                                                        x.shape[0],
                                                        x['start'].iloc[-1] - x['start'].iloc[0]))

0        ([A>G], 1, 0)
1     ([C>T], 2, 1000)
2        ([G>T], 1, 0)
3        ([C>T], 1, 0)
4    ([A>G], 3, 14000)

Tweak as needed.


Accuracy goes straight for about 200 epochs then start increasing

The accuracy depends on a threshold, whereas the loss doesn't. ML software tends to assume a threshold of 0.5, which is not a good fit in cases where there's some class imbalance.
I believe that, until epoch 500, your model is learning (loss is going down), but the default threshold doesn't allow you to see it in terms of accuracy.
If you pick another threshold you might see different results.
Regarding the loss going from smooth to noisy, it might be that it is learning the "easy cases" first, and then decreasing very easily and smoothly, and after epoch 400-500 it starts to overfit some of "hard cases", thus the loss becoming noisier.


Handling imbalanced data by deleting over represented rows vs. adding under represented rows

Technically what I do, is create a 50-50 dataframe, which has all of the under-represented rows and randomly selected over-represented rows.

You could use a random seed for the selection from the non-frauded cards dataset so that every time it selects the same rows. This way you are able to track and compare the results.

import numpy as np
from numpy import random

arr = np.arange(5)  # [0, 1, 2, 3, 4]
random.seed(1)  # Reset random state
random.shuffle(arr)  # Shuffle!
print(arr)

Just an example on how to use random seed from numpy, you'll have to come up with your own way to use it for your need (suggestion: create arr in range(0, len(frauded_card)) and shuffle it, then use arr as indexes for the frauded_card sample to use)


Deep neural networks for mostly zero valued data

There's this misunderstanding that deep learning is generally suitable if you have loads of data, which, judging from your comment, is your case. This is generally an inaccurate belief.

Deep learning (including CNN and RNN), are complex models with thousands of parameters that are able to model complex relationships. Such relationships are generally "hidden" in vast amounts of data, but this is not always the case. You may have at hand data that are generated from a simple distribution, and as such it will not need a complex model to approximate, even if your sample size is huge.

Here's a fabricated example: Let's say that you have the function y=a*x1 + b*x2 + c*x3 + d*x4. This function entails linear relationships between all independent variables and the dependent variable, y. You could sample this function a million times and still you would have at hand data that could be approximated through linear regression.

Coming to your case: to identify what kind of algorithm you would need, you first need to look at your data, perhaps through performing a statistical analysis. Then I would suggest to start simple. Try logistic regression, for starters. Is the model satisfactory in validation? If not, try perhaps decision trees. Or a shallow neural net. Always validate (you have loads of data so validation should be easy!). My (admittedly wild) guess is that your classification problem could be addressed with much simpler algorithms than DNN. But YMMV, of course.

FWIW, here's another answer similar to your case. And another one.


Deep Neural Network - Backpropogation with ReLU

Working definitions of ReLU function and its derivative:

$ReLU(x) = \begin{cases}
  0, & \text{if } x < 0, \\
  x, & \text{otherwise}.
\end{cases}$

$\frac{d}{dx} ReLU(x) = \begin{cases}
  0, & \text{if } x < 0, \\
  1, & \text{otherwise}.
\end{cases}$

The derivative is the unit step function. This does ignore a problem at $x=0$, where the gradient is not strictly defined, but that is not a practical concern for neural networks. With the above formula, the derivative at 0 is 1, but you could equally treat it as 0, or 0.5 with no real impact to neural network performance.

Simplified network

With those definitions, let's take a look at your example networks.

You are running regression with cost function $C = \frac{1}{2}(y-\hat{y})^2$. You have defined $R$ as the output of the artificial neuron, but you have not defined an input value. I'll add that for completeness - call it $z$, add some indexing by layer, and I prefer lower-case for the vectors and upper case for matrices, so $r^{(1)}$ output of the first layer, $z^{(1)}$ for its input and $W^{(0)}$ for the weight connecting the neuron to its input $x$ (in a larger network, that might connect to a deeper $r$ value instead). I have also adjusted the index number for the weight matrix - why that is will become clearer for the larger network. NB I am ignoring having more than neuron in each layer for now.

Looking at your simple 1 layer, 1 neuron network, the feed-forward equations are:

$z^{(1)} = W^{(0)}x$

$\hat{y} = r^{(1)} = ReLU(z^{(1)})$

The derivative of the cost function w.r.t. an example estimate is:

$\frac{\partial C}{\partial \hat{y}} = \frac{\partial C}{\partial r^{(1)}} = \frac{\partial}{\partial r^{(1)}}\frac{1}{2}(y-r^{(1)})^2 = \frac{1}{2}\frac{\partial}{\partial r^{(1)}}(y^2 - 2yr^{(1)} + (r^{(1)})^2) = r^{(1)} - y$

Using the chain rule for back propagation to the pre-transform ($z$) value:

$\frac{\partial C}{\partial z^{(1)}} = \frac{\partial C}{\partial r^{(1)}} \frac{\partial r^{(1)}}{\partial z^{(1)}} = (r^{(1)} - y)Step(z^{(1)}) = (ReLU(z^{(1)}) - y)Step(z^{(1)})$

This $\frac{\partial C}{\partial z^{(1)}}$ is an interim stage and critical part of backprop linking steps together. Derivations often skip this part because clever combinations of cost function and output layer mean that it is simplified. Here it is not.

To get the gradient with respect to the weight $W^{(0)}$, then it is another iteration of the chain rule:

$\frac{\partial C}{\partial W^{(0)}} = \frac{\partial C}{\partial z^{(1)}} \frac{\partial z^{(1)}}{\partial W^{(0)}} = (ReLU(z^{(1)}) - y)Step(z^{(1)})x = (ReLU(W^{(0)}x) - y)Step(W^{(0)}x)x$

. . . because $z^{(1)} = W^{(0)}x$ therefore $\frac{\partial z^{(1)}}{\partial W^{(0)}} = x$

That is the full solution for your simplest network.

However, in a layered network, you also need to carry the same logic down to the next layer. Also, you typically have more than one neuron in a layer.

More general ReLU network

If we add in more generic terms, then we can work with two arbitrary layers. Call them Layer $(k)$ indexed by $i$, and Layer $(k+1)$ indexed by $j$. The weights are now a matrix. So our feed-forward equations look like this:

$z^{(k+1)}_j = \sum_{\forall i} W^{(k)}_{ij}r^{(k)}_i$

$r^{(k+1)}_j = ReLU(z^{(k+1)}_j)$

In the output layer, then the initial gradient w.r.t. $r^{output}_j$ is still  $r^{output}_j - y_j$. However, ignore that for now, and look at the generic way to back propagate, assuming we have already found $\frac{\partial C}{\partial r^{(k+1)}_j}$ - just note that this is ultimately where we get the output cost function gradients from. Then there are 3 equations we can write out following the chain rule:

First we need to get to the neuron input before applying ReLU:

$\frac{\partial C}{\partial z^{(k+1)}_j} = \frac{\partial C}{\partial r^{(k+1)}_j} \frac{\partial r^{(k+1)}_j}{\partial z^{(k+1)}_j} = \frac{\partial C}{\partial r^{(k+1)}_j}Step(z^{(k+1)}_j)$

We also need to propagate the gradient to previous layers, which involves summing up all connected influences to each neuron:

$\frac{\partial C}{\partial r^{(k)}_i} = \sum_{\forall j} \frac{\partial C}{\partial z^{(k+1)}_j} \frac{\partial z^{(k+1)}_j}{\partial r^{(k)}_i} = \sum_{\forall j} \frac{\partial C}{\partial z^{(k+1)}_j} W^{(k)}_{ij}$

And we need to connect this to the weights matrix in order to make adjustments later:

$\frac{\partial C}{\partial W^{(k)}_{ij}} = \frac{\partial C}{\partial z^{(k+1)}_j} \frac{\partial z^{(k+1)}_j}{\partial W^{(k)}_{ij}} = \frac{\partial C}{\partial z^{(k+1)}_j} r^{(k)}_{i}$

You can resolve these further (by substituting in previous values), or combine them (often steps 1 and 2 are combined to relate pre-transform gradients layer by layer). However the above is the most general form. You can also substitute the $Step(z^{(k+1)}_j)$ in equation 1 for whatever the derivative function is of your current activation function - this is the only place where it affects the calculations.

Back to your questions:

  If this derivation is correct, how does this prevent vanishing?

Your derivation was not correct. However, that does not completely address your concerns.

The difference between using sigmoid versus ReLU is just in the step function compared to e.g. sigmoid's $y(1-y)$, applied once per layer. As you can see from the generic layer-by-layer equations above, the gradient of the transfer function appears in one place only. The sigmoid's best case derivative adds a factor of 0.25 (when $x = 0, y = 0.5$), and it gets worse than that and saturates quickly to near zero derivative away from $x=0$. The ReLU's gradient is either 0 or 1, and in a healthy network will be 1 often enough to have less gradient loss during backpropagation. This is not guaranteed, but experiments show that ReLU has good performance in deep networks.

  If there's thousands of layers, there would be a lot of multiplication due to weights, then wouldn't this cause vanishing or exploding gradient?

Yes this can have an impact too. This can be a problem regardless of transfer function choice. In some combinations, ReLU may help keep exploding gradients under control too, because it does not saturate (so large weight norms will tend to be poor direct solutions and an optimiser is unlikely to move towards them). However, this is not guaranteed.


Why activation function is not needed during the runtime of an Word2Vec model

From this StackOverflow Question

  while no activation is explicitly formulated, we could consider it to
  be a linear classification function. It appears that the dependencies
  that the word2vec models try to model can be achieved with a linear
  relation between the input words.

  Adding a non-linear activation function allows the neural network to
  map more complex functions, which could in turn lead to fit the input
  onto something more complex that doesn't retain the dependencies
  word2vec seeks.


Wanting to overfit model to say if data has been seen before

I'm not sure if a machine learning approach is a good choice for such a task. I am tempted to say no. The reason is mainly, that ML usually is used for the discovery of patterns or in other words of similarities while they are designed to generalize so they still recognize a pattern even if it looks different due to some randomness (noise etc.).

In your case however it sounds more like you want to discover exact matches.

You also have to consider, that you cannot store your data in an arbitrarily compact form without reducing information. So even if you would find a suitable ML model for your purpose, you cannot expect it to consume much less space as the part of the database for which you want to find matches in the other database.


You don't need to remove your features from your data set you may just drop the features which you don't want considered as features or lables.  Dropping a feature or label works by excluding the columns while fitting the data into the training set.


What should I do with a test and train set not having same classes?

Well, what you should do depends highly on what your goal here is.
Do you - for whatever reason - have to use the given test and training sets as such?
If not, a valid approach would be to shuffle the available data and determine a new test and training set, so that they each contain all classes.
Do you want a classifier only for d and e, and have to use the given test and training sets?
Then nothing really speaks against just training on the training set with all classes and testing only the ones you are interested in/ have test data for. Of course, then you can not make statements about the accuracy for other classes.
You could also try removing the other classes from the training set, if they won't appear in your application, which may or may not make the classifier perform better on the test data.
Essentially, it's not easy to provide a ceratin answer without having more context about why you have this kind of data, and what your goal is.
It would be great if you can provide a few more details, so we can work on a more fitting answer.


You could try also  R  h2o

You  have implementations  for   random forest,   gbm , linear models and also   deep  learning .
THe  advantage is  that    h2o is pretty   fast  compared  to   other  implementations


The standard textbooks that covers AI is "Artificial Intelligence: A Modern Approach" by Russel & Norvig. The book's website can be found here.

I also recommend "Artificial Intelligence: Foundations of Computational Agents" by Poole and Mackworth. The book can be read online.


Find the Weight Values using Linear Regression Analysis in Python

Try the scikit-learn library.

From the documentation for linear regression:
"LinearRegression fits a linear model with coefficients w = (w1, ..., wp) to minimize the residual sum of squares between the observed targets in the dataset, and the targets predicted by the linear approximation."

https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html


How to split train/test in recommender systems

Leave-one-out cross validation is probably the most straight-forward way to address this.  If you happen to be using a model that requires a lot of time to train, then leave n% out might be more appropriate.
The method involves eliminating one known rating and trying to predict it.  If you want to remove n percent of the ratings, just choose them randomly rather than choosing a specific number of every user's ratings.  And keep n pretty small - on the order of 10% or less.
Here is a good treatment of cross validation methods for recommender systems, An Evaluation Methodology for Collaborative Recommender Systems.  Generally:

Holdout is a method that splits a dataset into two parts: a training
set and a test set. These sets could have different proportions. In
the setting of recommender systems the partitioning is performed by
randomly selecting some ratings from all (or some of) the users. The
selected ratings constitute the test set, while the remaining ones are
the training set. This method is also called leave-k-out. In [17],
Sarwar et al. split the dataset into 80% training and 20% test data.
In [18] several ratios among training and test (from 0.2 to 0.95 with
an increment of
0.05) are chosen and for each one the experiment is repeated ten times with different training and test sets and finally the results are
averaged. In [13] the test set is made by 10% of users: 5 ratings for
each user in the test set are withheld.
Leave-one-out is a method
obtained by setting k = 1 in the leave-k-out method. Given an active
user, we withhold in turn one rated item. The learning algorithm is
trained on the remaining data. The withheld element is used to
evaluate the correctness of the prediction and the results of all
evaluations are averaged in order to compute the final quality
estimate. This method has some disadvantages, such as the overfitting
and the high computational complexity. This technique is suitable to
evaluate the recommending quality of the model for users who are
already registered as members of the system. Karypis et al. [10]
adopted a trivial version of the leave-one-out creating the test set
by randomly selecting one of the non-zero entries for each user and
the remaining entries for training. In [7], Breese et al. split the
URM in training and test set and then, in the test set, withhold a
single randomly selected rating for each user.
A simple variant of the
holdout method is the m-fold cross-validation. It consists in dividing
the dataset into m independent folds (so that folds do not overlap).
In turn, each fold is used exactly once as test set and the remaining
folds are used for training the model. According to [20] and [11], the
suggested number of folds is 10. This technique is suitable to
evaluate the recommending capability of the model when new users
(i.e., users do not already belong to the model) join the system. By
choosing a reasonable number of folds we can compute mean, variance
and confidence interval.

Hope this helps!


Why are results without Transfer Learning better than with Transfer Learning?

As @fuwiak mentioned, transfer learning may not work if pre-trained model has been fitted on a "very different" dataset. Typically if the pre-trained network extract information that is not relevant for your problem.
Moreover, in the paper License Plate Recognition System Based on Transfer Learning (that you shared with me), they have tried to freeze some layers of a pretrained Xception (based on ImageNet weights) to see the impact on the training. They conclude that ImageNet data and license plate data are too different to freeze layers. So your results are confirmed.
Now changing efnB0_model.trainable = False to True would allow the pre-trained network to update and to be more relevant to your problem. Generally, if you don't have time issues, it seems to be always better (see this post). Will it give better results than initialize the weights randomly ? I think one can presume but cannot know.


How to normalize test data according to the training data if the normalization on the training data is performed row wise?

You usually want to normalize features as you also pointed out. In case of tabular data almost every machine learning implementation will expect you to provide the features as columns and observations as rows. In your case if you have a feature in a row you may want to transpose it or if they are not the same features than you may apply a different transformation.

If you do normalization on features you have to use the same transformation on test data that you used on train data. (if you use i.e. sklearn implementations they will take care of it for you)


Given a target vector and a feature vector, how to computer the weight

The least square problem is to minimize $$\|Xw-t\|^2$$

Differentiating it with respect to $w$ and equating it to $0$, we have

$$2X^T(Xw-t)=0$$

Hence, we have

$$X^TXw-X^Tt=0$$

That is $$X^TXw=X^Tt$$

$$w=(X^TX)^{-1}X^Tt$$


Can a Logsitic Regression model continue making predictions after removing predictions from the data set?

The model is able to predict churn on any sample. The issue is that increasing the „customer life“ by 30 days does not lead to any change in the prediction (based on what the model has learned).

I think you can check two things:

A) More of a fundamental thing, try to improve the model, e.g. by applying regularization to get a better fit on you features. GLMNET is the thing to go for in this case: https://web.stanford.edu/~hastie/glmnet_python/

B) Check how „sensitive“ your predictions are to changes in features. You could plot or look at predictions in case you increase „customer life“ by say 10, 20, 30, ..., 100 or so days. Remember that you basically predict the probability of churn. So it is possible that (some) probabilities at 30 days of customer life are just below 50%. If you gradually look at increases of customer life, you get a good idea of how churn changes in X. This is kind of a marginal effect (dy/dX).


You can do the following:
A Pipeline can be made of other pipelines. Isn't that great?
A Pipeline inherit from the Estimator class and by definition, a PipelineStage can be either an Estimator or a Transformer.
This way, you can build smaller pipelines, save them separately and on the other software/class, join them again as a single one and call transform on the DataFrame.


bias variance decomposition for classification problem

My opinion is that the bias variance trade off is rooted in the Uncertainty principle. It behaves absolutely the same.


Extract company names/job titles from free text

I think what you want is to extract company names from "Job Title". In natural language process, we call this kind of research as "Name Entity Recognition(NER)". You can try to use  Stanford Named Entity Recognizer (NER)[http://nlp.stanford.edu/software/CRF-NER.shtml]. Stanford NER performs very well on English contents and there are lots packages for many programming language:

  UIMA: Florian Laws made a Stanford NER UIMA annotator using a modified version of Stanford NER, which is available on his homepage. [Old version.]

  Perl: Kieren Diment has written Text-NLP-Stanford-EntityExtract, a Perl module that provides an interface to Stanford NER running as a server.

  Ruby: tiendung has written a Ruby Binding for the Stanford POS tagger and Named Entity Recognizer.

  Python: Dat Hoang wrote pyner, a Python interface to Stanford NER. [Old version.] NLTK (2.0+) contains an interface to Stanford NER written by Nitin Madnani: documentation (note: set the character encoding or you get ASCII by default!), code, on Github.

  F#/C#/.NET: Sergey Tihon has ported Stanford NER to F# (and other .NET languages, such as C#), using IKVM. See also pages on: GitHub and NuGet.

  PHP: PHP-Stanford-NLP. Supports POS Tagger, NER, Parser. By Anthony Gentile (agentile).

If you are not satisfied with the performance of Stanford NER, you can also train you own models to extract company names by crawl company names from several popular sites with company names, such as Linkedin/Facebook/Glassdoor...etc