How to implement "one-to-many" and "many-to-many" sequence prediction in Keras?

The point of using any recurrent layer is to have the output be a result of not only a single item independent of other items, but rather a sequence of items, such that the output of the layer's operation on one item in the sequence is the result of both that item and any item before it in the sequence. The number of timesteps defines how long such a sequence is. That is, how many items that should be handled in a sequence, and affect each other's resulting output.

A LSTM layer operates in such a way that it accepts input on the form number_of_timesteps, dimensions_of_each_item. If the parameter return_sequences is set to False, which it is by default, the layer "compounds" the inputs of all the timesteps into a single output. If you consider a sequence of, say 10 items, a LSTM layer with return_sequences set to False will from such a sequence produce a single output item, and the attributes of this single item will be a result of all the items (timesteps) in the sequence. This is what you want in the case of a many-to-one design.

A LSTM layer with return_sequences set to True will for each item (timestep) in an input sequence produce an output. This is done in such a way that at any timestep, the output will depend not only on the item that is currently being operated on, but also the previous items in the sequence. This is what you want in the case of a many-to-many design.

As a LSTM layer takes a sequence of items as input, any layer before a LSTM layer in your model will need to produce a sequence as an output. In the case of your Type 1 model, the first few layers do not operate on sequences, but rather a single item at a time. This does hence not produce a sequence of items on which to operate for the LSTM.

Using TimeDistributed makes it possible to have a layer operate on every item in a sequence without the items affecting each other. TimeDistributed layers thus operate on sequences of items, but there is no recursion.

In the case of your type 2 model, the first layers will produce a sequence 5 timesteps long, and the operations done on each of the items in the sequence will be independent of each other, since the layers wrapped in TimeDistributed are non-recurrent. As the LSTM layer uses the default settings, return_sequences = False, the LSTM layer will produce a single output for each such sequence of 5 items.

The final number of output nodes in your model depends completely on the use case. A single node is suitable for something like binary classification or for producing some sort of score.


Model the predictive relationship between images

To answer my own question. A naïve approach is to train a convolutional neural net with an output layer of 4096 neurons $x_1,x_2,\ldots, x_{4096}$. Assume values in the response image are $y_1,y_2,\ldots,y_{4096}$. Let the loss function be $L=\sum_{i=1}^{4096}(x_i-y_i)^2$.


what is difference between fit and fit_transform in sklearn while applying feature scaling

In real life problems you may not not have the actual “test data” to fit on, for example in some time series forecasting problems. My recommendation if you want to keep safe and avoid data leakage is to fit on training data and transform on test.


Determining usage/context of terms/acronyms/abbreviations in python

After rubber ducking with a friend of mine, I decided to take a two stage approach.

First, whenever I see a word I've recognized as a multiple use word, I flag it in the system and give it the appropriate correction. Once it evaluates that part, it stores the corrected word and enough words such that there is sufficient context in a tuple... inside a list... inside a dictionary. It all looks something like use_case = {'abbr': ('five word abbr context string', 'correction').

Secondly, I've created a dictionary of scikit-learn LinearSVCs that looks like svm_dict = {'abbr': LinearSVC}.

Once these things are established and the program is run again, when it finds one of the multiple usage words it first checks use_case to see if that exact set of 5 words surround this case. If it does it swaps it out for the original correction, otherwise it tries to predict the usage with the appropriate SVC and asks the user if the prediction is correct.

I know it's not fully automated, but neither is the program in which it runs. I figured the occasional yes/no question to the user was easier than leaving it off the lists and rewriting it every... single... time...

Either way, I have it all in place but haven't gotten to the point where it's being used yet. So, I'll either be editing, deleting, or accepting this answer sometime next week when I've had some time to test'r out.


Can I create a layer with multiple rnn cell ? [question about a paper]

Not exactly familiar with this model architecture but, I don't think it works the way you described it.
From a quick skim, a simplified overview of the model may be

data input: $X$ multiple time series
RNN: pass each time series $x$ through a recurrent neural network and grab its hidden state
Joint embedding: Concat hidden states from each $x$ series into an embedding with $X$ channels
Convnet

But I didn't see there is a way to create multiple rnn cells in same layer , do I misunderstand?? If not , could you please give me a guide or an example ?

Essentially, any rnn will have a number of cells equal to the number of time steps. As an example, in NLP that number of cells would be equal to the number of words in a sentence, where each sentence in your trainset must have the same length.
I suggest you familiarise yourself with sequence models to better understand the concept of this paper.


Which outlier detection can detect these outliers?

You may view your data as a time series where an ordinary measurement produce a value very close to the previous value and a re-calibration produce a value with a large difference to the predecessor.

Here are simulated sample data based on normal distribution with three different means similar to your example.

By calculating the difference to the previous value (a sort of derivation) you get the following data:

My interpretation of your description is, that you tolerates re-calibration (i.e. points on greater distance from zero, red in the diagram), but they must interchange between positive and negative values (i.e. corresponding to shift from blue state to the yellow one and back).

This means you can set up an alarm seeing a second one red point either on the negative or positive side.


Custom Lambda layer Keras outputs predictions. I get 'An operation has `None` for gradient' error

I think it comes from your 'custom_layer'. Not all functions defined in keras backend are differentiable (as the error says, for example K.argmax) if you use these functions on your layer and they don't have a defined gradient, it will rise this error. You have to make sure that you use only functions that has a gradient associated. Although there is no an official list of these functions, basic operations (+/-*) are, for example equal_toor greaterare not differentiable.


Which animal has similar brain complexity to VGGNet

According to this wiki: List of animals by number of neurons

The closest ones you got are:

Pond snail with 11,000 neurons.
Sea slug with 18,000 neurons.

I think this was more of a biology question since you covered the data science part.


Linear discriminant analysis in R: how to choose the most suitable model?

I'm not familiar with LDA, but as far as I know you're not really changing the "model" (i.e. the way to measure impact) between the two versions, what you're changing is the features: in the 2nd version, instead of looking at whether the value of the feature impacts Y, you look at whether the log of the value of the feature impacts Y.
The first version is of course the most natural way to look at features, the second is common but usually this is used when we already know that the distribution of the feature (or the relation between the feature and the response variable) makes it relevant.


Solving classification task with deep network

If you are using two hidden layers, it may mean that your data is not linearly separable. If you just use one unit in a single hidden layer then you can claim that your data is linearly separable, which in your case you may say that there exist a hyper-plane that separates your data linearly.

As the answer to your question, yes. It is possible to reduce the amount of error by adding layers or adding extra units to the current architecture but there are some points that are necessary to be considered.

Whenever you add more layers, there will be vanishing and exploding gradients which may cause your network not to learn, or learning may happen so slowly. To avoid, you should use ReLU activation in order to avoid saturation of gradients. Moreover you have to use He or Xavier initialization techniques to avoid having bad random weights. There are other techniques for solving this problem which are called skip connections but at least I've never seen the use of them in MLPs Although they are really helpful for solving the mentioned problem.
Covariat shift is the problem of learning for deeper layers. As a solution you have to use Batch Normalization to somehow normalize the activations of the deeper layers.
Overfitting is a problem that happens for large architectures that are not fed with enough training data. You should use regularization techniques. There are so many papers about this but in your case which is using MLP I highly recommend you using Dropout technique which is invented by the so called God Father of deep learning.


Whenever you have features that they have different scale and it is significant for some features, you should standardize your feature. Take a look at here.


Spark Scala alternative Machine Learning Library?

The way you can use scikit-learn is basically broadcasting your data to the workers and then do different folds of crossvalidation or different parameter settings in your grid_search on different workers. That is all that the scikit-learn package in pySpark does as far as I know. This is similar to a normal mapping. Implementing this should be relatively easy for any Machine Learning library in Scala (I do not use Scala, so I cannot help you with suggestions in that regard)


Questions for data analysis of a class room

Very difficult question for someone to answer as ultimately your examples and many questions contain certain constraints you won't get from your sensors.
For example:
Your first question is predicting "optimum" time to turn on AC.  who defines optimum?  Optimum for the classroom at full capcity, half capacity?
Your second question looks like it one, requires a map of the classroom and where the sensors are, two you won't be able to predict 7 sensor values from 2 sensor values, I don't know why you would want to do this.
Your third questions is a social sciences problem.  Define "light needed to be comfortable".  I imagine you would get 150 answers if you had 150 students.

Your scenario lacks context/comparison.  Let's say you had sensors outside the classroom for your various data points.  Then you could do things like, can I predict the temp inside the room based on my other room and external data points.

Finally, yes machine learning can be used with your wireless sensors.


To be able to compare strings / Words or documents the data needs to be converted to a format the computer understand, vectors.
Google has a nice guide on Universal Sentence Encoder for sentence similarity that you can follow which explains how to generate a vector from the neural network that they have already trained.
What they do and what you can try is to use cosine similarity to compare the vectors. Compare one sentence vector with all other vectors and find the closest.
I am adding Annoy here, because I find it quite intuitive to use and you can get the n closest items.


Why compressed image size is greater than original one in kmeans algorithm?

Your are saving the same color format... if they are 24bits RGB you are currently saving also in 24 bits RGB. You should convert to RGB-8 that uses 8 bits per pixel.

I have used pypng and PIL to get the following result

from PIL import Image
import png

    s = []
    for i in range(rows):
        s.append(tuple(labels[i].astype(int)))

    palette = []
    for i in range(ncolor):
        c = (centers[i][0].astype(int), centers[i][1].astype(int), centers[i][2].astype(int))
        palette.append(c)

    w = png.Writer(width=cols, height=rows, palette=palette, bitdepth=8)
    f = open('tiger8bits3.png', 'wb')
    w.write(f, s)

Look at PNG formats: http://www.patrickhansen.com/2011/02/04/png-8-24-32-what/

The result is 269KB:


Assessing significance / confidence of a crossvalidated performance measure

You may use bootstrapping to estimate a confidence interval for the prediction error. Further help can be found in some Stanford online course slides, but I haven't done this.

Besides that, it should be no problem to compare the estimated performance obtained with cross-validation (mean and standard deviation) with a reference point (e.g., AUC = 0.5) or with the results of another benchmark model such as logistic regression or nearest-neighbor classifier using a simple statistical test using a given level of confidence.

A similar question is discussed here.


Manually creating plants images dataset for machine learning plant type classification and leaf segmentation

Well, it all depends on what you plan to accomplish with this dataset:

If you are planning on using drones to identify crops in the wild then you should use a drone to take the images and get images in the wild. (By in the wild I mean a free environment that differs from a lab or something like it)
If you are planning on monitoring plants on a constrained environment using fixed cameras so you should gather images on this conditions.

As Rajat pointed, the camera you choose needs to be able to grab all the feature needed to the identification, maybe machine learning approaches can identify plants with way less features than human eye does but maybe it can not. Either way you should consult with someone with expertise on plants to decide what method you gonna use.

Numbers can't hurt:

Well, they can give you some extra work but the more the merrier. One thing you need to keep in mind is that datasets must be representative enough to enable learning methods to work on the real world scenario. For example, if plan on training a algorithm to classify images token from above there is no reason to train it with images from under the plants.

If you plan on using deep learning, (i.e. you have the time and resources to train it) you should gather a really large number of images or find a way to augment them with meaningful transformations. One way is to use simulated images, from a computer rendering (that is used in UnityEyes for example) and maybe augment it to be more realistic using GANs.

In short terms, my advice would be to formalize what you want this dataset for, consider what restrictions you can guarantee on the test environment and this should give you a list of what you need to create your training dataset.


with easy if you know a little Python.
According to
https://gist.githubusercontent.com/pootzko/ac34906e11d9715d9514c75507c24dc0/raw/1c28b679d1ca23da56a930950f21a82dba379cd0/ip_generator.py

def ipRange(start_ip, end_ip):
   start = list(map(int, start_ip.split(".")))
   end = list(map(int, end_ip.split(".")))
   temp = start
   ip_range = []

   ip_range.append(start_ip)
   while temp != end:
      start[3] += 1
      for i in (3, 2, 1):
         if temp[i] == 256:
            temp[i] = 0
            temp[i-1] += 1
      ip_range.append(".".join(map(str, temp)))

   return ip_range

output = []
# sample usage
ip_range = ipRange("192.168.1.0", "192.171.3.25")
for ip in ip_range:
    output.append(ip)
fo = open("output.txt", "rw+")
line = fo.writelines(output)
fo.close()


Image classification problem using convolutional neural networks

In the __getitem__ method you should return both the image and the label, in your current example you are trying to get the image by indexing the img_dir variable or your class, which won't work since this is of type Path. The general steps for the __getitem__ method is that you should (1) get the path of image, (2) load the image and (3) convert the image to a tensor for pytorch to use. This would something like this:
from torch.utils.data import Dataset
import torchvision.transforms as transforms
from pathlib import Path
import pandas as pd
from PIL import Image

class LesionDataset(Dataset):
    def __init__(self, img_dir, labels_fname):
        self.img_dir = Path(img_dir)
        self.img_paths = self.img_dir.glob("*")
        self.labels_fname = pd.read_csv(labels_fname)

    def __len__(self):
        return len(self.labels_fname)

    def __getitem__(self, idx):
        img_path = self.img_paths[idx]
        image = Image.open(img_path)
        image = transforms.ToTensor()(image)
        label = self.labels_fname[idx]
        return image, label


Can someone explain the lambda collocation metric?

I will try to illustrate the lambda collocation metric, first we have to define a function c(x, z) that receives two K-word expressions where x is the K-word expression you want to score and z corresponds to all possible values of a K-word expression in the corpus. c(x, z) returns a binary vector (j1, ..., jk) such that ji = 1 if xi == zi, 0 otherwise. Now a binary vector can represent a decimal number up to  2^K = M, so c(x, z) = ci means that the binary vector returned was equal to the binary representation of the decimal number i. Now as it says on the documentation ni corresponds to the number of times the c(x, z) function was equal to ci through the corpus. Now an example, consider the following sentence:

  this sentence is an example for this sentence

The target expression is "this sentence" so K=2. You have the following possible values for z:

this sentence
sentence is
an example
example for
for this
this in
in this
this sentence

Given that K=2, the possible values for c(x, z) are 00 = 0, 01 = 1, 10 = 2 and 11 = 3. Then considering the target expression and all the values of z above you have the following, (for x = "this sentence"):

c(x, this sentence) = 11 = c3
c(x, sentence is) = 00 = c0
c(x, an example) = 00 = c0
c(x, example for) = 00 = c0
c(x, for this) = 00 = c0
c(x, this in) = 10 = c2
c(x, in this) = 00 = c0
c(x, this sentence) = 11 = c3

So we have 5 times c0 which means n0 = 5, 2 times c3 means n3 = 2 and n2 = 1. Finally bi is the number of bits equals to 1 in c1 which b0 = 0, b1 = 1, b2 = 1, b3 = 2. This completes all the values you need for computing the lambda collocation scoring metric. Also take into account you could add smoothing to ni to avoid 0 counts. Now with respect to

  The ni are the counts in a 2^K contingency table whose dimensions are defined by the ci.

The definition of a contingency table for the array data is simply the number of times a given value appears (i.e. its frequency). See this and the usage of the table command for more information. For the example above we have the following array data:

c3 c0 c0 c0 c0 c2 c0 c3

So the corresponding contingency table is:

c0 | c1 | c2 | c3
-----------------
 5 |  0 |  1 |  2


Understanding the multidimensional-nature of the data being fed to a RNN and its output

I think you’re confused about what a “batch” is. A batch has a very specific definition in machine learning.

In my experience:

Dimension 1 = number of bins or number of data points per each time step
Dimension 2 = window size, the number of time steps
Dimension 3 = batch size, the total number of examples which you’ll feed the network per training batch

Looks like this for [4,3,2]:

[[[0,1,4,3],[0,8,6,9],[9,6,7,4]],[[6,8,7,0],[1,7,7,9],[1,3,5,8]]]

4 values per time step, 3 total time steps per example, 2 examples in the batch.

Where [0,0,0] returns 0, [3,1,1] returns 9.

Also, one friendly piece of advice - don’t think in terms of time stamps. You’re not dealing with time in an RNN. You’re dealing with steps in a sequence. The sequence can be related to time.

Edit: With respect to the tensorflow stuff, a tf.dataset object can contain both the input data and the labels for each example. The point is to make writing code easier, managing data easier etc.

The process for training/testing is the same.


Here are a few ways, using some dummy data:

In [1]: import numpy as np

In [2]: a = np.random.randint(0, 10, (10,))

In [3]: b = np.random.randint(0, 10, (5, 10))

In [4]: a
Out[4]: array([4, 1, 0, 6, 3, 3, 6, 6, 1, 8])

In [5]: b
Out[5]:
array([[9, 0, 6, 1, 1, 1, 4, 7, 4, 7],
       [5, 8, 8, 3, 4, 8, 7, 3, 0, 4],
       [2, 2, 5, 3, 9, 6, 1, 5, 8, 3],
       [2, 0, 4, 3, 5, 3, 3, 4, 3, 3],
       [3, 3, 6, 4, 7, 5, 8, 6, 7, 3]])

Because of the dimensions you asked for, in order to compute inner products (a.k.a. scalar products and dot products), we need to transpose the matrix b so that the dimensions work out.
With a vector of length 10, numpy gives it shape (10,). So it seems 10 rows and no columns, however it is kind of ambiguous. Numpy will essentially do what it has to in order to make dimensions work. We could force it into a (10, 1) vector by using a.reshape((10, 1)), but it isn't necessary. The matrix has a defined second dimensions, so we have a shape (5, 10). In order to multiply these two shapes together, we need to make the same dimensions match in the middle. This means making (10,) * (10, 5). Performing the transpose on matrix reverses the dimensions to give us that (10, 5). Those inner 10s will then disappear and leave us with a (1, 5) vector.

That all being said, we can use any of the following to get equivalent answers:

The standard standard dot-product:

In [7]: a.dot(b.T)
Out[7]: array([174, 174, 141, 119, 190])

The convenient numpy notation:

In [6]: a @ b.T
Out[6]: array([174, 174, 141, 119, 190])

The efficient "Einstein notation", a subset of Ricci calculus (I leave the interested reader to search online for more information):

In [8]: np.einsum('i,ij->j', a, b.T)
Out[8]: array([174, 174, 141, 119, 190])

Here as in the comments from shadowstalker:

In [9]: np.array([np.dot(a, r) for r in b])
Out[9]: array([174, 174, 141, 119, 190])

If your matrices are of dimensions (100, 100) or smaller, then the @ method is probably the fastest and most elegant. However, once you start getting into matrices that make you wonder if you laptop will handle it (e.g. with shape (10000, 10000)) - then it is time to read the documentation and this blog about Einstein notation and the amazing einsum module within numpy!


How to impute missing value in Test Set using a custom Imputer created on training dataset

The way you are imputing your feature can't be replicated in the test set, because it needs knowledge of the target classes!
You need to select a different imputation strategy, that doesn't rely on your target feature.

Assuming that you are using another feature, the same way you were using your target, you need to store the value(s) you are imputing each column with in the training set and then impute the test set with the same values as the training set. This would look like this:
# we have two dataframes, train_df and test_df

impute_values = train_df.groupby('Another Feature')['Feature'].mean()

train_df['Feature'] = pd.Series(train_df['Feature'].values, index=train_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

# train your model ...

test_df['Feature'] = pd.Series(test_df['Feature'].values, index=test_df['Another Feature']).fillna(impute_values).reset_index(drop=True)

Example:
train_df = pd.DataFrame({'f1': ['a'] * 5 + ['b'] * 5, 'f2': range(10)})
test_df = pd.DataFrame({'f1': ['a'] * 3 + ['b'] * 7, 'f2': range(10, 20)})
train_df.loc[[1, 6], 'f2'] = np.nan
test_df.loc[[1, 6], 'f2'] = np.nan

impute_values = train_df.groupby('f1')['f2'].mean()

train_df['f2'] = pd.Series(train_df['f2'].values, index=train_df['f1']).fillna(impute_values).reset_index(drop=True)

test_df['f2'] = pd.Series(test_df['f2'].values, index=test_df['f1']).fillna(impute_values).reset_index(drop=True)


Could you help me to resolve this exercise of K-NN?

An algorithm implementing KNN for classification tasks goes as follows:

Compute the distance (in this case Euclidean) from the test point in question to all points in the training data.
Using the computations from 1), sort the training points in ascending order, according to their distance from the test point in question (smallest distance first).
Take the first K training points from the list in part 2), and collect the classes assigned to each in one list.
Assign to the test point in question the class which appears most often in the list from part 3).

In this case, since we are interested in the Euclidean distance, you can use the formula

$$ d(p, p') = \sqrt{(x_{1}- x_{1}')^{2} + (x_{2}- x_{2}')^{2}  + (x_{3}- x_{3}')^{2}} $$

to calculate the distance between a training point $p=(x_{1}, x_{2},x_{3})$ and a test point in question $p' = (x_{1}', x_{2}', x_{3}')$.

  With the euclidean distance, what is the prediction with K = 1 and with K = 3 for the test point (0, 0, 0)?

For the K=1 case, if my algebra serves me correctly, the training point with the smallest distance to the test point $p' = (0,0,0)$ is the point $p=(-1,0,1)$ (with distance $\sqrt{2}$). Assuming my calculation is correct, this means we should associate the class of the point p, namely the class "Green", to the test point $p'$. The K=3 case can be approached by the same method, but this time we will need to calculate all distances from train points to p', keep the train points with the 3 smallest distances, and assign the class which occurs most often among those 3 train points. I'll leave it to you.

  If the Bayes decision boundary in this problem is highly non-linear, then would we expect the best value for K to be large or small? Why?

Some intuition can be gained by thinking about the K=1 case. In this case, each training point has a neighborhood around it such that any new test points in that region will be classified in the same way as the training point. By definition, this neighborhood is given by those points which are closer to the training point in question than they are to any other training points. These neighborhoods are known as Voronoi cells, and together they form a Voronoi tesselation (see wikipedia for some pictures). The decision boundaries for KNN with K=1 are comprised of collections of edges of these Voronoi cells, and the key observation is that traversing arbitrary edges in these diagrams can allow one to approximate highly nonlinear curves (try making your own dataset and drawing it's voronoi cells to try this out). To answer the question, one can investigate how this ability to approximate nonlinear curves changes as K gets made larger, and try to relate the findings to the existence of a highly nonlinear decision boundary from a Naive Bayes classifier.


Different t-test results from R and LibreCalc

If alternative = 'greater' means that alternative hypothesis is "x has greater mean than y". In your code:

t.test(data[,1],data[,2],paired = TRUE,alternative = 'greater')

it translates to data[,1] has grater mean that data[,2]. Which is obviously not true. Just change the order of inputs and you'll get a correct p-value

t.test(data[,2],data[,1],paired = TRUE,alternative = 'greater')

I suppose that in LibreCalc arguments are flipped w.r.t. R


How to set hyperparameters in SVM classification

You could use cross validation with grid search as shown here


I read your question as:

  tl;dr. How do you know that a data scientist adds value to a company?

The job of a data scientist is to pick out the best data set, scrub it, fit a statistical model. And most importantly, a data scientist needs to know the problem statement and why is the analysis being done.

For example, consider a DS team in an e-commerce company. Their primary (let's assume) is to build a highly effective recommender system, which needs to adapt itself to a lot of variables, like the user behaviour, the session-based behaviour, user segments, etc.

So, if the recommender system is still not being able to get the company a considerable CTR or upsells/cross-sells, then maybe the team is not adding proper value to the company.

So, a data scientist needs to have a keen knowledge of the business value which an experiment is expected to add to the company. The decisions made from the experiments should ultimately add value to a company (monetary or otherwise). If not, then just like any other employee, it means that the person have failed in the job.


Can a GAN-like architecture be used for maximizing the value of a regression predictor?

Note that GANs are generally considered either as a way to perform implicit density estimation or defined by the game-theoretic minimax problem they usually imply. The situation you're describing doesn't really seem to fit in either of those, especially the latter, because your generator $G$ and $D$ are not really working against each other; rather, they are working together to maximize the function you are optimizing. In other words, the situation you're describing seems to simply be doing generation that maximizes some objective (in a somewhat complicated manner it appears), rather than realism. Since the objective is readily directly computable (as far as I can tell), we can just do standard optimization maximizing it. The reason why the alternating GAN optimization is needed is because the specific objective of realism is not directly computable nor optimizable without first training the critic.

Nevertheless, combining GANs and regression is quite sensible, so maybe I'm misunderstanding your question and notation.
Let $X$ be the data space, and let $f:X\rightarrow\mathbb{R}$ be the true value of the function you are trying to maximize (after regressing). Let $R:X\rightarrow\mathbb{R}$ denote a learned regression function approximating $f$. Denote $G:\mathcal{U}\rightarrow X$ as the generator and $D:X\rightarrow[0,1]$ as the critic (discriminator), aiming for realism.
Let me consider two possibly relevant scenarios, assuming a dataset $(x_i,f_i)$:

Simply directly optimize a GAN with an additional loss term maximizing $f$:
$$ \mathcal{L}(G) = (1-\eta)\mathbb{E}_{x\sim G}[\log D(x)] + \eta \mathbb{E}_{x\sim G}[R(x)] $$
for the generator. We can optimize $D$ and $R$ in the standard way together. In other words, we are telling $G$ to generate $x\in X$ that balance realism and high $f$, with the balance controlled by $\eta\in[0,1]$. When $\eta=0$, it is a regular GAN (not caring about $f$); when $\eta=1$, the critic is not used at all and there is no regard for realism - $G$ will learn to generate random samples that maximize $f$ without caring about how realistic they are (i.e., without matching the true distribution $p(x)$).
Separately, train the regressor $R$ and the GAN generator $G$. Then we can generate a high $f$ data point by gradient descent in the GAN latent space, i.e. doing:
$$ u_t \leftarrow u_{t-1} +\xi \nabla_u R(G(u_{t-1})) $$
so that our output $x=G(u_T)$ has high $f(x)$. Since $u$ is in the GAN latent space, we expect $x$ to be realistic as well.

In both of these simple scenarios, we are using the GAN to maintain realism, while maximizing $f$. Notice that if we don't care about realism then the GAN is not needed at all. Simply generating samples does not make a model a GAN: there are many generative models, such as Deep Boltzmann machines, VAEs, etc... that can do this. What sets GANs apart is adversarial learning, where two networks work against each other, and it's not clear that's happening in the case you described.

I think this question is somewhat relevant to yours as well.


algorithm to auto-download articles from the internet

what you are asking for is known as web scraping
, if you don't what it is, you can read about it.
can you please clarify your question more, so you can get the right answer.

what programming languages you'll be using, if you are planning on using python, there's a great framework for web scraping called Scrapy.
I found this Parser which does what you are asking for.
finally, you can check this related question.


Seems like it predicts the first class. Sklearns Random Forest implementation generates probabilities for each class by averaging the probability predicted by each estimator into an array of shape (n_samples, n_classes), and then uses np.take(np.argmax()) to select the highest probability, something akin to this:

# Pretend "a" is our averaged predictions for the forest. So the first sample is predicting 78% probability
class 0, 22% class 1. The second has the probabilities reversed and the third is 50/50 split.

a = np.array([[0.78, 0.22], [0.22, 0.78], [0.5, 0.5]])

np.argmax(a, axis=1)

The output is array([0, 1, 0], dtype=int64). These are the indices of the highest value in each sample of the array, and for the sample with an even split you can see it's picking class 0.

edit:

If you want to see it yourself, the relevant piece of code is line 540 in forest.py of sklearn:

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/ensemble/forest.py


Regression in Python with many NaN values spread across all columns

If you have 3 completely different data sources where there are no common columns consider creating 3 separate models instead of trying to force them into one.

If you have overlapping columns as well handling of NaN values can be influenced by your model choice. (i.e. most tree based model can handle missing values) If you use linear regression you need to have numerical values and you can experiment with several options:

adding new variables indicating if a certain variable is missing or not (0/1)
instead of using 0 for the missing values experiment replacing them with either median/mean/random value from the original distribution of the variable (as a result your new data will have the same distribution of values as the original one)
create model for predicting variables containing missing values and use it to replace NaNs
use some dimensionality reduction if your matrix is too big and sparse


I have a pandas dataframe and i need to clear all the special characters other than space

import pandas as pd
df=pd.read_excel("OCRFinal.xlsx")

whitespace = "\r\n\t"

df['OCR_Text']=df['OCR_Text'].apply(lambda x: x.strip(whitespace))
print(df['OCR_Text'])


Polynomial regression vs. multilayer perceptron

Polynomial regression can have multiple entries in the normal equation and it is not easy to say which polynomials you have to use in advance. Moreover, if you have lots of features you cannot handle memory errors most of the time. Nowadays people use MLPs and use batch normalization among layers for learning better. Those that you are referring to are a bit old algorithms but the former one is the logical mathematical solution for learning problems and the latter one is a beginning point for deep neural networks. I recommend taking a look at here and here.


Learning capacity: Deep Learning vs Traditional (Shallow) Learning

I believe this statement can be supported with the concept of VC dimension. This blogpost provides a simplified explanation of the term. VC dimension can be understood as an ability of a classifier to learn complex dependencies in the data. From other hand, models with a huge VC bound tend to overfit on small amounts of data.

However, the plot you provided here depicts what is happening when our training set grows indefinitely - the win situation for models with a large VC dimension.

The VC bound of neural networks is something like polynomial from number of weights and connections between them. While VC dimension of a SVM classifier is linear to the dimensionality of the space it operates in.

Thus, neural networks can benefit more from larger amounts of training data. And the more weights you have, the better will be the result. Of course, given that the number of training samples is at least 10 times more than number of weights (a rule of thumb), so you do not face overfitting. That is why data augmentation and regularization are so important in deep learning.


Test for heteroscedasticity in time series

You can use the same tests on the raw time series. You are not required to use it on the residuals. That is just what these tests are usually used for (if you are interested in the quality of your model).

However, heteroscedasticity is most easily identified by visual inspection.


Overfitting is visible when the performance on the training set is much higher than the test set, so you clearly had overfitting with your first experiment already.
Cross-validation is not a way to see overfitting by itself, it's a way to obtain a more reliable performance value by using several splits (removes chance factor) and using an overall larger test set (the full data).
One thing I notice in your experiment is that the max_depth parameter is set to 17 which is very high, especially in case your training set is not too large. This means that you're allowing the model to create very big and complex trees, so there is more chance that it would represent small patterns which happen by chance in the data. Try reducing this parameter value, hopefully this will avoid or at least decrease the overfitting. If it's not enough there might be some work to do at the level of features: too many features and/or not enough instances can cause overfitting as well.


It definitely can be associated with over-fitting. I would suggest plotting the training and validation accuracy in a single graph. So you see how both depend on the number of splits. If it is over-fitting one would expect that the training accuracy continues rising, while the validation accuracy gets to a maximum and then drops.

However, you have to make sure that your training and validation sets are OK. I assume you do something like k-fold cross validation. Where every sample is k-times in a training set and one time in a validation set. And you have enough samples to train your model etc.


For the missing data problem, one thing to be aware of, is the missingness mechanism. Depending of the dataset, the NA's (Missing Values) you have could be a result of a condition of the phenomenon and you shouldn't impute then using mean but maybe.

Besides, for the dependent variable, if you want to train a model with the independent ones to predict this, let's say Y, you wouldn't train a model using this observation with NA on the dependent (target?). Then, you would drop this lines or maybe using another technique which takes into account the dependence of the other variables.

I think a good start is to give look at this:
Missing-data imputation

It shows the limitations of using some approaches like yours and defines the mechanisms of missing data.


How to find 4th point equidistant from 3 other points

As Ankit said in comments, this is not possible in your concrete case, as you cannot just expand 1d point coordinates to 2d without inventing some information.

In 1d, where each point it represented by one single number, it would only be possible to find the 3rd point, equidistant to two original points. If you imagine the two points on a line, the third point would then just be the centre of those two points, which also happens to be the mean of those two points.

For three points, you would require a 2d space, i.e. each point must be a vector of length two: so basically an (x, y) coordinate pair.

If you did have (x,y) coordinates for three unique points, they would form a triangle, and the equidistant position (i.e. your fourth point) is called the circumcenter, and it found by finding the centre of each of the sides of the triangle, then drawing a line through each, which is perpendicular to its corresponding side of the triangle. These three lines will cross at the circumcenter - and it is not necessarily inside the triangle!

Example:


Creating Strings corresponding to Location Co-ordinates

Your question has nothing to do with NLP or text-mining (as you claim in the attached tags), or data science in general. It's a pure programming question best suitable for StackOverflow.

Moreover, you don't really need any libraries to do, what you want to do. A simple function will do. NOTE: I am using map and reduce functions on purpose to include at least a little bit of data science-related stuff (WINK):

coord = (1.23, 4.56)

def get_coord(coord):
   coord_str = map(lambda x: str(x).replace('.', ''), coord)
   for level in range(1, len(coord_str[0])+1):
      yield reduce(lambda x, y: (x[:level]+'_'+y[:level]), coord_str)

print list(get_coord(coord))

Running this code will result in printing:

['1_4', '12_45', '123_456']

You're not discussing any corner cases in your question, so I assume there are none.


Do RL agents learn the optimal "degree" of an action to take?

If you are using a value-based method, like Q-learning in Deep Q Networks (DQN), then the "degree" concept has little meaning to the agent, and you are effectively training an agent to learn the best discrete action out of 200 actions.

Yes this could take a lot longer than learning a simple 2-action scenario, as the agent will not easily learn that the value of "Right 49%" is correlated to the value of "Right 52%" in any particular state.

This is a similar problem you might face in supervised learning if you wanted to move from classifying a dog vs cat image classifier to add bounding boxes for where the creature is. You don't add the bounding box as a large number of classifications - instead you add it as a regression, mapping form the image directly to some vector of real numbers.

There might be ways to address this using value-based methods, but the usual approach here would be to move to a continuous action space (maybe with -1 for 100% Right and +1 for 100% Left), and use a Policy Gradient method. There are a few different choices of Policy Gradient, but they all share the basic principle of modelling a policy function - which might be a probability distribution over action space which you need to sample from, or a specific action - converting from state to action.

The most basic Policy Gradient approach, which you could use as an introduction for your problem if it is simple enough, is REINFORCE. Algorithms like A3C, DDPG, PPO are based over this idea and can cope with more challenging environments.


I agree with the idea of using a similarity or distance measure (approximate string matching). I would try a bunch of them and test them on a sample: Levenshtein, Jaro, overlap coefficient or cosine (optionally with TF-IDF) over bi/tri-grams of characters.

I would also try to capture the most common abbreviations and have a lookup table for these common cases because:

Computing similarity/distance measures takes time, so it's inefficient to compute the same result many times for the same string (and it's likely that some of these abbreviations are used many times).
That gives you an opportunity to check that the mapping is correct (or to fix if it's not) at least for the most common cases, thus minimizing the overall amount of noise in the data.


Why is my training accuracy decreasing higher degrees of polynomial features?

I disagree with the assertion of, "Theoretically the accuracy on training set should increase with degree". The goal of polynomial regression is not to randomly try new polynomials. The goal is to use a polynomial that better fits your data because the correlation is not linear.

Let's think about the end result of linear regression - it usually something like y = mx + b

If you show that to a data scientist, they're going to tell you it's linear regression.  You show that to a math student and they will tell you its the formula for a straight line. Either way, it's just a formula for a graph. But, note that this is for a straight line and not all data is linear. So, knowing that you're just coming up with a formula, you should think about polynomial regression in the same way - what graph am I trying to draw?

If you use a scatter plot and you are seeing a correlation but that relationship is exponential, then you should use the corresponding polynomial; same goes for all of the other variations. There is no logical explanation to use a polynomial that will not draw a graph that will closely align with your data correlation.


I am assuming this is competition data?

Basically, start with the previous_data and information_history data which seemingly provide extra information for observations in the training and testing datasets. You are probably given lookup ids that allow you to relate any particular row of these extra datasets to that of the training/testing datasets.

Most people will aggregate these extra variables so that one row = one unique lookup id (i.e., aggregate these datasets by id) by using various sample statistics; means, maxes, mins, variances, kurtosis, percentiles (like the median), etc. I've even seen people attach the target variable from the training dataset on to the "extra" datasets so that they can run regressions/classifiers on these datasets to get more features that give better aggregations. One extra suggestion: information_history and previous_data sounds like there is a time element. Perhaps be mindful of using more recent data, perhaps having features like "x mean in past k time periods", or "y median in n most recent loans", etc. I don't know what the datasets are but that is just a suggestion.

After that is done, then simply join these aggregated summary statistics to the training/testing data sets by the lookup = id column. In R, this is quite easy to do with the tidyverse and left_join/right_join, for example.

I hope this answers your question.


Which Big Data technology stack is most suitable for processing tweets, extracting/expanding URLs and pushing (only) new links into 3rd party system?

I'd suggest Apache Kafka as message store and any stream processing solution of your choice like Apache Camel or Twitter Storm


Set value of a column based on values of other columns (Pandas)

I would use apply. It allows you to perform a function row-wise or column-wise. Check this post on StackOverflow, it provides good examples of what this looks like.

The gist is:

create a function that creates the value you want based on a row
use apply to create a dataframe containing the results
if you're happy with the results, merge that dataframe into your existing one


Training LSTM Recurrent Network in TensorFlow

I removed the gradient noise (which did not seem to help, at least as you did it) and replaced your momentum optimizer with Adam using the default hyperparameters and it just worked. After 10,000 epochs I got a loss of ~8 with bcdefghijklabcdefghijklabcdefghijklabcde (actual) vs ccdeeffgghhiccdeeffgghhiccdeeffgghhiccde (predicted).

The moral is that optimizing neural networks is still an art.

p.s. It seems weird to use a regression loss with a classification problem.


Convert Neural network to Keras Classifier

There are a couple of Github issues on this - see here and here. In short, while scikit-learn native models include an _estimator_type attribute:
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
clf._estimator_type
# 'classifier'

this is not the case with a KerasClassifier; using your own NN gives
NN = KerasClassifier(build_fn=NeuralNetwork , epochs=100, batch_size=100, verbose=0)
NN._estimator_type

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-4-a2fd193c154f> in <module>()
----> 1 NN._estimator_type

AttributeError: 'KerasClassifier' object has no attribute '_estimator_type'

As reported in a comment here, setting the attribute manually seems to work:
NN._estimator_type = "classifier"

so add this line immediately after your NN definition.


Using pretrained LSTM and Bert Models in CPU Only Environment - How to speed up Predictions?

There are a couple of options:

Optimize tensorflow for your specific CPU. Sometimes the official versions of tensorflow are not compiled with support for some instruction sets (e.g. SSE4.1, SSE4.2, AVX, AVX2, FMA). Usually, there is a tensorflow runtime warning message stating so. This prevents some computations to take place in parallel. You can either download a version that is optimized for your CPU (e.g. from intel) or you can compile tensorflow yourself.

Prune the model. BERT is a multi-head attention model, and it is possible to remove some of its attention heads while retaining most of the quality. Here's a blog post that shows how to do it for tensorflow.


How to transform predicted results when doing cross-validation in sklearn?

If your target values are just 0 and 1, you should probably be treating it as classification, and use e.g. XGBClassifier instead of XGBRegressor.
You brought up (originally in the comments on your SO post, now edited into your questions) a scenario when your true values might be limited to 0.5, 1.5, 2.5.  That's unusual enough that I suspect the best answer depends on context, but:

Consider ordinal regression, which treats the target values as ordered but non-numeric.

If the values really are to be treated numerically, then you're left with (I think) doing it manually, which is closest to your original question.  (I'd like to stress one more time that this is odd, and optimizing your loss function may do something unintuitive.)  Since the predictions out of sklearn will be numpy arrays, maybe try numpy.digitize.
Notes: This is useful for post-processing, but won't be easy to incorporate into the model fitting nor a pipeline (since sklearn's predict doesn't behave the same way as transform).  If you really need to process the continuous predictions before scoring inside a model fit, I think you're left with hacking the model's code a bit; and with xgboost, that'd be quite a task.


Customer Targeting for CRM Marketing Campaign

Can you share all available columns in the data set? its hard to tell what data is there to use.
Given the data you show, I would have started with some explaratory analysis:
Group data per shops, check user activity ( how much purchases they do per month/week, what is AOV, LTV, etc).
Idealy, its good to use email marketing, because its cheaper that paid ads and you have list of users that can be mailed - given the low costs of email marketing campaigns, you can basicaly target all users that stopped shopping in given places and allowed you to send them mails.

Additional approach: you can use paid ads for remarketing and geo-based ads to recover lost clients - given the fact that you want to cut the costs, its might be beneficial to retain only loyal customers ( which have more that 2 or 3 transactions, this number should be based on your average sales per client).

So, answering the questions:

What are your recommendations for the campaign? Should we do it? How should we target and why?
Should we do it?

Select all users that were lost after opening new shop and calculate lost income per month (or week, based on your data) - you want to know how much money you can expect from recovered users, this can be your marketing spend baseline ( Recoverd Income - Costs - Margin ~ Marketing Budget). Marketing Budget + Recoverd Income will indicate if its logical to run the campaign.

What are your recommendations for the campaign?

I would start with email - cheap and effective, we can use targeting in emails based on what items users buy, their gender and age.  If you have segment of loyal users with high income - you can develop dedicated email campaign based on their needs, give bigger discounts ( its good to know why they left in the first place).

Is there any additional customer information that would help you? What is it? How would it help?

Any data that will help to better segment users will be usefull - age, gender, social status, all this info can influence email campaigs.
Also it would be good to have data about costs and effectivness of Paid Remarketing Campaigns - based of this data we can try to use remarketing for our loyal customers.

Have both stores been impacted similarly? How has customer shopping behavior been affected?

Split data per shops, calculate statistics, do some good-looking and easy to undestand graphs, cant say more without actual data.

Hope this help!

EDIT

Thanks for sharing data, here is my opinion about this task:

You can split data before-after 25 of may, and target users that have losses in sales comparing to period before 25 of may.

Based on Customer_Data dataset, it looks like you have user that can be mailed and emailed, so you can launch 2 campaigns for different users based on Mailable_flag / Emailable_flag flags.

Talking about losses: Based on given data is looks like your shops lost around 3k in sales and ~60 transcations for both shops after 25 of may and the losses are equal amont users that can and connot be emailed.

This leads us to a point of "additional" data - ideally you want to know the past performance of mailed and emailed campaigns to adjust the targeting.

Also the thing I havent done - you should calculate LTV ( basicaly how much money the users brings you per month) - if some users have very low LTV, you dont want to spend money advertasing them.

I`ve attached ipython notebook with brief workflow for you to check . Please, be aware that its just my ideas and they cant be totally wrong :)


what is a good performance measure for comparing different neural network architectures in unsupervised clustering task?

A solution is using some metrics like silhouette measure to measure the performance of ANN in clustering.


Finding optimal threshold in multi-class classification task

One solution is to explore a One-Vs-Rest classifier which creates separate binary classifiers for each class.


How can I measure the reliability of the specificity of a model with very small train, test, and validation datasets?

Real world data is "test dataset", right?
Data has to be divided in such a way that train-validation see part of data more than once while test data will be seen only once. In that sense, if the model is robust enough, it will perform well even on the test dataset. The assumption is that test data is as close as possible to real-world data.


Activate your conda environment, then use the pip that will also be in your environment (just like the Python interpreter is that of you activated environment).

For example:

source activate your_tf_env          # now we are in the conda env
which -a pip                         # should list all pip executables

The first pip one should be in your_tf_env somewhere.

Now you can simply install your package, e.g. using pip install fastai, which will use the pip of your environment and the package will also be installed in your environment. You do not need to manually copy the files anywhere.

With your conda env still activated, you can then confirm the package is installed by running:

conda list                     # list all installed packages
conda list | grep fastai       # grep filters results, e.g. for  "fastai"


What is the relationship between "landmark Detection" and "landmark localization"

Landmark detection and Landmark localization are the terms that have not been defined well. It seems like they are just "created" because someone (scientists) needed them. They are similar to object localization and object detection. After seeing the question I have been searching for finding the contexts that they are used so they can be compared. However, I can say either people confuse them or they are used for the same meaning interchange. Before, proving that I will note one another thing. Normally, localization means to simply localize an object that is, finding the position of an object, or specifically drawing a bounding box around an object. Detection uses localization as a function, for example, in object detection all the objects in the image are localized and labeled accordingly. Thus, it might be expected that if a single set of landmarks (human face) are identified then it would be a landmark localization, and if multiple sets of landmarks (human face, human hands) are identified then it would be landmark detection as in object localization and detection.
So now, check this link where Nvidia names identification of different sets of "objects" as landmark localization.
Secondly, here Andrew NG explains exactly the same thing in the introduction of the paper that you provided, but he names it landmark detection. If you search for it, you will see there are a lot of papers that use landmark detection and landmark localization for explaining the same thing.
I don't claim they are the same thing, de jure they might be different concepts but de facto they are used to define the same thing, maybe because they are extremely confused. Thus, it does not "give chance" to understand their meanings.
Update: I asked the question to my Prof. from the university. I will briefly paraphrase the respond: When it comes to localization, it seems to give more importance to location. For example, in a robot application, the robot does object localization or landmark localization to determine the exact coordinates of the object to maybe hold it or touch it. In detection, we actually mean there is something here. For example, I took a photo. There is a tower in this photo. As a tourist I want to know which tower this building is. Finding the tower in the picture can be described more as a detection problem. Another example is a tumor. For example, there is a tumor in a medical image. I might want to localize this tumor in the real body using the image. Or I can detect the tumor in a biopsy image and say that this person has cancer. Although they are usually used for the same thing, you can see one is used more than another in different application areas.


The common procedure is this:

Fit the model on train split and adjust hyperparameters using the training split and validation split, as you already said.

Then, to get a "final measure" of performance, train model on both training and validation splits using the hyperparams found during validation phase, and evaluate on test split.


How to optimize the lambdas of a hybrid loss in a deep learning model

When there is an optimization problem involving more than one objective function to be optimized simultaneously, it is called multi-objective optimization.

Gradient descent methods can still be used to minimize the overall objective function. A paper entitled "Gradient-Based Multiobjective Optimization with
Uncertainties" goes into greater detail.

In PyTorch, autograd.backward can handle multiple objectives. You need to provide a list of tensors for backpropagation.


What are some good papers that discuss Tufte's 'data density index' and 'data-to-ink ratio' in data visualisation?

There are critcism's of Tuft's principles.  Some examples:

Sometimes We Must Raise Our Voices, Stephen Few, Perceptual Edge

  Even after many years of working in the field of data visualization, which has involved a great deal of experience and study that has expanded my expertise into many areas that Tufte hasn’t specifically addressed, I have only on rare occasions discovered reasons to disagree with any of his principles. The topic that I’m addressing in this article, however, deals with one of those rare disagreements.

Minimalism in information visualization: attitudes towards maximizing the data-ink ratio

From the abstract:

  People did not like Tufte's minimalist design of bar-graphs; they seem to prefer "chartjunk" instead.


TypeError: Expected binary or unicode string, got array

Try this:

imgs = tf.convert_to_tensor(images[0])

If all of your images are of the same shape. Otherwise you have to resize them to a same dimension then call the code.


Analysis for basic weight training analysis?

I'd start with PCA to see the main correlations and maybe clustering. With the PCA you should be able to find nice sub-spaces, which you can plot and see, which parameters correlate the most/least. Clustering might help you identify "growth spurts". You might also try some linear regression to predict weight repetitions given the other parameters. Here I would be careful though, as the best predictor of performance will probably be the implicit 10th variable: time. You might want to tweak the regression problem to predict changes in performance given sleep, stress etc. In general, depending on what you are looking for, it might make sense to consider "change in repetitions" rather than the repetitions themselves.

I hope this helps.


Is it possible to use Word2vec for text paraphrasing?

Someone has defined a problem for paraphrasing on Tensor2Tensor:  https://github.com/tensorflow/tensor2tensor/releases

You can also define your own problem but you might have to supply the corpus:  https://tensorflow.github.io/tensor2tensor/new_problem.html

If you are just looking for a vectorized representation of your sentences, Google BERT might be worth looking at.  Bert-as-service is quite convenient:  https://github.com/hanxiao/bert-as-service

Perhaps you could reframe your problem slightly as a question-answer type problem for which I think both T2T and Bert are well equipped to handle. The questions could be of type, "The three little pigs each had their own home?"  For which an answer paraphrasing the question could be, "Each of the three little pigs had their own house."  (But I'm taking a guess here).  Maybe there is a way to tweak the problem so you don't have to include a question mark?  Then it would be a paraphrasing task...

I'm sorry I can't provide you with a more professional answer but I think T2T is worth looking at for sure.


Ship detection on high resolution unlabeled image dataset

So, the question concerns how to go about detecting ships in images, count the number of ships in the image and get the model to predict whether the ship is parked or not.
From the problems you have posed, I think it is best that we implement supervised machine learning. This means that we need labelled data. For this, I would recommend taking a sub-sample of the data, making a data log which maps the images to the labels (e.g. the file name and then the number of ships and whether it is parked or not.).
In terms of preparing the images for ship detection, it would be a good idea to mask these images by drawing a box around the ship(s) in your subset of images.
When it comes to models, the obvious choice is going to be something which follows a convolutional network, for problems 2) and 3), you might need to use a CNN model as the encoder, which will encode the visual information into a hidden representation. From this hidden representation, you would then decode with a normal feedforward neural network.
For problem 2), you would get the overall model to output a scalar value, which will denote how many ships are in the image.
For problem 3), you would have a final 2-node softmax output layer, which will output a probability distribution over whether the ship identified is parked or not.
I think someone would need to fill in for me on problem 1, as I do not have much experience in the way of semantic segmentation (which is essentially what ship detection is in your case).


Imbalanced dataset - Positive majority class

There are two things that might help you:

1)
If you use a classifier $f$ that returns a value $f(x)\in[0,1]$ between 0 and 1 instead of a direct class assignment, then you can use a threshold $\theta\in[0,1]$:

$f(x)=\begin{cases}
    1,& \text{if } f(x)\geq\theta\\
    0, & \text{else }
\end{cases}$

The higher the value of $\theta$, the more you want to be sure to classify somebody as positive, which means that there will be fewer false positives, but at the same time you will have a fewer sensitivity (recall) because there will be more false negatives.

2) if your classifier always predicts the majority class, you could try oversample the minority class (using SMOTE or other oversampling techniques) to make the training data set balanced.


You want to decide this based on how well your model performs and generalizes. If your model is underfitting, you want to increase your model's complexity, increasing variance and decreasing bias. If your model is overfitting, you want to regularize the model and/or feed it more training data, decreasing variance and increasing bias.


What could make a set of the train data more predictive than the whole train data

In absence of noise and if the difference of accuracy you observe is significant, the only reason I see is that, by luck, the distribution of the training data subset happens to be closer to the distribution of your validation data than the distribution of the whole training data.
This should not happen with truly representative validation data (i.e. with training and validation data built by random split of a given dataset).


How to display the entire output and not shortened versions

Alternatively to Marce's answer, you can display all the rows without finding N by using:

with pd.option_context("display.max_rows", -1):
    display(train.isnull().sum())


Do zero weights receive zero gradient in ReLU neural networks?

The derivative (of a loss function) with respect to $w_i$ can be any value, independent of the value of $w_i$.  (E.g. $f(x)=x$ has derivative 1 everywhere, even at $x=0$.)


Why does averaging a sentence's worth of word vectors work?

It works for the same reason why the good old bag-of-words + TF-IDF works. Despite loosing some word ordering information, a text can be still classified by the typical keywords. Since texts on different topics differ a lot with respect to the vocabulary used, simply putting together the words' embeddings might work surprisingly well.

Here is a paper that shows that a simple sentence embedding methods beats sophisticated supervised methods including RNNs and LSTMs. Their method is just a weighted average of the word vectors, modified a bit using PCA/SVD. Section 4.3 tells that the word ordering plays a role, but not too much.


How to optimally train deep learning model using output as new input

The first point I would make (and you are likely aware of this given you specified "lstm" as part of your tags), is that you should use a long-short term memory network for this purpose.

Given that you are working with time series data, the risk with a regular neural network is that it treats all the observations as independent when this is not the case with a time series.

  Instead of training with a completely correct input sequence and its
  correct output, i was thinking the network should train on its own
  predicted, not completely correct, inputs as well.

This is somewhat of a slippery slope, because one must assume that you are using actual outputs. The model might be falsely trained to believe that the predicted values (which ordinarily would not yield the correct output) do in this case, and this skews your results. The model is only as good as the data that is being fed into it - if the data is wrong then the results will be as well.

In general, your error reading looks quite good, although I am not familiar with your data or model.

If you wanted to generate dummy data, one possibility could be to use a bootstrap simulation of sorts. For instance, suppose you have a time series with 10% mean and 20% volatility. You then choose to generate a random time series based on these two readings. The neural network could then be further trained on such data, and may potentially improve accuracy.

However, I would be cautious of mixing inaccurate inputs with correct outputs - as this has the potential to skew your results.


Large Numpy.Array for Multi-label Image Classification (CelebA Dataset)

I ended up writing a python generator, which actually works very well, for manually feeding desired number of images chunk by chunk into my CNN model like this:

def image_batch_generator(df,images_path, batch_size):

    '''
    A generator that takes a dataframe (for image names) and
    with a given image path goes to conver images to numpy array over
    batch (chunk by chunk).

    "df": is the "Attributes Annotations" text file from CelebA dataset.
    It has a column for image names, and another 40 attributes columns
    (binary) for each image.

    "images_path": it is the path to CelebA dataset image files.

    "batch_size": the batch size by which image will be read chunk by chunk.
    '''

    L = df.shape[0]
    files = df['Images'].tolist()

    #this line is just to make the generator infinite, keras needs that
    while True:

        batch_start = 0
        batch_end = batch_size

        while batch_start < L:
            limit = min(batch_end, L)

            X= np.array([np.array(skimage.transform.resize(io.imread(os.path.join(images_path,fname))/255., (64, 64))) for fname in files[batch_start:limit]])
            y=df.loc[df["Images"].isin(files[batch_start:limit]), :].drop(['Images'],axis=1).values

            yield (X,y) #a tuple with two numpy arrays with batch_size samples

            batch_start += batch_size
            batch_end += batch_size

Hope it will help someone out there.


No, it sounds like an odd metric. I suspect the parametric distributions often calculable for the median and mean would be rather nasty for the average of the two which would make this unpopular among statisticians.
I suspect you mean "outliers" by extreme values. If you consider the definition of an outlier:

In statistics, an outlier is a data point that differs significantly from other observations.

https://en.wikipedia.org/wiki/Outlier
(I checked a few definitions and most of them are like this).
I think on a discrete scale of 1 to 5, the idea of an outlier sounds odd. As you can see "differs significantly" is open to interpretation and I don't think 1 or 5 differ greatly from 2-4 unless the rest of your user ratings are at 3 only.
In any case, it is probably best to just use the mean. If you lack the variation in the data to seperate out the objects you want using the mean +- std dev or confidence intervals then that's just the data and there's not much you can do about it.


How to use flow_from_directory in keras for csv

Something Like This Should Do The Job
When you are doing something new, Mistakes are likely..
Use At Your own Risk Or Try It Out on A Sample And Try it on a Seperate Directory Completely

import pandas as pd
import os
import numpy as np
import shutil

# source is the current directory
# Open dataset file
dataset = pd.read_csv('dataset.csv')
file_names = list(dataset['filenames'].values)
img_labels = list(dataset['labels'].values)

folders_to_be_created = np.unique(list(dataset['labels'])).values

source = os.getcwd()

for new_path in folders_to_be_created:
    if not os.path.exists(".//" + new_path):
        os.makedirs(new_path)

## Be sure that there is nothing else in your directory except the data, csv and the code file, IT's Better to only have your data in that directory and reference the CSV file from a different Directory...

folders = folders_to_be_created.copy()

for f in range(len(file_names)):

  current_img = file_names[f]
  current_label = img_labels[f]

   ## **Check this Line Accordingly**

  shutil.move("path//to//current//file", "path//to//new//destination//folder//current_label//")

Link for Shutil

Another Link


Multi-label classification with nested features

I think it's always a good idea to start simple, so I'd simply suggest to try with all the features, including the different levels of "nesting" so around 2k apparently. Given that the dataset is large, I don't see any obstacle to trying this way. For the same reason I would start with a very simple model like Decision Trees or SVM, which have the additional advantage that they're fast to train. This could be a first step which provides you with a decent baseline, at least.
If the number of features turns is an issue for a more advanced option, I think this is a good case for using feature extraction (for example PCA): this would reduce the number of features and also merge features which represent the same information.


How to filter in pandas based on string value?

You can just groupby id and do mean for the remaining columns.
df.groupby('id').mean()


I am afraid both of your description is incorrect but the first one is the closest.

Each cell/timestep will actually produce a hidden state representation. However, this is surpressed by default on keras. You can re. As LSTM without Bidirectional wrapper produce a one-way sweep through your sequence, your output from LSTM(5) is the hidden state of the final LSTM cell(cell output not cell state).

Now to enable outputting the sequence of hidden output similar to what you might have learned about LSTM, simply supply return_sequence=True.

To compare output shape:

Without return_sequence=True, Input shape from [batch_size,sequence_length,previous_hidden_dim] to [batch_size,hidden_dim]

With return_sequence=True,
Input shape from [batch_size,sequence_length,previous_hidden_dim] to [batch_size,sequence_length,hidden_dim]

What you are referring as "folding" of LSTM is actually iautomatically based on sequence length of previous input.

In case of any doubt on my explanation, please read the material I give you below. You can verify my description by calling model.summary().

So basically what led to your confusion is due to keras LSTM default behaviour of only outputting the final cell output.

LSTM link


What is the proper way to bin variables for calculating WoE during credit scoring?

Or is there any algorithm/function that bins continuous variables in most practical way.

Sure there is, but that's the wrong question: the standard method for discretizing a continuous variable consists in splitting the values into equal intervals, that's it. Of course it doesn't guarantee any of the 5 conditions, since these conditions are about additional constraints almost exclusively related to expert knowledge and the specifics of the data.
Note that these conditions can certainly be automated, there's no need for manual verification. There might be some domain-specific packages which do this for you, but there's no reason a standard ML/statistics library would provide methods for every specific problem like this one.


Multiclassification Error: NotFittedError: This MultiLabelBinarizer instance is not fitted yet

This code will work. Just let sklearn.linear_model.LogisticRegression handle the multiclassification for you.

from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier
from sklearn.linear_model import LogisticRegression

X = ["How to join amazon company ","How to join google ",'Stay home']
y = ["Career Advice", "Fresher",'Other' ]

# Perform multi-label classification on class labels.

clf = LogisticRegression()

p = Pipeline([
('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', OneVsRestClassifier(clf))
])

# Use multilabel classes to fit the pipeline.
p.fit(X, y);
p.predict(X)
```


In which cases shouldn't we drop the first level of categorical variables?

First, if your data has missing values, get_dummies by default will produce all zeros, so that perfect multicollinearity doesn't actually hold.  Also, from a data manipulation standpoint (without regard for modeling), it makes some sense to keep the symmetry of having a dummy for every value of the categorical variable.
In a decision tree (and various ensembles thereof), keeping all the dummies is beneficial: if you remove the first dummy, then the model can only select on that level by selecting (through several steps in the tree, rather unlikely!) "not this other dummy."
Then again, it's probably better not to one-hot encode at all for decision trees, but for now some packages don't deal innately with categorical variables.
K-nearest neighbors seems like it would also benefit from keeping all levels.  The taxicab distance, limited to the dummies of one feature, between two points with different values is 1 if one of their values was the removed dummy, otherwise 2.
But again, it seems like KNN would be better off without one-hot encoding, but instead some more informed measure of distances between the category's values if you can come up with them.
See also https://stats.stackexchange.com/questions/231285/dropping-one-of-the-columns-when-using-one-hot-encoding
(In particular, when using regularization in a linear model, it may be worth keeping all dummies.)


Mahalanobis distance between two clusters

---------Please check the edits also for this answer---------------------

According to me, its very application specific, and depends on what you want to do. I will prefer second approach in a generic application because if 2 clusters between whom we are calculating distance are having high standard deviation, should have small distance.
Another approach I can think of is a combination of the 2. Calculate the Mahalanobis distance between 2 centroids and decrease it by the sum of standard deviation of both the clusters. I thought about this idea because, when we calculate the distance between 2 circles, we calculate the distance between nearest pair of points from different circles. Now think of circumference of the circle centered by centroid of circle. and the rest is obvious :)

EDIT:

As pointed out by @MarcusD in the comment, I will try to explain a bit:

I remarked "very application specific" because in some cases where our answer should remain same irrespective of the standard deviation of the data, then first approach will work better.
For reference check. Kevin murphy- ML A probabilistic approach Pg. 104, 2 Class LDA. The second approach is exactly this one. If you are not having this book, google Linear discriminant analysis.

The approach I gave is not theoretically different from LDA, but it is somewhat easier to implement in cases where the number of clusters is less.


How to favour a particular class during classification using XGBoost?

XGBoost has the scale_pos_weight parameter to help with this, depending on how you want to evaluate it (see tuning notes). It should be the ratio of negative count to positive count (or inverse based on how you indexed your classes).

An example in Python is here.


Training a classifier when some of the features are unknown

Welcome to Data Science SE!

Well, we say that most of our jobs is to wrangle with data, and that is because data is usually trying to deceive us... jokes aside:

You have a missing data problem

that means your have to clean your data and fill those missing values. To perform this cleaning process your must take the most classic statistician inside of you and ask:

Why is this data missing?
How much data is missing?

There are many reasons for a specific information to be unavailable. This will demand you to make assumptions and decide how to deal with this.

Jeff Sauro posted at MeasuringU: 7 Ways to Handle Missing Data, some which I list here:

Delete corrupted samples:

If you have a large dataset and there is not much data missing, you can simply remove those corrupted data points and go on with life

Recover the values:

Some problems will allow you to go back and get missing information.

We usually ain't that lucky, then you can

Educated Guessing:

Sometimes, you can infer what would be the feature value by simply looking their pears. That is a bit arbitrary but it might work.

Average:

This is the most common approach, simply use the average of that value whenever it is missing. This might artificially reduce your variance but so does using 0 or -9999... for every missing value.

Regression Substitution:

You can use a multiple regression to infer the missing value from the available values for each candidate.

Some references on missing data are:

Allison, Paul D. 2001. Missing Data. Sage University Papers
Series on Quantitative Applications in the Social Sciences.
Thousand Oaks: Sage.
Enders, Craig. 2010. Applied Missing Data Analysis.
Guilford Press: New York.
Little, Roderick J., Donald Rubin. 2002. Statistical Analysis
with Missing Data. John Wiley & Sons, Inc: Hoboken.
Schafer, Joseph L., John W. Graham. 2002. “Missing Data:
Our View of the State of the Art.” Psychological Methods.

About your experiment:

Adding -99... is creating outliers and that bit of information is heavy (numerically speaking, it is huge) and will affect parameter tuning. For example, suppose you have this data:

| Feature1 | Feature2 |
|----------|----------|
| 0        | 8        |
| -1       | 7        |
| 1        | -        |
| -        | 8        |

And you try filling the missing values with -99, now try to fit a linear regression trough the data. Can you see that you don't be able to fit it properly?

The line won't fit, and this will yield bad performance.

Adding 0 values on the other hand will give a slightly better line:

It is still not good, but slightly better since the scale of the parameters will be more realistic.

Now, using average, is this case will give you even better curve, but using regression will give you a perfect fitting line:

Note: I need to remake those images, but these should do until I have the time for it.


In the context of machine learning, what is the relationship between "Normalization" and "norm"

Let's take L2 norm as an example.

This figure from wiki illustrates "in a right-angled triangle the square of the hypotenuse is equal [to the sum of] the squares of the two other sides"

given a = 3, b = 4, then c = 5.

consider c as a vector,
$
c =
\begin{bmatrix}
3 \\
4 \\
\end{bmatrix}
$

the L2 norm of c is the length (magnitude) of c = 5.

to normalize the vector c, divide each component by its norm 5, then the normalized vector =
$
\begin{bmatrix}
3/5 \\
4/5 \\
\end{bmatrix}
$

this processing is called normalization.


Different learning rates for each dimension

The answer to your question is yes, and it is already implemented due to popular approaches which are introduced in academic papers. For instance, Adam optimisaiton approach is an optimiser which assumes it is not valid to multiply each dimension, feature, with a specified value. It uses the idea of momentum and geometric average and a bunch of other methods to employ the idea that you should not go down the hill with the same speed for different dimensions. What it says is that you may be in a location which each dimension has different slope, and due to this fact you should not go downhill with the same learning rate. In that approach, although you specify the same learning rate for the optimiser, due to using momentum, it changes in practice for different dimensions. At least as far as I know, the idea of different learning rates for each dimension was introduced by Pr. Hinton with his approache, namely RMSProp.


Why does it speed up gradient descent if the function is smooth?

My guess it is due to the derivative, as ReLU has a discontinuous derivative at 0. Hence if you use the definition:

$$
f' \approx \frac{f(x+\epsilon) -f(x-\epsilon)}{2 \epsilon}
$$

and $x$ is very close to 0, you will get many of those 'jumps'.


K-Fold and Random Subsampling (RSS) Dataset generation?

K-fold cross-validation is a technique you use when you have too little training data. You use it to expand your training dataset, leaving only a tiny bit for testing (for example, 5%). Then you use k-fold cross-validation to select a different 5% for testing each time, and a different 95% of the full dataset for training each time.  That way, you can get a good idea of the performance of your model, even if you only kept 5% of the data apart for training.

This is just an example, it could be that 5% is fine, or even too much, to hold out for testing. It depends mainly on the size of your dataset, and in case of classification, for example, it matter if one class has very few examples, so that perhaps in your 5% for testing there may not be a single example of that class.

What you're trying to do is called dimensionality reduction. The go-to technique for that is PCA. Make sure you normalize your features first (subtract the mean and divide by the variance), and then drop features until you still have, for example, 95% of the variance of your data explained. If in the end you find that you cannot get good performance from your model, it could be that you have lost too much information. In that case, you may have to increase this to 99%, adding features back in.

RSS is ok to reduce the number of training examples you have in your dataset, but it is only necessary if the order of examples is not random already. If it's random already, then you can just train on the first 10%, or whatever you need, and save yourself the trouble.


score or cost function for AgglomerativeClustering

Don't run agglomerative clustering with multiple n_clusters, that is just unnecessary.

Agglomerative clustering is a two-step process (but the sklearn API is suboptimal here, consider using scipy itself instead!).

Construct a dendrogram
Decide where to cut the dendrogram

The first step is expensive, so you should only do this once. It does not yet produce clusters, but the Dendrogram which can be helpful to decide on the number of clusters. In the -cheap- second step, clusters are then extracted. You can use the height value of the dendrogram in a similar way as inertia with k-means. In fact, the height with Ward linkage is very closely related to inertia. But the same drawbacks apply: the value is (for most linkages) monotone. So it's not too helpful for choosing upon the number of clusters naively.


IDE for Machine Learning workflow - AWS SageMaker

Some ide you can use in your local machine :

Spyder
Jupyter Notebook
Jupyter Lab

Pycharm VSCode also has intregated support for notebook but i would suggest jupyter lab. It's really good
If you want to run on VMs then google colab and kaggle kernel is really good for free.


I think you might have to keep overall Data pipelines and Machine learning pipelines in mind. For which you need a robust framework to move data between table like and graph like storage apart from powerful distributed processing. From my understanding Spark GraphX is promising to build these pipelines. Joseph Gonzalez's (one of the creator of GraphLab from CMU) talk on GraphX on youtube is worth watching.


Difference between advantages of Experience Replay in DQN2013 paper

Isn't the third advantage just another case of breaking correlation?

It could be viewed that way, but it is a different kind of correlation.

The second advantage is about breaking correlation due to samples being from adjacent time steps on the same trajectory. This is a more important problem when state vectors evolve slowly/incrementally per time step.

The third advantage is about breaking correlation due to samples being taken from the same policy. This is a more important problem when some actions have very similar reward and state progression independently of state, or may result in no change to state (e.g. an agent tries to move into a wall - agents getting stuck in corners due to runaway feedback of the direction being "best" is a thing you can sometimes observe whilst DQN is learning).

Both can also be a factor when an early event puts the remainder of a trajectory into a single part of the overall space, which can occur in environments where state history is important - in those cases, almost all of the time steps in a single episode could be correlated. Think of a resource management game, where expending or keeping a key resource early on in the game has a large influence on eventual success at a task. This is impacted by elements of second and third advantages - i.e. the states in the trajectory are correlated due to state of the resource throughout, and across multiple episodes the current policy might prefer expending or keeping that resource at a specific stage.

The correlations avoided by the second and third advantages typically occur over different scales of time, although that does depend strongly on the specific problem. The second advantage might be gained with a relatively small replay memory (depending on how the state evolves and lengths of episodes). The third advantage will usually require a larger memory, in order that it captures episodes with varying policies. In addition the replay memory should include exploratory actions which also help with this third issue.


K-Means anomaly detection not clustering anomalies

By multiplying the maximum by 10 in a loop, you are repeatedly multiplying by $\sim 10$, so that the final point is $\sim 10^{53}$ (hrm, the plot actually doesn't go quite that far?).  Hence the last added point is so far away from the rest, it becomes too costly to include any other points in that cluster.  (Doing so drags the mean of the cluster very far away, and then the cost function blows up; see the sklearn manual.)


Dropping less frequently used categorical data?

The most straightforward approach would be to consider these small categories as single 'unknown' category - if they're really small, your model shouldn't change drastically.

  Another approach would be to replace all categories to dummie where
  the mean is close to the total mean of target column? then I would
  ditch code_189 (the most frequent) because its varies barely from
  0.072?

This idea seems to be close to mean encoding approach. Mean encoding consists of creating a column where categories are encoded by mean of target for this category. This can incorporate knowledge of small categories naturally - you can just replace their means with global mean, or use smoothing. I encourage you to see this segment on mean encodings (the next video is on regularization).


Behaviour of Learning Algorithms on Random Data

They will of course still learn some best decision boundary. We know it will be meaningless, but there will still be better and best coefficients for the algorithm to learn when fitting to this particular instance of data from this random process. It may produce better than 50% accuracy on the data set, but of course this is purely due to overfitting whatever the data happens to be. It will not predict future outcomes with more than 50% accuracy.


How can I use a string as input in a neural network?

there is no other way i guess. since each one is of different type(Categorical varaible) you can only one hot encode them.


Can you interpret probabilistically the output of a Support Vector Machine?

One standard way to obtain a "probability" out of a SVM is to use Platt scaling.  See, e.g., this Wikipedia page and this question on Stats.SE.  Platt scaling involves fitting a logistic regression model to predict the "probability", based on the distance to the hyperplane.


why an advanced LSTM model produce the same results as a simpler one?

If you can achieve high results with a simpler model, it is great news! Always choose simpler models because they may be wrong on fewer things than complex ones (Occam's Razor).

Nothing to be worried about, this can happen.

Obviously, it is always possible that there is an implementation problem, so always make sure your code works better.

Reproducing results from papers is always difficult since many things can differ between your work and theirs:

Code might be different
Data might be different
Package versions may be different
Randomness might be controlled differently
Weights may be initialized differently
Data might be fed into the model differently
Hardware may be different which, especially with GPU can cause slight difference in results


Difference between packaged sentiment analysis tools (TextBlob/NLTK) and training your own classifier?

In general, training your own classifier is likely to perform better but it's going to take more time and effort:

Probably the NLTK system was trained on some generic data which might be very different from your target text. Since any supervised system assumes the same distribution between the training and test set, it's always better to train on a sample of your target data.
By implementing your own method you can adapt it more specifically to your use case.

So it's a trade-off between the level of quality and the time you're ready to spend on annotating your own data and tuning your own classifier.


Evaluating clusters (e.g. built by kmean) using Random Forest

If you were to have fully formed trees in your random forest, then they should be able to recreate the rules you ended up clustering by, and accuracy would be 100%. Now, if you are cutting off the depth of the trees in the random forest (which is pretty standard, let's say you set max_depth = 8), then you will have an accuracy that may be less than 100%. This means that there are salient features that you are not consistently getting information from because they are not part of the 8 levels that one or more decision trees are using to classify.

My question would then be: why do you want to do this? Once you have the centroids for each cluster using k-means, it is pretty quick to be able to determine which cluster a new record would belong to. I'm not sure I understand why you would want to use a random forest in this fashion.


Linear Regression - Metrics to consider?

Aditya already mentioned rightly so in the comments RMSE, MSE or MAE is preferred while evaluating a linear regression model. So to answer your questions I will provide with the pros and cons of using RMSE and MAE and further more put forward some points to why R-squared is not preferred.

1. MAE vs. RMSE - Which metric is better ?

Ideally this depends on the loss function of your model, the most common one being L2 - euclidean distance (you can create your own loss function too). RMSE gives relatively higher weight to large errors. It does not necessarily increase with the variance of the errors but increases with the variance of the frequency distribution of error magnitudes. While, MAE is comparatively steady.

typically, it makes more sense to give a higher weight to points away from the mean in which RMSE would be preferred. In a scenario where being away from the mean by any magnitude would be penalized equally, MAE would be preferred.

RMSE can be troublesome when calculating on different sized test samples (As is the case when training a model). RMSE has a tendency to be increasingly larger than MAE as the test sample size increases ([RMSE] ≤ [MAE * sqrt(n)].

2. Is R-squared useless ? (Kinda harsh but sorry not sorry)

To keep it short and simple.

R-squared does not measure goodness of fit.
R-squared does not measure predictive error.
R-squared does not allow you to compare models using transformed
responses.
R-squared does not measure how one variable explains another.


Can word embedding be used for text classification on a mix of English and non-English text?

In this case, I think it should be fine to use word embedding on both languages since word embedding learns the meaning of individual words regardless of languages...Is this reasonable? Or maybe I do need to separate the languages and build different models for each language?

  If I think logically you are correct. Word embedding is merely a
  collection of Tokens, which derived its features on the basis of
  nearby words in a sentence. So if you have sufficient raw data(mix of
  both), I think its good to go, though results will explain you more
  :).

However its good to see how such models will behave in case we have mix of LeftToRight(LTR) and RTL languages.


Tensorflow - simple multi-layer perceptron not stabilizing around mean of normally distributed y-values

Try deeper network. Something like this.

self.model = models.Sequential()
self.model.add(layers.Dense(self.num_layers, activation="relu", input_shape=(self.x_train.shape[1],)))
self.model.add(layers.Dense(self.num_layers//2, activation="relu"))
self.model.add(layers.Dense(self.num_layers//4, activation="linear"))
self.model.add(layers.Dense(1))
self.model.compile(optimizer=self.optimizer, loss='mean_squared_error', metrics=["mae"])


i'm assuming this will be migrated to stack overflow, but instead of trying to do if var1=1 or 2 wouldn't it be better to use if var1 in (1, 2)?

...and somebody with enough reputation should probably create an sas tag (and an spss tag while you're at it) unless data scientists only use open source languages like r and python now...


can I say that my variable is "approximately" normally distributed?

Your data is already binned. You should do a Chi-Squared test to see if the hypothesis that your data is normally distributed is correct. You can refer to this question to see how you can go about this.

You will set your baseline as the Normal distribution with the mean and variance from your dataset. Then you will test the difference in expected value between the Normal distribution and yours.


Handling unbalanced datasets with XG boosting

This is a tricky question because it depends on your objective.

If your objective is to have comparable performance on the two classes (i.e. comparable sensitivity and specificity), and the imbalance in your training data is $1:20$, then yes, it makes sense to give the minority class examples a weight 20 times greater than for examples of the majority class.

However, in most applications, this is not exactly the objective. You should use the real costs of misclassification, or an estimation thereof. Ideally, you would quantify the costs of false negatives and false positives (although this is easier said than done, especially when comparing the cost of illness to, for example, the cost of a medical professional's wasted time).

Then, if your training set had the same class priors (how often a class occurs) as the real data, you would use these costs directly as weights for, respectively, the positive and negative classes.

In your case, it's more complicated, because the priors in your training data are different from those in the real data. You cannot use the misclassification costs directly as weights, you have to adjust for the real distribution.

Using a dataset with a $1:20$ imbalance instead of $1:2000$, you've effectively already given the negative class about 100 times more weight than it naturally has. It's like there's already a weight hidden in there. For the misclassification costs to be used as weights, you would have to cancel out that factor.

Suppose $c_{fp}$ is the cost of a false positive (misclassifying a negative example) and $c_{fn}$ is the cost of a false negative (misclassifying a positive example). If rate of positive data is $1/20 = 0.05$ in the training data and $1/2000 = 0.0005$ in the real distribution of your domain, then you would use:

$$
\frac{\frac{1}{2000}}{\frac{1}{20}}\cdot c_{fp} = \frac{c_{fp}}{100}$$
as the weight for errors on the negative class and
$$\frac{\frac{1999}{2000}}{\frac{19}{20}}\cdot c_{fn} \approx 1.052 \cdot c_{fn}$$

as the weight for errors on the positive class.