How to identify new clusters that the training data has never seen

A possible way to do this is to learn some compressed latent representation of your training data, and check how well new data matches this representation.

In practice, you could try training an autoencoder on your training data, by minimizing the reconstruction error between its input (your raw features) and its output (the reconstruction).

After training, you can check whether a new operational state comes up by checking if the reconstruction error is larger than a threshold. The intuition behind this is that if the new data belongs to a known or normal operational state that the model learnt from the training data, the model will be able to reconstruct it properly and the error will be low.


Should augmentation also be performed on the validation set when the dataset is imbalanced?

You have the idea of augmentation wrong I suppose. Image augmentation is used to introduce variations in your existing image dataset by using different operations like rotation, slicing, mirroring etc to make the model more robust. But using image augmentation on unbalanced data would keep the resultant data unbalanced as long as all the operations are performed on the entire dataset. So the question of performing augmentation on just train or both train and validation doesn't make much sense if you look at it the above way.
I would recommend you to opt for image augmentation only if you feel you have less examples to train from and also you want to make your model more robust but do it on your entire dataset.
You can oversample your minority class as well if you feel it's suffering during training or recall is less.
To oversample your minority class, you can perform image augmentation on your minority class only.


Unsupervised learning if existing image captions match the images

A thought:

Maybe an approach like Show and Tell: A neural image caption generator would be useful to look at. There they use exactly as you say a combination of CNN (for image) features and LSTM (for text) + search to generate text given the features from the image.

It is possible that then a discriminator can be used to learn whether 'generated text' is close enough to 'given label' in order to give the final result.

It is also possible that you do not need the discriminator, and instead of using the output of the show and tell network directly (like removing the last linear layer) and feed one of the earlier steps (which represents a probability distribution of possible texts to generate) into the discriminator so that it has more possible combinations of text outputs to work with...

This is probably more supervised than you want, but maybe starting with pre-trained weights of this model can get you somewhere faster.


Is image sharpening a good idea for data augmentation?

At first, I have to mention that $5k$ cannot be considered as a large dataset for training a deep neural network. Anyway, about the question. In general, yes you can, but you have to be aware of some points. Data augmentation can be helpful or it can damage your entire predictions. The reason for each is that whenever you utilize data augmentation, you are somehow changing and manipulating the distribution of your data in hand. Because neural networks are considered to deal with random processes which their behaviour is iid, your data samples should not be dependent on each other. There is also another perspective. Your training data should have the same distribution as your test data. If you are sure that while testing what you are going to face can be sharpened images, so go ahead and carry out that, but if you're sure that your sensors are placed in a noisy situation, something you can encounter in self-driving cars, you can be pretty much sure that your raw data is blurry, and it is almost impossible to have sharp images due to velocity unless you take a preprocessing step and after that you feed it to your network.


No graph is displaying while plotting value with time interval using python

In python you generally have all the libraries available to you. It is hard to find sometimes but you should rarely need to write out so much code. Try this out.

I created some dummy data using the same date formats as you have:

import pandas as pd
import matplotlib.pyplot as plt

data = {'date': ['08/06/2018', '8/6/2018', '8/6/2018', '9/6/2018'],
        'time': ['6:15:00', '12:45:00', '18:15:00', '6:15:00'],
        'x2': [1, 4, 8, 6]}

Now we will make a pandas DataFrame with this dummy data

df = pd.DataFrame(data)

Now we can get our x-axis datetimes by first concatenating the dates and times together separated by a space. Then we will get pandas to parse these datetimes.

datetimes = pd.to_datetime(df['date'] + ' ' + df['time'],
                           format='%d/%m/%Y %H:%M:%S')

You can then plot your data using

plt.plot(datetimes, df['x2'])

Put your csv file in your workspace. Then you can use this following code

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv(r'temp.csv')
datetimes = pd.to_datetime(df['date'] + ' ' + df['time'],
                           format='%d/%m/%Y %H:%M:%S')

plt.plot(datetimes, df['x'])
plt.show()

import matplotlib.dates as mdates

fig, ax = plt.subplots(1)
fig.autofmt_xdate()

plt.plot(datetimes, df['x'])
plt.xticks(rotation=90)

xfmt = mdates.DateFormatter('%d-%m-%y %H:%M')
ax.xaxis.set_major_formatter(xfmt)
plt.show()


Ways to simulate weather data over several periods (Python or R)?

What about creating a Pandas DataFrame and adding a new column such as "Temp_simulated" and simulate the temperature?


How can I minimize features of the trainded model?

There are several algorithms which can help you in a smart way.

Usually, those algorithms are used to tune the hyperparameters of a model, so this is what you will find in the tutorials/examples. In your case, you have to find a good set of features instead of a good set of hyperparameters, but the principle is the same.

My suggestions:

1) SMAC. This is based on Bayesian optimization. It's an iterative process where a proxy function is built and maximized:

the function to be optimized (your XGBoost model) is evaluated in a point (in the feature's hyperspace) where the optimizer believes it can find the maxima (or, in the very first iteration, in a point given by the user);
the result is added to the set of all the evaluation points, and this set is used to build the proxy function;
the proxy function is maximized, and the coordinates of that maximum are believed to be the same where the original function will have a maximum too.

Those three steps are repeated as much as you want. So, repeat from the first step;

It works both for continuous and for categorical features, and you can also impose some constraints between features.

Here an example for your case, in Python (code not tested):

from smac.configspace import ConfigurationSpace
from ConfigSpace.hyperparameters import UniformFloatHyperparameter, UniformIntegerHyperparameter
from smac.scenario.scenario import Scenario
from smac.facade.smac_facade import SMAC

#a continuous feature that you know has to lie in the [25 ~ 40] range
cont_feat = UniformFloatHyperparameter("a_cont_feature", 25., 40., default_value=35.)

#another continuous feature, [0.05 ~ 4] range
cont_feat2 = UniformFloatHyperparameter("another_cont_feature", 0.05, 4, default_value=1)

#a binary feature
bin_feat = UniformIntegerHyperparameter("a_bin_feature", 0, 1, default_value=1)

#the configuration space where to search for the maxima
cs = ConfigurationSpace()

cs.add_hyperparameters([cont_feat, cont_feat2, bin_feat])

# Scenario object
scenario = Scenario({"run_obj": "quality",   # we optimize quality
                     "runcount-limit": 1000,  # maximum function evaluations
                     "cs": cs,               # the configuration space
                     "cutoff_time": None
                     })

#here we include the XGBoost model
def f_to_opt(cfg):

    #here be careful! Your features need to be in the correct order for a correct evaluation of the XGB model
    features = {k : cfg[k] for k in cfg if cfg[k]}
    prediction = model.predict(features)

    return prediction

smac = SMAC(scenario=scenario, rng=np.random.RandomState(42),
        tae_runner=f_to_opt)
opt_feat_set = smac.optimize()

#the set of features which maximize the output
print (opt_feat_set)

2) dlib optimisation. This converge much faster than the previous. As disclaimer, I have to say that this is an algorithm which in principle works only with functions that fulfill a certain criteria, and XGBoost models as functions do not. But in the reality it turns out that this procedure works also for less stringent functions, at least in the cases I tried it. So maybe you want also give a try.

An example code:

import dlib

#here we include the XGBoost model. Note that we cannot use categorical/integer/binary features
def f_to_opt(cont_feat, cont_feat2):
    return model.predict([cont_feat, cont_feat2])

x,y = dlib.find_max_global(holder_table,
                           [25, 0.05],  # Lower bound constraints on cont_feat and cont_feat2 respectively
                           [40, 4],    # Upper bound constraints on cont_feat and cont_feat2 respectively
                           1000)         # The number of times find_max_global() will call  f_to_opt


Which time series analysis is appropriate for pooled time series data analysis?

It really depends on the outcome you want from your analysis.

The most straightforward approaches might be resampling:

Interpolate the 10-year census data to month granularity. The interpolation might be linear, polynomial, etc. This effectively imposes a strong assumption in the dynamics of the data. This may make sense for some variables but not for others. And you are probably introducing a lot of variance in the obtained conclusions.
Aggregate the monthly time series into 10-year granularity. This may not be very useful if you want to predict something with an horizon less than 10 years...


Problems with Linear Discriminant Analysis Classifier

As you say; it has something to do with the shape of the matrices. In this case, you have matrices in the same of (155,2) - whereas it's expecting them to be of (310, 1).

Try looking into the shape of each matrix as you transform them (tip: insert a print statement wherever a transformation occurs); and make sure everything stays consistent.

Also, can you provide us with more information, such as exactly where this error occurs? It looks to be an issue with the dot product.


Maybe you are making a mistake, put your code here. But without seeing your code, these are possible points:

Vanishing problem, I don't think you this problem due to having a very shallow network. You can change your activation function to relu for avoiding that.
Covariat shift, What it means is that similar to input features which have to be normalized, the inputs of the deeper layers have to be normalized to. The normalization process is in a way that the inputs to each layer should have a special distribution that does not change. You can use batch normalisation for avoiding that problem.
Bug in coding, you may have fed the activations of each layer to the next layer in a wrong way or you may have updated the weights not simultaneously If you have not used vectorization. There are also numerous different reasons which may lead to bugs in your code.
The number of neurons may not be enough for each layer, try to increase them. For understanding the meaning of increasing the layers and the number of neurons in each layer, you can take a look at here.


How to deal with broad and narrow variance within classes in classification tasks

You will want to have many diverse examples of the high variance classes and pick a model that has a high learning capacity. In other words, a large number of samples and a big model to capture the properties of the data.

After training the model, look at the confusion matrix by class for the hold-out dataset to see if the model is learning to generalize for all classes, including the high variance classes.


Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)

Question 1:
To do so, I would use the Gensim wrapper of FastText because Gensim has a predict_output_word which does exactly what you want. Given a list of context words, it provides the most fitting words.
Question 2:
It is up to the user. FastText isn't inherently CBOW or Skipgram. See this
Question 3:
Yes, even though CBOW and SkipGram are different training procedures, they share a common goal. Both will generate word embeddings where (hopefully) words that are semantically close also have embeddings that are close. The main difference between SkipGram and CBOW is the inherent heuristic used for semantic closeness.


Why is Distributional DQN faster than vanilla DQN?

This is meant to be a comment, but I can't comment since I have insufficient reputation.

As for the second question, intuitively speaking, instead of taking a scalar value for an action, which initially, may be highly inaccurate and noisy, taking a distribution instead would be more accurate. I'd recommend https://flyyufelix.github.io/2017/10/24/distributional-bellman.html which explains the intuitive reason for using a distribution

In terms of convergence, actually, there is no guarantee of convergence. In the paper, however, explains that for distributional DQN to guarantee to converge, the gamma-contraction must be satisfied, which would be true if you measure the the distance between the distributions using wasserstein distance, but it would be impractical to try to minimize that distance, so distributional DQN uses cross entropy instead which you can find the gradients of, and perform backpropagation....etc

You may be interested in "Distributional Reinforcement Learning with Quantile Regression"
https://arxiv.org/pdf/1710.10044.pdf
which aims to improve the original distributional DQN algorithm


Python Pandas - Interpolation with fixed value

Current solution, I am using

def interpolate_with_fixed(s, value=0):
    i = s.first_valid_index()
    j = s.last_valid_index()
    s.loc[i:j].fillna(value, inplace=True)
    return s


CUDA_ERROR_OUT_OF_MEMORY: out of memory. How to increase batch size?

Environment:
1.CUDA 10.0
2.cuNDD 10.0
3.tensorflow 1.14.0
4.pip install opencv-contrib-python
5.git clone https://github.com/thtrieu/darkflow
6.Allowing GPU memory growth

Reference


Analyzing country data. Does duplicating observations by population for regression make sense?

I see what you're doing, and understand the thinking behind it, but don't think it makes sense with this problem. By assigning 300 variables the same values, you're effectively saying that each of these million people have the average economic output (i.e., if you're using GDPpercapita) for all the people in your dataset. That's a pretty misleading understanding of the distribution of that variable, and will lead to poor analysis. Same goes for other aggregate factors.

If you're looking to do a regression where those are incorporated into your analyses, using the unscaled population data as one of your features will include that information as one of the coefficients you return. So you can get back the result on happiness of having a million more/less people. No need to duplicate the data.


Decision Tree Optimize Deviation From Objective

It is possible, is not an easy task but it's possible.

Custom Loss Function

  Gradient boosting is widely used in industry and has won many Kaggle competitions. The internet already has many good explanations of gradient boosting (we’ve even shared some selected links in the references), but we’ve noticed a lack of information about custom loss functions: the why, when, and how. This post is our attempt to summarize the importance of custom loss functions in many real-world problems — and how to implement them with the LightGBM gradient boosting package.


Does it make sense to parallelize machine learning algorithms as part of PhD research?

From practise point of view, just sharing some thoughts. I don't have any research (PhD) type of experience, and your supervisor may communicate similar contents below.

And, paralleling calculation here, personally, I defined as single PC that utils graphical card GPU cores for calculation acceleration as oppose to clustering calculation.

Some Theoretical Thoughts

Sometimes, it worth building multiple models, or even, same model multiple times on the same data in ML world.

The example would be like:

using cross validation (same method multiple times) to have robust model output or parameters. Or
model ensemble with lots of weak learners (multiple models) to have better accuracy.

Those ML process are time consuming, and paralleling can help reducing the time.

Also, from the info provided, I assume your ML project is image recognition with GPU acceleration. But I've no idea what's the main purpose of the project.  It could be developing/improving new ML methods, or comparing known ML methods to form a academy review.  Whatever the case, I assume, the result should still achieve some certain accuracy level.

Hence, it would make sense to consider some efficient method (e.g. paralleling calculation) to accelerate the modelling process.

Some Practical Considerations

In practise, efficiency is very import.  When you have a theoretically accurate model, but takes a long time to build models, it won't be acceptable.

You can step back to say let's come back to some simple quick model with less accuracy. But what's the point of ML when traditional methods have similar or better accuracy?

Personal Answer to Questions

1) Should I spend a large chunk of my time parallelizing with CUDA?
Paralleling is useful to make the ML training quicker.  And CUDA is a cool technique to learn and apply to ML.  Just balance the time between main purpose of the project and time.

2) Is CUDA still a viable framework for research?
This meant to be the first chapter of your report / essay / dissertation.  At least, more and more ML uses deep learning, which can be much quicker with GPU involved.

3) In the world outside of research, will this make it easier to get a ML job?
I cannot definitely say knowing CUDA/paralleling would make you a top player (as other aspects would also be considered). But when other candidates have similar background, one with paralleling calculation experience would stand out.


How to create a matrix from two given vectors in R in RStudio?


How to give importance to recent data in a large dataset

Time-series methods like LSTM, Kalman filters and Markov models handle this automatically. So, assigning more importance to more recent data is in their structure.


How to incorporate keyboard positions on character level embeddings?

Character keyboard position information is an example of noisy channel model information, an error that depends on how a word is transmitted. It is very common to add noisy channel model information to spell checkers, including spell checkers that use character-level embeddings.

Most character-level embedding models would automatically learn to model common transmission mistakes. Characters that are frequently confused in the dataset would be embedded closer to each other because they frequently co-occur. There would be minimal gain by explicitly adding channel information to a character-level embedding model during training.


You can look these paper. First one is quite related to your task.
http://icamt2016.org/papers/SS-LTMLBDA-06-05.pdf
http://researcher.watson.ibm.com/researcher/files/us-kephart/icml00_qrt.pdf


I'm not sure if it meets all your criteria (mostly because I'm not sure I understand all your criteria!), but you could have a look at ELAN:

  Description: With ELAN a user can add an unlimited number of textual
  annotations to audio and/or video recordings. An annotation can be a
  sentence, word or gloss, a comment, translation or a description of
  any feature observed in the media. Annotations can be created on
  multiple layers, called tiers. Tiers can be hierarchically
  interconnected. An annotation can either be time-aligned to the media
  or it can refer to other existing annotations. The content of
  annotations consists of Unicode text and annotation documents are
  stored in an XML format (EAF).

See also the Wikipedia page about ELAN. There is also a Wikipedia list of related software, I don't know if this is relevant.


update model prediction for each day closer to the event

I would rephrase the problem into a business problem. I.e., What we are trying to accomplish here.

From the sound of it, it seems like a Churn prediction problem. The article outlines some ideas and potential solutions.

In simpler terms, if the goal is to prevent churn by executing the right marketing messages, etc.

How can we predict the risk of someone leaving? Since you have historic data, it can be a supervised learning algorithm to help you identify such risk.
What other features do we want to add to the model? Actions / usage data / web visitation / email open, etc. leading to the renewal days. You can normalize the data by the days prior to renewal, etc.
Determine what actions you want to take; it will help you determine if you want to get a risk score, or a classifier with differing level of risk, etc.


Use this syntax:

val joinedDF = students_positive_grande.as('a).join(
                          df.as('b),
                          $"a.Customer_ID" === $"b.Customer_ID")

joinedDF.select($"a.Customer_ID", $"b.Customer_ID")


If you decided to remove outliers. Please remove them before the split(even not only before a split, it's better to do the entire analysis(stat-testing, visualization) again after removing them, you may find interesting things by doing this).

If you remove outliers in only any one of train/test set it will create more problems.
(EX: An outlier in train set may not be an outlier in combined/full set, also the model will have high variance if you do so)


Why Matlab neural network classification returns decimal values on prediction dataset?

Even for classification problem, the output activation can be any slashing function like sigmoid, tanh, using which the error is back-propagated.

I guess, its trying to give you continuous value for a labels more like probabilities, you use a cut off (say 0.5) and convert the values to 0 and 1 labels


Visualizing deep neural network training

The closes thing I know is ConvNetJS:

  ConvNetJS is a Javascript library for training Deep Learning models (mainly Neural Networks) entirely in your browser. Open a tab and you're training. No software requirements, no compilers, no installations, no GPUs, no sweat.

Demos on this site plot weighs and how do they change with time (bear in mind, its many parameters, as practical networks do have a lot of neurons). Moreover, if you are not satisfied with their plotting, there is access to networks parameters and you can plot as you wish (since it is JavaScript).


Mathematical bias and weight vs machine learning bias and weight

Okay so let's start with the first question:

Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).

You'd be correct had you used the word difference rather than distance. Bias is the difference between the estimated value and the true value. Think of it in this way, if your weighing machine always showed 5kg less for every measurement then you'd be adding 5 to every weight it measures, as a way to offset all measures made by the machine. So if your data is 'biased' towards one side then this bias term offsets this behaviour by adding/subtracting the 'biases of the weights. Here's some more rigorous explanation to it.

what is the difference between loss and bias?

Loss, on the other hand, is a measure of how close your estimator is to true prediction, there are different loss function that one can employ with different type of training, for linear regression we can use MSE. Using loss functions we can objectively optimize our weights and biases. So, having an appropriate value of bias(and weight) causes our loss to be lower, thereby increasing the accuracy of our model.

So my confusion is that is the procedure of finding weight is same as finding slope in mathematics?

The procedure is different, but the end result is the same. To see why you'd need to realize is that you are calculating $\hat y$ as the following: Given $m,b$ and a point $x$, calculate the value of $\hat y$ which is then used to calculate the loss. This value along with $m$ and $b$ is then used to calculate a better value for $m$ and $b$ using Gradient Descent. Notice at no point are we talking about directly calculating $m$ using $\hat y$, We are iteratively calculating better values for weights and biases. The end result? For single-dimension data, the weight that the model has learnt is indeed the slope (when plotted) and bias is the y-intercept. This stands true as the number of dimension increases but then rather than being single values, it becomes a multi-dimensional vector.


Making Use of the Target Values for Regression

Clustering is definitely something that can help. As you describe, the issue is that you can see that not all instances "behave" the same way. So if you can cluster them into groups that do behave more similarly, you would probably improve.

Any clustering technique would work in theory. Although, techniques like K-means for instance do favour clusters of even-size, and so if your problem isn't necessarily "balanced", I would be cautious.

What you could also do to improve your clustering is to look at which features are more discriminative (perhaps look at the information gain) and use a weighted distance to favour these features.

Another way I feel could help you is if you used meta learning/stacking. By that I mean that you could use decision trees to split your data in a supervised way (as opposed to clustering), and at the leaves, use a more elaborate regressor than majority voting.


How to Keep Missing Values in Ordinal Logistic Regression

Computationally, there is just no way to use the data with the values missing. You do still have a whole three options, though:

Drop the rows. This is "listwise deletion." Like you said, you don't want to do that - and it can cause biased results, as the observations with missing values may be ones that speak against some theory, or (in more general terms) represent some consistent data-generating pattern.
Drop the column. This will eliminate any bias due to the above, at the cost of inefficiency. It may also be a non-starter if the effect of this variable is what you're concerned with anyway.
Impute. You can fill in the missing values with your best guesses (i.e. the maximum-likelihood point-predictions of an imputation model - or even just the "null model", i.e. the global column means) and treat them like real values, though this is bad practice because it doesn't reflect the inherent uncertainty about imputed vs. actually observed values. Standard practice is to ensure your imputation model produces a best-fit distribution for each observation, generate 5 versions of your dataset with 5 sets of random draws from those observation-specific distributions, fit the same model to each of those datasets, and average the results.

Note, though, that if the model is an arbitrarily complicated machine-learning algorithm (i.e. a universal approximator like a neural net), then you can just fill in the missing values in each column with any "free" value - one that's not actually present in the data, and won't be - and the model will learn to treat those values not just differently but appropriately. It will effectively do its own imputation, in fact. But still, you can't have the actual inputs as missing values.


Finding the appropriate CNN Model Architecture and Parameters

In many cases in deep learning it works well  to start off with a model which has a very high capacity and potentially overfits. From thereon you can reduce the model capacity to narrow the gap between train and validation error. In this chapter of the Deep Learning Book by Goodwell you find a good description of manual hyperparameter selection and how they influence model capacity.

Moreover, for many tasks well engineered solutions already exist. So check what worked for similar tasks and try out these architectures. For example, MNIST handwriting recognition is somewhat similar to your task. Wikipedia gives several architectures which work well on MNIST.

The article "Handwritten Digit Recognition using Convolutional Neural Networks in Python with Keras" contains architectures for MNIST, too. Again, this might be close enough to your tasks so I suggest to give these a try. The article includes an architecture very similar to yours but also a more complex one.

In other cases you could also check out pre-trained models. But here it might not even be required here.


Strictly speaking, the k-means algorithm does not have a definition for "inside the cluster" and is therefore not a great candidate for anomaly detection.  In k-means, every point is assigned to one of k clusters and then a new cluster centroid is calculated.

But as previous uses have pointed out, you could construct some sort of ad-hoc system where you process a set of data and then define new data as anomalous when its extends beyond 2 standard deviations of the centroid location. DON'T DO THIS!

K-Means will not work well for this ad-hoc method.  If k is poorly chosen, then the distribution within a cluster will not be normally distributed.  You very, very frequently see natural distributions of points which are split between two clusters.  For instance, take a look at the ad-hoc segmentation of points in this location data for a cell phone user's location over a month:

I suggest you use another clustering method.  The first option that comes to mind is DBSCAN.  This allows one to set a threshold for noise and the cluster numbers are not set a-priori.  DBSCAN is therefor much more likely to return normal distributions within a cluster.  Here is a single DBSCAN cluster of the same data:

Finally, I'll point out that the method you are proposing is not as good as other novelty and anomaly detection methods.  You should consider doing novelty detection using a single (or possibly even multiple) class support vector machine (SVM) with a nonlinear kernel. The nonlinear kernel will allow you to recover multiple "clusters" while the SVM will do much better at predicting which points are inside the class.


It's not exactly mandatory to start recording at the same time, but it would be helpful.

You definitely need to be able to align the timeseries recorded by the microphones.  If they start recording at exactly the same time, then the time-series will already be aligned.  If they don't start at the same time, then you need to know exactly how much time passed between microphone 1's start time and microphone 2's start time, so you can align the timeseries after-the-fact.  Also, you won't be able to run ICA for the time period when microphone 2 was off.


Is training in Biostats sufficient to get into Data Science?

a question, and then some comments.  First, what kind of market analyst do you want to be?  This field is rapidly becoming specialized, and the answer will affect the decision.  If you want to specialize in the pharma or especially the medical field, the biostat background could be an advantage.

Generally speaking, you need to think outside the box of what you will learn purely as defined by the program, either stats or biostats.  You will need to branch out into areas not defined by either program.  This could include (1) software language, (2) data science problems (Kaggle or external projects run through a campus department), or (3) courses not expressly given in either department, whether it's in "hard" skills like data mining or "soft" skills like marketing.  Ask for exceptions to do things that don't fit the program, but fit your career goals, because being a market analyst doesn't fit squarely into either field.


Can natural language generation algorithms generate valid words too?

Outside of context of NLG (thus not a direct answer to your whole question, but an answer to your question's title): Generating words from a character-level model has been done using RNNs exposed to large corpora of text, such as Wikipedia content, and trained to predict text character-by-character.

Used to generate content, the model is normally fed a few starting characters and asked to predict the next one. A choice is made from its most-likely predictions and fed back to it to continue the sequence.

Here is a blog showing some examples trained on some Shakespear and Wikipedia.

Such a network can and does generate nonsense words, although they are often fitting and might read like e.g. a noun or verb as you could expect depending on context. The sentence structure and grammar can come out sort of right, but the semantic content is usually complete gibberish.


how to transform a folder of images into csv file?

import os
import pandas as pd

BASE_DIR = 'images/'
train_folder = BASE_DIR+'train/'
train_annotation = BASE_DIR+'annotated_train_data/'

files_in_train = sorted(os.listdir(train_folder))
files_in_annotated = sorted(os.listdir(train_annotation))

images=[i for i in files_in_train if i in files_in_annotated]

df = pd.DataFrame()
df['images']=[train_folder+str(x) for x in images]
df['labels']=[train_annotation+str(x) for x in images]

pd.to_csv('files_path.csv', header=None)

Try this.


What algorithm can be used to divide a time series

First of all, there is nothing wrong with using a threshold.

If you can't manually choose the threshold, use standard statistics. For example trim the largest 10% of points. Then compute the mean, standard deviation, and use 2*stddev as threshold.


How to build a comunication interface between a reinforcement learning and a videogame?

You might wanna check OPEN AI GYM The project has several environments for atari games, robotics and a lot of other useful resources on reinforcement learning.
I think this good place to start. If you need a bridge for some specific game, you need to google, for example there is repo for TorchCraft - A bridge between Torch and StarCraft.


Is there any way to explicitly measure the complexity of a Machine Learning Model in Python

I have not heard of any model agnostic way to measure model complexity. There are several strategies but they are model dependant.
You can tackle the problem using different families of models.

For linear models you can count the number of nonzero parameters that is using. Number of features used for the prediction.

For decision tree you can count the maximum depth that the tree achieves.

For Neural Networks you can count the number of parameters that your NN is optimizing.

For ensemble methods (random forest, gradient boosting) you can use an aggregation of the different weak learners used in the model.

For the python implementation there are several implementations depending on for which model you want to measure it. Some of them if you notice are really easy to measure.
Its intuitively hard to compare complexity between different model families. What is more complex a linear regression with 4 coefficients or a decision tree with max_depth=3?
On the topic of deep learning complexity, Hinton, Oriol, Jeff Dean published a paper Distilling the knowledge of a Neural Network. Where they talk about simplifying the complexity of a Neural Network.


a simple way to test wether a tree-based classifier would transfer well to a target population?

Not sure if you've tried this already, but you might dig into XGBOOST feature importance to establish how your model is making predictions, then do a more in-depth comparison between the two populations at those splits.

This isn't very different from what you've proposed, but it does make for a more focused analysis; you might have time to look at higher order interactions between the more important features. (I'm assuming it's not possible to cheat by training a classifier on your population B and comparing   how each model makes predictions!)

ELI5 and LIME also come to mind for debugging feature importance and explaining model prediction.


Should the cost function be zero using TensorFlow's sigmoid_cross_entropy_with_logits?

For a binary classification (1 or 0), you should use softmax cross-entropy with logits since the output classes are mutually exclusive / discrete.

The function you used is for cases with multiple outputs (multilabel) all of which can be ones (their documentation for example talks of the case where image can contain both elephant and a dog). The calculation is different than mutually exclusive case.


How to limit a function's cosine value as it approaches 0

$\alpha e^{-\beta x^2}cos(x)$ where $\alpha$ is your peak and $\beta$ is proportional to the strength of the decay
This one's slightly longer and the peak is at $\alpha+1$ but the graph is easier to manipulate:

$cos(x)e^{-\beta x^2}+\alpha e^{-x^2}$


Calculating distance between data points when there are more than 3 features in KNN algorithm

No, you can definitely search for k-NN with more than 2-dimension data. Here is an example based on sklearn:
X = [[0, 0, 0], [3, 3, 3], [1, 2, 3]]
from sklearn.neighbors import NearestNeighbors
neigh = NearestNeighbors(n_neighbors=2)
neigh.fit(X)
print( neigh.kneighbors([[2,2,2]]) )

PCA is used to reduce the input dimensionality but this is not mandatory before searching k nearest neighbors (it is often used in tutorials so the data could be visualized on a 2-d plot).
One thing to know/understand about k-NN is that if you plan to use it for classification, it will handle features with a lot of information the same way as the features with no information (if you normalize them). PCA could be used to handle this problem (but this is not the only way and would not always work, but I think this is another question :) ).


This is a bad idea. The cross validation score that you obtained from your first model is valid only for that model. If you train a new model there is no guarantee that it will have similar performance, you have to test each and every model.

This could work only if you can guarantee that the data for the second model will be very similar to the data of the first model. You would have to be very sure of that.


Assuming you want to learn sentiment this is a problem. What happens when you feed this to a Machine Learning algorithm is that it will give more weight to the tweets that are in there multiple times, while not learning more information. It's unlikely that a tweet that has been retweeted 10 times carries more significant information about sentiment than one that hasn't been retweeted. What you could do is add the number of times the tweet was in the set as a feature to your sentiment model, it's possible something can be learned from that fact (maybe positive tweets are retweeted more often), but you should keep it at 1 row for every distinct tweet.

Getting rid of these redundant records should not be difficult programmatically though. I don't know what language you are using but if you only consider the body of the tweet (the content) you could iterate over your tweets, keep a list of all the unique bodies, combined with other meta information (like user, labeled sentiment) and if the content of the next tweet is already in there, just do not add it. Look for 'distinct' functionality as opposed to redundant, enough information out there.


Running Neural Network experiments in loop

I using InteractiveSession() and closing it every time worked

sess = tf.InteractiveSession()
#code here ...
sess.close()


How to dual encode two sentences to show similarity score

How can I create a dual encoder though? Do I use two different neural
  networks? Or does output just contain one neuron that outputs
  similarity?

You don't need to create two different neural networks (or two different encoders) for it. Whenever you have a symmetric pieces problem, like in your case, its two sentence similarity, you should encode both the sentences using the same encoder and then use cosine similarity. (It can be the similarity of two faces also - just the way FaceNet is expected to be used.)

  we will perform sequential search on whole database by comparing
  cosine similarity (highest similarity vector is picked). But this is
  extremely slow...

You are right, I am assuming you would be using a for/while loop for computing the similarity. In computing the vector similarity of one sentence against a database of thousands or millions, you should make use of a high-performance library like BLAS which would do the same calculations sub-second.

Facebook three years ago came out with a faster matrix library called FAISS. This would reduce your speed drastically.


Input shape for simpler time series in LSTM+CNN

For LSTM in tensorflow the tensor has three inputs. So, let's assume we have:
[samples, time steps, features]. This means that you have n number of samples, and each sample is divided in m time steps. All of the samples have the same number and the same features.

So in your case n_steps, n_length = 4, 32 means that n_steps are going to be taken in the data with 32 samples, meaning each 4samples are fed into the LSTM at one single point of time. Or 8 different subsets are going to be fed into the LSTM. LSTM usually takes subsets(not single row-samples!)

According to your reshape  (1000,100,1) you say that you have 1000 samples(imagine them as subsets of the dataset) and each of these 1000 subsets you divide into 100 time steps. This means that in each of the 1000 subsets, you must have at least 100 samples or a number divisible by 100, otherwise, it won't work. The third parameter "1" is referring to the number of features in your dataset/subset. When you work with accelerometer data I don't believe that you work with only one feature. My guessing is that there you need to have at least 3 features(x,y, and z) or maybe more.

Please check their documentation site: https://www.tensorflow.org/tutorials/sequences/recurrent


Why is it necessary to clean/preprocess data if the unseen real data is bound to consist missing values, outliers etc.?

It depends. Actually there are research papers finding that neural network can sometimes cope very well with sparse or noisy labels. But basically when you apply any machine learning approach, you want your model to pick up patterns in your data. In order to be a pattern something needs to be repetitive and predictable. Outliers and errors in your data might not adhere to that. They rather break the pattern. It can still work but it is harder to recognize it.

Making a model "learn" outliers/missing values etc. will only work, if these incidences also appear in a pattern which makes them unlikely to be outliers. As a rule of thumb it is therefore a good idea to remove this noise.

Finding patterns in exceptions & errors

As I mentioned some researches try to address the noise issue for example by classifying if there is random ("white") noise or structured noise. They not only predict the target value but also the type of noise and then try to make another prediction based on both pieces of information. So there are some active approaches to address noise but usually this is due to the fact that it would be to cumbersome or even impossible to remove the noise.


Why is large decision tree likely to overfit

Overfitting means that a model is giving a good fit on a dataset (whatever the measure you use to assess fit), but this is not a general case (i.e. when new data comes in or on another dataset, the error will explode. Or said otherwise, the model variance is high).
In the case of trees, adding a node to a leave based on one feature should be done only if the feature really brings information at this level. The feature could be random though and this would deteriorate greatly the fit.
As a simplistic example on a classification task; if we want to sort out apples and oranges based on some features, including one of the features that is a value, 0 or 1 chosen at random.
If it happens that in our dataset the values 1 correspond in 80% of the time to apples, then we would be tempted to add a node saying "if value is 1, then apple", but you can see that this is absolutely not a generality: the tree wouldn't fit at all to another dataset.
Hence this node shouldn't be added, i.e. the tree should be pruned.


Rather than oscillations, it looks like white noise, like a random walk. In other words, as you said, your model is not learning anything.
Unfortunately it's impossible to say what's wrong, since we can't see any code. We need more information about dataset, how you processed it, model implementation, all the hyperparams you chose, what other versions you tried before that one, ... the list is countless. But most importantly it's really hard to help you without code.
If the dataset is a good one the problem must be some error you made along the way.

EDIT:
Here's what I think:

You don't need Conv layers followed by RNN layers. This doesn't really make sense. Let the LSTM receive raw input.

Don't use Dropout with RNNs, they don't go along very well together. Dropout makes sense with Dense and Conv data, but in RNNs, where sequence is everythin, they can actually make things worse. Somebody uses recurrent dropout as an alternative but it's not necessary.

Don't use return_sequences=True between an LSTM and a Dense layer. That must be used between LSTM's only.

That Lambda layer at the end is probably causing most of the error. If you multiply all your predictions by 1000, what you get is by definition a prediction that on a completely different scale than your target value.

The Network overall is too deep and has too many parameters. I assume you are working with the famous Beijing air quality dataset. In this case, It is enought to work with one LSTM layer, followed by a Dense node to make the prediction. Everything else is overkill and not necessary for a simple dataset like that.

Something much more simple, like:
model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(24, input_shape=(seq_len, n_vars)),
    tf.keras.layers.Dense(1),
])

has higher chance to work. (Please specify the input shape correctly). Try playing with its hyperparameters, after you made sure all variables are properly scaled between train and test data.
Good luck!


How to select features when performing classification with a dataframe of multiple columns?

You could use logistic regression with regulation (Lasso, Ridge, Elastic Net) to get an idea of what features are relevant. Regulation means, that features which are „not so important“ are shrunken. In R glmnet is a good starting point which is also available for Python.

R: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

Python: https://web.stanford.edu/~hastie/glmnet_python/

See Section 6.2 in Introduction to statistical learning (ISL) for some background and examples. You may find a PDF of the book online. http://www-bcf.usc.edu/~gareth/ISL/

Ridge is based on the L2 norm, so that model coefficients are shrunken, but cannot be zero. Lasso in contrast uses the L1 norm, so that coefficients can be shrunken to zero.

Another approach to model selection would be forward / backward stepwise selection (see: ISL, Ch. 6.1).

For forward stepwise selection, the process would be:

Let M0 be the NULL model, which containes no predictors. Now for k=0, ..., p-1:

Consider all p-k models that augment the predictors in M with one additional predictor.
Choose the best among these p-k models (e.g. based on smalles RSS / best R2).
Finally, select a single best model from the M models using cross-validated prediction error, AIC, BIC, or adjusted R2.

Backward stepwise selection works in the same way, but you start with the full model and successively leave out predictors/features.

For Python, sklearn comes with a greedy stepwise (backward) selection module: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html

Also note, that many methods/packages come with a module that allows to assess feature importance, e.g. in catboost (https://catboost.ai/docs/concepts/fstr.html) or lightgbm (https://lightgbm.readthedocs.io/en/latest/Python-API.html). However, a disadvantage here is, that you need to implement a proper model (with some hyperparameter tuning) first, which is time consuming.


The key difference is that statistical analysis reports on different aspects on your data, while data mining mines for a new knowledge about it, discovering some information sometimes not expected in advance.


Multivariate Time-Series forecasting using LSTM

What you could try to do is the following (pseudocode ahead):

input_past_24 = Input(shape=(24,5))
input_today = Input(shape(1,4))
output_from_LSTM = LSTM(some_args, ...)(input_past_24) # this has some size of your choice, let's say N
# if you want to add other layers, you'd do it here.
prediction = Dense(some_args, ...)(concatenate(input_today,output_from_LSTM))
model = Model(inputs=[input_past_24,input_today], output=prediction)
model.compile(some_args)
model.fit([data_from_past_days,data_today],expected_pollution, some_other_args...)

Above, the crucial point is to concatenate the output of the LSTM layer with the input of the day, before sending this to a final Dense layer for the final prediction.

For extra details, you may want to check Keras' Guide to the functional API.


replace values based on Number of duplicate rows are occured

I would do this manually. First, let us create the index set of entries whose state must remain active. To do this, I iterate over all rows and record active instances. Note that the later occurrence overrides earlier ones, so we keep only the last one occurrence of active event.
last_active = dict()
for i, row in df.iterrows():
    if row['Active'] == 'Active':
        last_active[row['site']] = i
keep_active = last_active.values()

Now I assign the state 'Active' to those entries whose index is in keep_active and InActive otherwise.
df['refined_active'] = df.apply(lambda x: 'Active' if x.name in keep_active else 'InActive', axis=1)


Customer Intelligence - How to measure success?

I'm assuming that the "propensity model" predicts a customer's likelihood to take some desired action.

If it's possible at your organization, then a good way to measure the model's effectiveness is an A/B test.  Select a set of leads to act as a control group using a baseline selection method.  Ideally, the baseline method would be whatever is currently in-place at your organization.  Then allow your model to select a set of leads as the experimental group.  Compare conversion rates (or whatever else you care about) between the control group and the experimental group.

If A/B testing isn't viable, then you can usually evaluate your model using the traditional techniques.  You can split your customer base into train/test sets.  Train your model on the training set, then made propensity predictions for the test set.  Compare the propensity estimates with actual conversion rates.


How can I see a long string in my dataframe?

At first, here is the Data.

data = {
    'Capacity' : [
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze",
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze",
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze",
        "qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze"
    ]
}
df = pd.DataFrame(data)
print(df['Capacity'])
# 0  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...
# 1  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...
# 2  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...
# 3  qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvg...

You can set pd.options.display.max_colwidth to increase the number of characters you want to display.

pd.options.display.max_colwidth = 20
print(df['Capacity'])
# 0    qsfjvanzicgbaefz...
# 1    qsfjvanzicgbaefz...
# 2    qsfjvanzicgbaefz...
# 3    qsfjvanzicgbaefz...

pd.options.display.max_colwidth = 100
print(df['Capacity'])
# 0    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...
# 1    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...
# 2    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...
# 3    qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzb...

# This seeks the maximum length of the current column 'Capacity'.
pd.options.display.max_colwidth = int(df['Capacity'].str.len().max()) + 1
print(df['Capacity'])

Or you can use the .values method instead, so that you are watching the 'raw' data.

#   Reset the option.
pd.options.display.max_colwidth = 50
print(df['Capacity'].values)
# ['qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze'
#  'qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze'
#  'qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze'
#  'qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze']

Or simply select one row from the column you're interested in.

print(df['Capacity'][2])
# qsfjvanzicgbaefzrkrsdvjcfhnefzraezsufchnezsuvgcfnezefzrdfchnzefzreicfjnefzrzekcrghzebefzrkcfjfzbefzrfkjhnvezkqcfhaze


In a random forest, are all decision trees given same priority?

As you mentioned, each decision tree is trained on p (sometimes sqrt(p) random features). This ensures that each tree is “grown” (trained) differently so that the model 1. Does not overfit the training data (reduces variance) and 2. Generalizes better to new data (reduces bias). Therefore we don’t weigh the trees differently, as this would be similar to having all tress trained on the same features.

You can change the voting threshold however from the standard 50% to anything you want (e.g 40%, 70%, 90%) which will change the precision and recall accuracies of the model.

EDIT: changing the voting threshold means changing the number of trees needed to make a classification. For example, in binary classification most standard random forests require 50% or more of the trees to vote for a class for that class to “win” (be predicted to be that class). But if you change this threshold, to say 70%, then 70% or more of the trees need to vote for the same class for that class to win.


tensorboard showing the epoch loss and accuracy for validation data but not training data

It is hard to know what is happening from just that screenshot and no code.

The training and validation plots are usually separated on the page, not lines on the same graph.

If you are using Tensorflow 2.0, there is a known issue, regarding the syncing of TB and the tfevent file (where logs are stored). A couple of things to try:

Try adding the TensorBoard callback with the argument: profile_batch=0
Try restarting the tensorboard a few times... reading can fail or be very slow to load

I am referring to the tensorflow.keras API:

tf.keras.callbacks.TensorBoard(
    log_dir='logs', update_freq='epoch',
    profile_batch=0,                      # <-- default value is 2
)


How would you deal with inf. or NA for rate or ratio as a feature variable

It depends on how would you define sales growth from a business point of view. For example if you trying to detect sales peaks with considering zero sales as normal,  you can fill zeroes, nans or infs with its most recent previous that has an ok number.


Some confusions on Model selection using cross-validation approach

I'm not exactly sure of cvpartition's routine, so I'll try to provide a more generalised answer.

"Whole set" v "full data set"

These are the same in this instance. Model hyperparameter tuning can be done by feeding the full / whole / complete data set into a cross validation process. Inside this cross validation, the data is split into k folds, where (k - 1) folds are used to build a model, and the remaining fold is used to test the model (since the remaining data is essentially unseen to the model). This is repeated k times; results can be averaged and standard deviation calculated.

Seen v unseen data (a.k.a. modelling and hold out samples)

This is essentially a variation on the cross validation approach above, except the split is done only once: the full data is split into two subsets -  Xtrain and Xtest in your case. In this approach, you would build a model using Xtrain only and test it on Xtest (there is no repetition).

So, what's the difference?

Both approaches attempt to do the same thing - somehow validate a model's performance and ability to generalise on unseen data. Cross validation is arguably the stronger technique, especially on smaller data sets.

Once you've successfully parameterised your model (using either approach), building your model on the whole data set is advisable since more data > less data.


How to improve accuracy further for forest cover prediction

Couple of suggestions:
Your training set is smaller than your test set? It should be the other way around.

You should also tune the RF hypermeters using the held out set or Cross Validation. The two parameter people tune on RF are ntrees and mtry.


Should you use the same algorithm in your feature selection as your model

Yes - You can use the same the same algorithm for both feature selection and prediction. The most common examples are L1 regression and tree-based algorithms. Those algorithms both find the most important features and predict targets using the same fitting mechanism.
The difference between those algorithms and your examples is that those algorithms do both steps concurrently and your examples are sequential.
There is no single best algorithm for feature selection that can also be used with other models for prediction. Different algorithms will perform better on different datasets.


The documentation for StackingEstimator is surprisingly poor, but it's relatively simple: fit the estimator on the data, and tack its predictions onto the dataset as a new feature.  Source, github issue.
So, your pipeline fits an ExtraTreesRegressor on the original inputs, and appends its predictions to the dataset going forward.  The data (original + 1st predictions) get scaled, then another ExtraTreesRegressor is fit (on scaled original and 1st preds, and with different hyperparameters), its predictions also getting tacked onto the dataset.  Finally, an AdaBoostRegressor is fit on scaled-original + scaled-1st-preds + 2nd-preds.


What to do when testing data has less features than training data?

Use the extra features for unsupervised learning. You might enjoy Vladimir Vapnik's take on this in the context of SVMs, which he calls privileged learning: Learning with Intelligent Teacher: Similarity Control and Knowledge Transfer


When to use (He or Glorot) normal initialization over uniform init? And what are its effects with Batch Normalization?

The normal vs uniform init seem to be rather unclear in fact.

If we refer solely on the Glorot's and He's initializations papers, they both use a similar theoritical analysis: they find a good variance for the distribution from which the initial parameters are drawn. This variance is adapted to the activation function used and is derived without explicitly considering the type of the distribution. As such, their theorical conclusions hold for any type of distribution of the determined variance. In fact, in the Glorot paper, a uniform distribution is used whereas in the He paper it is a gaussian one that is chosen. The only "explaination" given for this choice in the He paper is:

  Recent deep CNNs are mostly initialized by random weights drawn from Gaussian distributions

with a reference to AlexNet paper. It was indeed released a little later than Glorot's initialization but however there is no justificaion in it of the use of a normal distribution.

In fact, in a discussion on Keras issues tracker, they also seem to be a little confused and basically it could only be a matter of preference... (i.e. hypotetically Bengio would prefer uniform distribution whereas Hinton would prefer normal ones...) One the discussion, there is a small benchmark comparing Glorot initialization using a uniform and a gaussian distribution. In the end, it seems that the uniform wins but it is not really clear.

In the original ResNet paper, it only says they used a gaussian He init for all the layers, I was not able to find where it is written that they used a uniform He init for the first layer. (maybe you could share a reference to this?)

As for the use of gaussian init with Batch Normalization, well, with BN the optimization process is less sensitive to initialization thus it is just a convention I would say.


Probability for label correctness in semi-supervised learning

I do not know any papers, I would be greatly appreciated if someone would link some.

In my case, I always test by un-labelling my already known labelled Data by using a "traditional" 67% - 33% (train - test) split and checking how the labelling performs in various metrics (Accuracy, Logloss, etc.).

Moreover, there are various categories of Semi-Supervised Learning. For instance, if you would use Active Learning (pool based approach, includes incrementally sample/s each time), you could view how the performance ranges.

Finally, you may use as well Cross-Validation in the same sense for hyper parameter tuning for your Semi-Supervised Learning algorithm. All in all, in my perspective Semi-Supervised Learning, has a lot yet to offer and your question may be rather more domain, data specific and visualsing the Data and Clustering may give you new insights.


How can I fixed the filter and Kernel Size of a CNN?

Since the images are just 4x4 in size, you can do the following :

You can resize the image to a much larger dimension like 28x28 and then use sharpen or histogram equalization to bring out the contrast. Then use a 3x3x16, 3x3x 32 kernel arrays in 2 convolutional layers. The rest is fully connected.

The images can be input as 4x4 images but use a shorter neural net. One convolutional layer with padding and 3x3x 32 shape would do.

Use something similar to google nets, inception module, in which output from multiple convolutions are combined.


How is the min_rank used with the flights database

You can put the results of min_rank in to a new column, e.g.:

f2 <- flights %>%
  mutate(rank = min_rank(dep_delay))

Where showing just the selected rows:

f2 %>%
  select(dep_delay, rank)

You get this:

# A tibble: 336,776 x 2
   dep_delay   rank
       <dbl>  <int>
 1         2 208140
 2         4 219823
 3         2 208140
 4        -1 164763
 5        -6  48888
 6        -4  94410
 7        -5  69589
 8        -3 119029
 9        -3 119029
10        -2 143247
# ... with 336,766 more rows

Here is another example where you can see all the rows.

aa <- tibble(y = c(9, 8, 3, 4, 5, 7, 6),
             x = c(1, 2, 3, 4, 4, 4, 5))

aa %>%
  mutate(rank = min_rank(y))

Which gives us:

# A tibble: 7 x 3
      y     x  rank
  <dbl> <dbl> <int>
1     9     1     7
2     8     2     6
3     3     3     1
4     4     4     2
5     5     4     3
6     7     4     5
7     6     5     4


Multiple Categorical values for a single feature how to convert them to binary using python

I'm starting with the following dataset:

import pandas as pd
data = pd.DataFrame({'title': ['Avatar', 'Pirates', 'Spectre', 'Batman'],
                 'genres': ['Action|Adventure|Fantasy|Sci-Fi',
                            'Action|Adventure|Fantasy',
                            'Action|Adventure|Thriller',
                            'Action|Thriller']},
                columns=['title', 'genres'])

     title                           genres
0   Avatar  Action|Adventure|Fantasy|Sci-Fi
1  Pirates         Action|Adventure|Fantasy
2  Spectre        Action|Adventure|Thriller
3   Batman                  Action|Thriller

First, you want to have your data in a structure pairing titles with one genre at a time, multiple rows per title. You can get it in a series like this:

cleaned = data.set_index('title').genres.str.split('|', expand=True).stack()

title
Avatar   0       Action
         1    Adventure
         2      Fantasy
         3       Sci-Fi
Pirates  0       Action
         1    Adventure
         2      Fantasy
Spectre  0       Action
         1    Adventure
         2     Thriller
Batman   0       Action
         1     Thriller
dtype: object

(There's an extra index level that we don't want, but we'll get rid of it soon.) get_dummies will now work, but it only works on one row at a time, so we need to re-aggregate the titles:

pd.get_dummies(cleaned, prefix='g').groupby(level=0).sum()

         g_Action  g_Adventure  g_Fantasy  g_Sci-Fi  g_Thriller
title
Avatar        1.0          1.0        1.0       1.0         0.0
Batman        1.0          0.0        0.0       0.0         1.0
Pirates       1.0          1.0        1.0       0.0         0.0
Spectre       1.0          1.0        0.0       0.0         1.0


Is a neural network able to learn to map a completely different feature vector to the same class

Yes, neural networks excel at finding highly non-linear decision boundaries. For example, take a look at this demo (don't forget to click the play button to make the network learn). Even this very simple network is able to learn that the opposite corners of the input space belong to the same class.


Spatial pyramid pooling layers (https://arxiv.org/pdf/1406.4729.pdf) should solve this problem for you. These layers allow you to use input images of any dimension, instead of being restricted to 224x224 images, for example.


How to increase sales and revenue of a Client?

I think the question was asked to see how would you approach the problem. In similar questions, there is not a single answer, and the interviewer does not expect a certain answer instead expects a reasonable approach by you. It is like the famous interview question "How many golf balls can you fit into a swimming pool?". Such a question is asked to see the analytical thinking of the interviewee.
A structure of the answer may be like,

Churn analysis: understand the reason why customers "not coming" anymore, by doing so you will be able to reduce it.
Do a market analysis to understand the future state of the market to adjust the strategy and products according to the market.
Understanding customer segments (This is one of the most important implementations since you can come up with creative ideas after understanding the segments). Then, suggest the relevant products to relevant segments.
Understanding the competitors and their products. What are the differences between their strategy and ours, their products and ours?
Understanding the market size thus keep the predictions in a realistic range and to make solid steps.
Understanding costs and putting forward optimizations to reduce them.
etc.

These will make them sure that you have good knowledge over-analyzing the company, its strategies, products, etc. The above-mentioned solutions will state them you are not just able to increase the products and revenue, but also decrease the costs to maximize the profit. In this way, you can be a consultant to them.


How can I replace some words in my text file by keywords provided in another file using python?

Try i.lower().

It might be that document is not a string, but a function at the time in which you call document.replace(t, j)


Can PyLearn do everything that Theano can?

For your info, Pylearn2 is now deprecated. You can check it's alternatives i.e. platforms that are build on top of theano, which are: lasagne, blocks and keras.


How do I get latent features for a new row of data when doing non-negative matrix factorization?

Given a dataset represented by a data matrix $X$, we can solve $||X-HT||$ using gradient descent, setting coefficients equal to zero if they become negative. This gives us an $H$. If we get a new row of data $x_{new}$, then we can find the corresponding reduced features $w_{new}$(weights for the latent features) by solving the equation:

$argmin_{w_{new}} = ||x_{new}-w_{new}T||$.


Hive / Impala best practice code structuring

The biggest help to me was learning about how MapReduce works on a distributed environment. I don't have exact rules of thumb for you, but I might be able to help.

In Hadoop/Hive, the hive.execution.engine definitely seems to matter. Make sure understand the differences and try changing it for certain problems.
Subqueries can actually HELP performance, especially with really wide tables, MapReduce as your execution engine, and Hive. For example, if you only want 2 or 3 columns from a 2nd table, its better to:

SELECT T1.*, T2.col1, T2.col2
FROM T1
LEFT OUTER JOIN (SELECT key, col1, col2 FROM T2) T2
ON T1.key = T2.key;

This is all due to how the execution engine works under-the-covers, so learning about that and the differences of the execution engines will really help.

Temp tables are your friend - prior to hadoop, I avoided staging data in temporary tables a lot and wrote more complex SQL to avoid it. But with Hadoop and big data, I've actually found it much faster to create tables and then subsequently joining to them helps in some cases with massive data.

Learn how the table's underlying data is stored. STORED AS TEXTFILE is a lot different than data stored as parquet

Impala vs Hive also has performance implications. I've found to find Hive more stable, but slower.

How the data is stored (skewed or not, bucketed or not, etc) makes a huge different. Keep an eye out for queries where you have 100 reducers and 99 of them finish super fast and 1 of them takes forever.

Like I mentioned, spend time learning how the execution engines do the work in a distributed environment. Once you understand how the execution happens behind the scenes, you can start to pick up on other subtleties like why this:
select count(distinct user)
from table;

can be a low slower than this:
select count(*) from
(select user
 from table
 group by user) q;


Here is the code implemented by TensorFlow:

Implementation 1:

Implementation 2:


Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?

I thought about it this way: the training and test sets are both a sample of the unknown population. We assume that the training set is representative of the population we're studying. That is, whatever transformations we make to the training set are what we would make to the overall population. In addition, whatever subset of the training data we use, we assume that this subset represents the training set, which represents the population.

So in response to your first question, it's fine to use that shrinked/resmpled training as long as you feel it's still representative of that population. That's assuming your untouched training set captures the "real characteristics" in the first place :)

As for your second question, don't merge the training and testing set. The testing set is there to act as future unknown observations. If you build these into the model then you won't know if the model wrong or not, because you used up the data you were going to test it with.


How to encode positional information of a string in a NN effectively

If you just want to compare the letter and position, you could just use the strings as lists of characters and either compare up to the index of the shortest list or by sliding the shorter list over the longer list and grabbing the average number of matches.

Depending on your needs though I would recommend using more than just the positional data in the string. Two approaches I would probably try in addition to the above.

You could try converting the string to m length subsequences and then compare or aggregate in some way the number of matching or similar subsequences.
You could try converting the string to a regex and then compare the regex similarity.


this might take a few iterations, but perhaps it can be moved to chat if it gets too long.  are you wondering about a specific application or just the general concept?  as far as i know, the canonical form of something is the unique/simplest arrangement/representation of that something.  as for isomorphism (one to one relationship, a function gets you from one set to the other), you can oversimplify it and think about factors of numbers.

for a moment consider an analogy - instead of graphs, using sets of numbers.  this is randomly off the top of my head so hopefully there aren't too many holes in it.  we have

$A = 6, 10, 14$

and

$B = 18, 30, 42$

let's call $A$ and $B$ isomorphic because to get between any value in $A$ to the corresponding value in $B$, you multiply or divide by 3.  now let's introduce another isomorphic set:

$C = 3, 5, 7$

it looks like $A$, $B$, and $C$ are all isomorphic because there is a one to one relationship between any of their initial values.  the same can be said for any of their middle values, or any of their ending values.  while it's great that they're all isomorphic, we'll consider something additional about $C$.  i'm going to call $C$ canonical because if you remove the greatest common factor from $A$, or you remove the greatest common factor from $B$, you get $C$.  there is no common factor we can remove to reduce $C$ itself.

extending this, you can see that a canonical representation of a graph is an isomorphic version of that graph, but not necessarily the other way around.  all isomorphic versions of said graph will reduce to the same canonical form.

EDIT:

that being said, the definition of canonical is more complicated for graphs.  first, for graph isomorphism, this page about a code named bliss presents some small graphs as an example.  they define $G_{1}$ as

and $G_{2}$ as

and offer two transformations for $G_{1}$ that will make $G_{1}$ and $G_{2}$ isomorphic.  first, they consider $\{\langle 1\to 2, 2\to 4, 3\to 1, 4\to 3\rangle\}$, and then $\{\langle 1\to 2, 2\to 4, 3\to 3, 4\to 1\rangle\}$ - both of these transformations swap vertices.  the transformations result in the following isomorphic graphs:

each pair of connected vertices has a corresponding pair in $G_{2}$ ($2\to 1,3,4; 1\to 4; 3\to 4$).

now, to discuss canonical graphs, we must identify a cost function with which we can measure what is the 'most canonical' form.  this paper mentions metrics such as area, minimum angle, and total number of bends.  for simplicity, i'll try to minimize bends and favor angles that are closer to $90^{\circ}$.

if you use a coordinate transformation, you can see that this simple graph

and this graph

are isomorphic.  the same edges connect the same pairs of vertices, and if you move vertices from one around, you can make the other.  this graph

is also isomorphic with the first two, but since it has the optimal amount of bent edges (zero) and optimal angles (four right angles), it is also the canonical representation for these graphs.

for further reading, you might consider this paper, which discusses several algorithms used to find canonical representations for web mining, or this paper, which discusses canonical representations of binary decision diagrams, word-level decision diagrams, binary moment diagrams, taylor expansion diagrams, and finite field decision diagrams.

tl;dr:

just have a look at the minion with bananas on slide 10 of this presentation.


How to add 'other' as one group to clustering algorithm inference pipeline

Clustering doesn't work like this: for example k-means assigns an instance to the closest centroid, and since there is always a closest centroid there is a always a cluster that an instance "belongs to".
So you need a different approach if you plan to have the possibility of in instance "not in any group":

redo the clustering on the full set of instances
apply a first step which detects outliers
train a one-class classification model for every cluster


Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document embeddings: not all words equally represent the meaning of a particular sentence. And here different weighting strategies are applied, TF-IDF is one of them, and, according to some papers, is pretty successful. From this question from StackOverflow:

  In this work, tweets were modeled using three types of text representation. The first one is a bag-of-words model weighted by tf-idf (term frequency - inverse document frequency) (Section 2.1.1). The second represents a sentence by averaging the word embeddings of all words (in the sentence) and the third represents a sentence by averaging the weighted word embeddings of all words, the weight of a word is given by tf-idf (Section 2.1.2).


How to plot time series data from SQLite database

Since you are already using python this will be very easy for you :).
I presume the table name as it is not clear to me and a db file can have several.

You can use pandas to access the data using your connections.

import pandas as pd
def fetch_table_data_into_df(table_name, conn):
    return pd.read_sql_query("select * from " + table_name, conn)

This will give you a pandas Dataframe- they are very well suited for plotting.

I usually use pylab directly but I think this should also work:

df = fetch_table_data_into_df("NameOfTable", conn)
df.plot("READ_TIME", "VALUE")

Let me know if you need more information :).

Here is the entire code (using yours and mine) as well:

import sqlite3
from sqlite3 import Error
import pandas as pd
import pylab

def fetch_table_data_into_df(table_name, conn):
    return pd.read_sql_query("select * from " + table_name, conn)

def create_connection(db_file):
    """ create a database connection to the SQLite database
        specified by the db_file
    :param db_file: database file
    :return: Connection object or None
    """
    conn = None
    try:
        conn = sqlite3.connect(db_file)
    except Error as e:
        print(e)

    return conn

def main():
    database = r"E:\Data\SINCAL_Models\AusNetData\20180128_PQV\pq.sqlite"

    TABLE_NAME = "pq_data"
    # create a database connection
    conn = create_connection(database)
    with conn:
        print("Database connected:")
        df = fetch_table_data_into_df(TABLE_NAME, conn)
        # reading what you want to do, I presumed you want
        # to plot per measuremnt
        for measurement in df.MEASUREMENT.unique():
            df[df.MEASUREMENT == measurement].plot("READ_TIME", "VALUE")
            pylab.savefig(f"{measurement}.png")
            pylab.clf()

if __name__ == '__main__':
    main()


Where does spacy, nltk, prodigy, sklearn fit in the AI project?

I am not sure you can neatly fit the above tools and libraries into your schematic. What even is the meaning of "feature" here, is it an output (like "text classification" is an output of the algorithm)? In which case I challenge the usefulness of this schematic.

Nevertheless nltk, sklearn, etc. are libraries that contain multiple and diverse tools to help you do everything along the NLP AI modeling workflow. From feature extraction to modelling and validation.

As an example all three libraries of spacy, sklearn and nltk have the ability to construct models. So they certainly fit into "ML algorithm" but they also help in feature extraction so they fit in that bucket.

Beyond that they offer you general data wrangling capabilities as well as validation measures which do not appear in your schematic.

The main difference

NLTK and spacy are mainly focused on NLP and text-based data whereas sklearn is very much multipurpose.

The Alternatives

The alternatives really depend on your use cases, NLTK and spacy do very similiar stuff so are already alternatives to each other for NLP. Sklearn is almost standard when it comes to ML in python but if you want to build deep-learning you will probably work more with Keras, pytorch, etc.


It seems you have a data set for one component where the component suffered a fixed number of failure modes. You want to find out which data (let's assume continuous in time, so, what time) correspond to what failure mode. In other words, you are doing "pattern recognition" in your failure data.

Have you thought of using Self-Organizing Maps (SOM)? They are a sub-branch of artificial neural networks and have great capability in such problems.

You should also consider that not all failure modes appear in shape of a "peak" value. So, only looking at peaks is not a very smart way. It most probably will cover most of the failures, though, there will be moments that you miss. SOM could take care of this too.

Data pre-processing is done before the analysis. Be careful in normalization. You could miss the peaks or valley points easily if you don't pay enough attention in normalization. Don't just use any code or normalization method you find online. Test and check it with you data. For instance, some normalization could make all negative values positive which a negative value may have an important meaning in your work.

I assume you have another variable called "failure". Another approach I suggest is building a Neural Network (NN) model, which is very common. You have three input data that you mentioned, consider the "failure" variable as your target variable. Build the neural network and apply it to your data again. If the numbers of failures are little, the NN will be able to rebuild the normal behavior of your data (NN here is called a normal behavior model). When you apply it to your input data, the NN model will detect any deviation which is not expected.

MATLAB has a very good support for both of these approaches.


Images are two dimensional signals. The use of Conv1D is for one dimensional signals like voice and sound. CNNs are good for these signals too because of taking care of local input patterns. Definitely there are standard one dimensional filters which are most used in signal processing like high pass filters and low pass filters which are so much popular. In order to show you an example take a look at the following figure which illustrates the convolution of two signals:


I think most people would simply average a statistic like accuracy or kappa over the several runs, rather than compute one overall statistic.

While it's not necessarily true that an average (weighted by the test set size in each case) of these statistics equals the statistic computed over all of the runs at once, it happens to be true for accuracy and kappa. Hence it shouldn't matter.

I think the reason you may be getting a difference is that you're normalizing away the test set size. I don't think you should.


How to find farthest data points from a predefined cluster in a data set with Python?

You want to know the nearest neighbor of you unlabeled data in you labeled cluster.
Using sklearn, you can fit a NearestNeighbors() class with a giving metric, algorithm (Ball-tree, KD-tree,...) and all other parameters (see here).
Then get the labeled nearest neighbor from your unlabeled datapoint and its distance by using kneighbors() method.
Here is a sample code:
import numpy as np
from sklearn.neighbors import NearestNeighbors

# Fake data
labeled_samples = [[0, 1.2], [0, 1.3], [0, 1.4]]
unlabeled_samples = [[0, 1.7], [0.5, 0.5], [1, 1]]

# Create your class with your labeled cluster
neigh = NearestNeighbors(n_neighbors=1)
neigh.fit(labeled_samples)

# get the distance/index to the nearest neighbor of you unlabeled data
distances, indexes = neigh.kneighbors(unlabeled_samples, 1, return_distance=True)

Then you just have to sort the result.
Note: using this approach is more optimized than computing all distances from all labeled datapoints and then sort them. See this note for more info.


Creating a neural network for predicting next vote in a series of votes

Recurrent neural network may fit your needs.

Read about LSTM & GRU , which has been implemented into various NN libraires.

Here is the link to the keras documentayion of RNN :

https://keras.io/layers/recurrent/

Some interesting music project that takes advantage of RNN :

https://github.com/tensorflow/magenta

https://github.com/jisungk/deepjazz


Is text recognition by definition a part of image recognition?

Yes. Please take a look on how to do text recognition in images using opencv and Tesseract opensource libraries.


Is this a tried alternative to word embedding for NLP?

It has never been tried as far as I know. And the reason is that, IMHO, it would give you inconsistent results. I am talking in particular about unwanted arithmetic properties of such vector.
Do you want to represent an a, b, c vector of letters {1,2,3} knowing that
a < b < c  ?
This makes sens with ints, not with char/tokens.
The good news is you can still use 2d conv, once you assembled embedding vectors into larger phrase vectors.


Question about the simple example for batch normalization given in "deep learning" book

I think what the statement meant was when given weights $w_1,...,w_n$ are fixed, output is linear proportional to $x$, but as it mentions

  nonlinear function of the weights w_i

Given a set of weights (more than one being varied), they do not linearly add to produce an output like

$y=w_1x_1+...+w_nx_n$

but rather non-linearly like

$y=w_1w_2..w_n*x$ (each $w_i$ is a dimension in the hyperspace)

And I think it would become more clearer from the statement "Output is linear to any weight $w_i$ but non-linear to weights $w_i$".


Question about graphing the clusters in K means

StandardScaler subtracts the mean from each variable and then divides it by the standard deviation. It's a common preprocessing step, certainly for k-means because this algorithm heavily depends on the scaling of the data.

If I understand correctly you want to visualize the original data and make use of the labels from k-means by doing so. You could either add the labels to the original data (assuming the order of the records did not change):

original_with_label = numpy.concatenate(original, labels, axis = 1)

Or you could transform the data back to its original scale:

transformed_back_to_original = scalar_fit.inverse_transform(transformed_data)


weather and products sell relationship which one should be the target variable?

Apparently, you have a target variable (y) and a set of input variables (input variables (X). This seems like a supervised problem. A regression analysis can be a solution to implement.

Feature engineering needs to be completed before running a machine learning model. Such as text field columns can be encoded with Label Encoder and One Hot Encoder.

Irrelevant input columns for analysis can be dropped. If the city variable is same all the way, meaning the all of the data is from a single city. Also N/A and null values need to be cleaned.

In some models, normalization of the data can produce better results.

As a Machine Learning model, starting from Linear Regression would be feasible. Later Random Forest Regressor, Lasso, Ridge Regression can be examined.


Comparative Analysis of two sets of data

No, this is the classical application of statistical test. You should consider the problem in the scope of two-sample test. Machine learning does not fit in the picture.


GAN Loss Function Notation Clarification

Generally, the notation $\mathbb{E}_{x\sim p}f(x)$ or $\mathbb{E}_{x\sim p(x)}f(x)$ refers to the expectation of $f(x)$ with respect to the distribution $p$ for variable $x$ (e.g. see the explanation in the notation section in "Pattern Recognition and Machine Learning" by Bishop).

In the context of GANs, it means that for an objective function such as
$$\mathbb{E}_{x\sim p_{data}(x)}[logD(x)] + \mathbb{E}_{z\sim p_z(z)}[log(1 - D(G(z)))]$$
the first summand is the expectation with regards to $x$ coming from the data and the second summand is the expectation with regards to $z$ which you sample from $p_z$ as an input for G.

Since the first summand stands for correctly classified real data (coming from $p_{data}$) and the second summand stands for correctly as fake classified images coming from G using $z$ sampled from $p_z$, D tries to maximize this expression. And G does the opposite.


How to feed my JSON dataset in Keras for character level text classification

As mentioned by @Frankstr you want a tutorial on "character-level classification", not handwritten digit recognition as the one you have linked.

Character-level classification is typically done with an RNN or a 1D CNN. See this implementation of Character-level Convolutional Networks for Text Classification for example.

In order to convert your json data to x_train and y_train which can be fed to Keras, you can read it into a dictionary, extract keys and values separately, and binarize your class labels:

import json
from sklearn.preprocessing import LabelBinarizer

with open('data.json', 'r') as f:
    train = json.load(f)

y_train = list(train.values())

lb = LabelBinarizer()
x_train = lb.fit_transform(list(train.keys()))

Make sure your json is properly formed. The example you gave is missing commas after each value. It should be:

{"candidate00001": "graceful ones.",
 "candidate00002":"One more,Marvelous said, sounding royally bored from his seat.",
 "candidate00003":"She tired,Joe said, though not unkindly."
}


Why is my validation loss going up while my validation accuracy also goes up?

It is possible that accuracy and cross entropy increase at the same. For example, for a positive sample your predicted probability could go from 0.4 to 0.1 (still wrong but worse increasing the entropy loss) and for another positive sample your predicted probability could change from 0.49 to 0.51 (changes from wrong to right improving accuracy). The first case would increase your entropy loss while the second would improve accuracy without significantly changing the entropy loss.

It's a little difficult to say if cross entropy is not a good metric for your case without knowing any details. But most likely you would want to stick with it for a couple of reasons. For example, the cross entropy gives better probability estimates and has nice properties for training with gradient descent (smooth gradients, training doesn't stall for large values because the log off-sets the exp in the sigmoid activation etc.).


Confused about how to apply KMeans on my a dataset with features extracted

For clustering, your data must be indeed integers. Moreover, since k-means is using euclidean distance, having categorical column is not a good idea. Therefore you should also encode the column timeOfDay into three dummy variables. Lastly, don't forget to standardize your data. This might be not important in your case, but in general, you risk that the algorithm will be pulled into direction with largest values, which is not what you want.

So I downloaded your data, put into .csv and made a very simple example. You can see that I am using different dataframe for the clustering itself and then once I retrieve the cluster labels, I add them to the previous one.

Note that I omit the variable timestamp - since the value is unique for every record, it will only confuse the algorithm.

import pandas as pd
from scipy import stats
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv('C:/.../Dataset.csv',sep=';')

#Make a copy of DF
df_tr = df

#Transsform the timeOfDay to dummies
df_tr = pd.get_dummies(df_tr, columns=['timeOfDay'])

#Standardize
clmns = ['Wattage', 'Duration','timeOfDay_Afternoon', 'timeOfDay_Evening',
         'timeOfDay_Morning']
df_tr_std = stats.zscore(df_tr[clmns])

#Cluster the data
kmeans = KMeans(n_clusters=2, random_state=0).fit(df_tr_std)
labels = kmeans.labels_

#Glue back to originaal data
df_tr['clusters'] = labels

#Add the column into our list
clmns.extend(['clusters'])

#Lets analyze the clusters
print df_tr[clmns].groupby(['clusters']).mean()

This can tell us what are the differences between the clusters. It shows mean values of the attribute per each cluster. Looks like cluster 0 are evening people with high consumption, whilst 1 are morning people with small consumption.

clusters  Wattage     Duration   timeOfDay_Afternoon  timeOfDay_Evening timeOfDay_Morning
0         225.000000  85.000000             0.166667           0.833333  0.0
1         109.666667  30.166667             0.500000           0.000000  0.5

You asked for visualization as well. This is tricky, because everything above two dimensions is difficult to read. So i put on scatter plot Duration against Wattage and colored the dots based on cluster.

You can see that it looks quite reasonable, except the one blue dot there.

#Scatter plot of Wattage and Duration
sns.lmplot('Wattage', 'Duration',
           data=df_tr,
           fit_reg=False,
           hue="clusters",
           scatter_kws={"marker": "D",
                        "s": 100})
plt.title('Clusters Wattage vs Duration')
plt.xlabel('Wattage')
plt.ylabel('Duration')


How to combine sparse text features with user smile for sentiment classification?

If you are using a RNN and the 5000 features refers to size of your dictionary, I would recommend you consider an embedding of your words, eg word2vec. Each word would then result in a maybe 200-300 - sized vector, to which you can concatenate the single smile feature, and then run that through your RNN.

If that is not doing much, you can run the embedded text through an RNN, and then take the result of that (lets say 10-20 features per time-step) and concatenate the smile feature, and run those through a second RNN. The idea is the first one is a further compression over time, and the second is your 'final' RNN which works at a higher level of abstraction.

In general, this is a classic case of combining a high-dimensional feature set with a low dimensional one, to which there are many approaches, the question being, when do you join the features. Take a look at this paper, which covers a few ideas on multi-modal features aka when and how to merge them (diagram 3.2), though I think it is more instructive than useful since their features are more rich than what you have with the single additional feature.

Another thing to try is to soften the 'smile' feature if it is not so already, aka if it is a 0 or 1, make it a random (0, 0.1) and (0.9, 1) respectively.


deap is an evolutionary algorithm library. In an evolutionary algorithm you usually want to optimize a function. For this, you define individuals as a collection of genes (e.g. a string of numbers) that condense a possible solution, you create a population of such individuals, and define a fitness function to evaluate how good they are; then you apply evolutionary operators (e.g. mutation, pairing) to evolve the population, effectively searching the solution space.

The problem referred to as "max one problem" in the linked page can be reworded as:

  let's create a toy evolutionary algorithm where we want to evolve a population of individuals (where each individual is a list of N integer numbers) until one of them is exactly comprised of N ones (i.e. 1,1,....,1).

The "maximum satifiability problem" is not related to this.


Random Forests Feature Selection on Time Series Data

Some EDA might be needed to create new features for each time-series item. You might want to mine for patterns and have random forest reduce the overfitting. Exactly how mining is done depends on the nature of the problem, which might indicate for things like:

interesting time periods,
events that happen at a time,
time lag between different series,
dynamical systems,
latent variables,
scedasticity

Breiman's landmark paper on random forest gives some theoretical guarantees that random forest works well when individual classifiers are good and the correlation between these individuals are low. This can also be a heuristic to prune features.


One option is not to measure the performance of the hyperparameters on the loss function of the training data but measure performance of the hyperparameters on the elevation metric on the validation data. The end goal of the most machine learning systems is the ability to predict on unseen data. Focusing on "best solution" as measured by loss function on training will lead to overfitting / non-robust solutions.


Does it make sense to use train_test_split and cross-validation when using GridSearchCV to play with hyperparameters?

Yes it makes sense. Your "validation" set is typically called the "test" set (at least in my experience). CV is creating train/validation sets to pick the hyperparams.

The most complete process would be to then build a model one more time with the best hyperparams on the 90%, and evaluate on the 10%. This evaluation gives you a reliable estimate of the generalization error. It can be slightly lower than the error you got from the the CV process for the same parameters; the hyper-parameter fitting can itself 'overfit' a bit.

Some people skip this last step, if you don't care about estimating generalization error. Whatever it is, it's still the best you can do, according to your tuning process. The upside of skipping it is, I suppose, 10% more training data.