Are there any tree-based models that use a genetic algorithm to generate the trees?

Very new release:
genetic-tree

The main objective of the package is to allow creating decision trees that are better in some aspects than trees made by greedy algorithms.
The creation of trees is made by genetic algorithm. In order to achive as fast as possible evolution of trees the most time consuming components are wrtitten in Cython. Also there are implemented mechanisms for using old trees to create new ones without need to classify all observations from beggining (currently in developmnet). There is planned to allow multithreading evolution.
The created trees should have smaller sizes with comparable accuracy to the trees made by greedy algorithms.
Also worth to be checked:
PyGAD

PyGAD is an open-source Python library for building the genetic algorithm and optimizing machine learning algorithms.
References:
https://hal.inria.fr/hal-01405549/document
https://www.hindawi.com/journals/tswj/2014/468324/
https://www.kdnuggets.com/2018/07/genetic-algorithm-implementation-python.html


Not a typical problem formulation indeed. I suggest two approaches:

Use a simple bayesian model where the states or nodes are each of your features. Connect the nodes directly when they haven a direct time dependency (for example vote1 -> vote2 -> vote3, but not vote3 -> vote1) and calculate the probability of a final (output) node to be something (for example user type 1, user type 2, etc). You could also use a hidden markov model that naturally models transitions between states. In your case it would output just 1 state (the prediction). I am not very fan of this model for your problem, but it could be nice to try. You will need a lot of data though.
You could use Fuzzy Inductive Reasoning (FIR). It is definitely not a simple algorithm, but works quite well for classification having time series or time-related features. In FIR you want to find an optimal mask of a given depth.
If you have, say, 5 features per sample, and for instance you want to know which features to select you will learn an optimal mask for your data that address this (and creates fuzzy rules, etc...). Also if you want to relate two or more examples you will define a depth higher than 1 for the mask. This will allow you to find a mask that for instance uses features 1,2 and 4 of sample t, features 2,3 ,4 and 5 of sample t+1 and features 1 and 4 of sample t+2. Moving t from 1 to N number of samples. So, interesting combinations of features from samples that have a time-relation (or some sort of sequence) are created.


What is the definition of imbalanced data set

The problem of an imbalanced dataset is the problem of generative classifiers that use the prior probability for calculating the predicted label. As the labels have a lower prior they get a lower probability.
There are several ways to cope with imbalanced datasets:

Oversampling the minority classes ,randomly add observations from the minority classes, so the prior probability of every class will be the same.
Undersampling - if you have a dataset with a lot of observations but the majority class is few times larger than the minority, choose randomly a subset of the whole dataset that includes the same number of observations for each label.
Use Data Augmentation to generate synthetic data that tries to simulate the same distribution of the features in a label.
Weighted classifiers - there are classifiers support weights for the labels.
If you use Neural Network model, you can do Transfer Learning. Copy the weights of the model from a model with balanced data (you told that you have models with a similar feature vector), copy the network (with the weights) and replace the last layer to randomly initialised (better to use Xavier initialiser) . Then freeze all of the layer weights except for the last one and train it. It's better to keep the same proportion between the classes using (1) or (2) and it's also recommended combine with (3).


Question on decision tree in the book Programming Collective Intelligence

There are four features:

referer,
location,
FAQ,
pages.

In your case, you're trying to classify an instance where FAQ and pages are unknown: mdclassify(['google','France',None,None], tree).

Since the first known attribute is google, in your decision tree you're only interested in the edge that comes out of google node on the right-hand side.

There are five instances: three labeled Premium, one labeled Basic and one labeled None.

Instances with labels Basic and None split on the FAQ attribute. There's two of them, so the weight for both of them is 0.5.

Now, we split on the pages attribute. There are 3 instances with pages value larger than 20, and two with pages value no larger than 20.

Here's the trick: we already know that the weights for two of these were altered from 1 to 0.5 each. So, now we have three instances weighted 1 each, and 2 instances weighted 0.5 each. So the total value is 4.

Now, we can count the weights for pages attribute:

pages_larger_than_20 = 3/4
pages_not_larger_than_20 = 1/4 # the 1 is: 0.5 + 0.5

All weights are ascribed. Now we can multiply the weights by the "frequencies" of instances (remembering that for Basic and None the "frequency" is now 0.5):

Premium: 3 * 3/4 = 2.25 # because there are three Premium instances, each weighting 0.75;
Basic: 0.5 * 1/4 = 0.125 # because Basic is now 0.5, and the split on
pages_not_larger_than_20 is 1/4
None: 0.5 * 1/4 = 0.125 # analogously

That's at least where the numbers come from. I share your doubts about the maximum value of this metric, and whether it should sum to 1, but now that you know where these numbers come from you can think how to normalize them.


How to include both origin and destination in your features?

Since your goal is to predict the price, I think it would be more useful to include features such as:

distance between origin and destination
whether origin and destination are in the same state/country/continent/area...

However the actual origin and destination might still be useful, at least the frequent ones, so it's worth experimenting. Those which appear only once or twice in your training data are unlikely to help the model, on the contrary they might cause overfitting. So you could filter out the origin/destination cities which appear less then $N$ times and keep the other ones. It's very likely that this will reduce the number of possible values a lot.


CNN Architecture for Multiple Instance Learning

As I understand there is some information that needs to be present in order for the example to "pass".
I have a few questions as well as a suggestion.
First off do the information come in different forms? Meaning, different fonts, different kind of documents for the same information etc?
Is the cnn mandatory? Any more info, regarding the domain of the application or the nature of the problem that you could share?

Suggestions:

Since you have done the OCR I would suggest a pipeline of different models. Process the outputs of the OCR so they can create a secondary dataset in order to employ NLP. Meaning decode the OCRs to produce plaintext outputs, which would serve as the secondary data-set.
Another thing you could try is to identify the category of each document just like any other multi-class classification. After that you could, employ another classifier based on which categories pass your test (if there 's a discrete number) and continue like so.
Example: Identify Type A through Type F documents. Set the combinations that would serve as true labels, e.g occurrence of Type F, occurrence of Type A,B,C (any other combination would be False) and then train.

Finally there's a paper on a different domain, though an approach similar to theirs could be followed. They present a Bag-of-Visual Words (BOV) method for object-based classification in land-use/cover mapping.

  ...we can characterize an image by a histogram of visual-word count [6]. The visual vocabulary provides a “midlevel” representation which helps to bridge the huge semantic gap between the low-level features extracted from an image and the high-level concepts to be categorized

Object Classification of Aerial Images With Bag-of-Visual Words


How to combine nlp and numeric data for a linear regression problem

It sounds like you could use FeatureUnion for this. Here's an example:

from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest

iris = load_iris()

X, y = iris.data, iris.target

# This dataset is way too high-dimensional. Better do PCA:
pca = PCA(n_components=2)

# Maybe some original features where good, too?
selection = SelectKBest(k=1)

# Build estimator from PCA and Univariate selection:

combined_features = FeatureUnion([("pca", pca), ("univ_select", selection)])

# Use combined features to transform dataset:
X_features = combined_features.fit(X, y).transform(X)
print("Combined space has", X_features.shape[1], "features")

svm = SVC(kernel="linear")

# Do grid search over k, n_components and C:

pipeline = Pipeline([("features", combined_features), ("svm", svm)])

param_grid = dict(features__pca__n_components=[1, 2, 3],
                  features__univ_select__k=[1, 2],
                  svm__C=[0.1, 1, 10])

grid_search = GridSearchCV(pipeline, param_grid=param_grid, cv=5, verbose=10)
grid_search.fit(X, y)
print(grid_search.best_estimator_)

Hopefully it is clear from that example how you could use this to merge your TfidfVectorizer results with your original features.


How to visualize multi-instance multiclass classification?

You could use a 3d-scatter plot, where each class would correspond to one axis and the color-intensity of the point would indicate the score value(e.g. for a grayscale colormap, the whiter the closer to 1 the value of the score).

Using the above format:

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame([[0.529, 5.0, 7.0, 4.0], [0, 3.0, 4.0, 2.0], [0.774, 10.0, 7.0, 6.0], [0.774, 10.0, 8.0, 5.0], [1, 3.0, 0.0, 2.0]])
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(df[1].values, df[2].values, df[3].values, c=df[0].values, cmap='gray', s=50, vmin=0.,vmax=1)

ax.set_xlabel('X Label')
ax.set_ylabel('Y Label')
ax.set_zlabel('Z Label')

plt.show()

You can change the scatter plot attributes for marker-size, colormap etc. according to the documentation to match your tastes.


What is the difference between data analysis and machine learning?

For me, data analysis represents something like....

What is an insightful KPI to get out of our data? If we are an
eComm retailer, for example, that might be what % of page visits did
we have to pages which we still had in stock?    or
What percentage of orders are dispatched within 2 days?

Machine learning, is a whole different game. You might re-state Q1 and Q2, for example:

Can we build a predictive algorithm that predicts what pages
will be out of stock?
Can we build an optimizer which helps us to dispatch orders faster?

In some deeper level, data analysis you are typically using the machine to automate some fairly trivial tasks. With machine learning, you are letting the machine automate a lot of the learning process - where you would be typically passing in that knowledge as a data analyst.

Does that help clarify at all?


Choosing attributes for k-means clustering

First of all, if you know that certain attributes shouldn't after the clusters, you should remove them altogether. There is no point in hoping that K-Means will figure it out on its own if that can be fixed upstream.
Second, obviously, not every attribute should affect the clusters equally. K-Means is based on the concept of distances between your points. Based on the distance matrix, the algorithm will find different clusters. The good thing is that you can tweak how the distance is calculated. You could weigh the different attributes such that differences between certain attributes are more important than others.
Third, if you want to programmatically find the "best" attributes for clustering, I don't know of any efficient ways to do it. Meaning that your best bet is to try different combinations of attributes and see how good the clustering becomes. To rate the quality of clustering, there exist metrics like the Dunn Index, or the Davies-Bouldin Index (see this link for more detailed information: https://medium.com/@ODSC/assessment-metrics-for-clustering-algorithms-4a902e00d92d)


Incorporate luck in statistical modelling

Use a probabilistic model. For example, if the probability of one team beating the other is $σ(w⋅(team1−team2))$, where $σ(⋅)$ is the logistic function, then there's always a probability that the weaker team will win.

If you wanted to use a neural network you'd use it to replace the logistic function.

Answer copied from comments.


Changing the shape of the input layer in tensorflow

Actually, if you change the input the input size, nothing goes wrong with the convolutional layers but the outputs of these layers increases and that will cause to the increased number of inputs to the dense layers 1. Consequently, you will have to have extra weights and you have to train them. That is why it is better to have a fixed size input. But there are other solutions that can be expanded to your task. Take a look at the contents of the third week of this course. As you will see, you have to input patches of the image to the network but what is done there is for detection tasks, you have to expand it to your task, but maybe resizing all inputs to a predefined size is simpler.


Keras: visualizing the output of an intermediate layer

Consider this network

model = Sequential()

model.add(Convolution2D(32, 3, 3, input_shape=(1,28,28)))
convout1 = Activation('relu')
model.add(convout1)
convout2 = MaxPooling2D()
model.add(convout2)

model.add(Flatten())

model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.2))
model.add(Dense(10))
model.add(Activation('softmax'))

Now you can visualize the activation using this function.

def layer_to_visualize(layer):
    inputs = [K.learning_phase()] + model.inputs

    _convout1_f = K.function(inputs, [layer.output])
    def convout1_f(X):
        # The [0] is to disable the training phase flag
        return _convout1_f([0] + [X])

    convolutions = convout1_f(img_to_visualize)
    convolutions = np.squeeze(convolutions)

    print ('Shape of conv:', convolutions.shape)

    n = convolutions.shape[0]
    n = int(np.ceil(np.sqrt(n)))

    # Visualization of each filter of the layer
    fig = plt.figure(figsize=(12,8))
    for i in range(len(convolutions)):
        ax = fig.add_subplot(n,n,i+1)
        ax.imshow(convolutions[i], cmap='gray')

# Specify the layer to want to visualize
layer_to_visualize(convout1)

If you want you can refer my jupyter notebook.


Ngram based Langauge Models learned using an Encoder-Decoder Model

For your case, I feel that skip-gram model will be fit for your business problem. Where u can predict a sequence of a word rather than one word. I recommend you to train the skiagram model on your corpus.


Not enough memory for operations with Pandas

When Pandas finds it's maximum RAM limit it will freeze and kill the process, so there is no performance degradation, just a SIGKILL signal that stops the process completely. Speed of processing has more to do with the CPU and RAM speed i.e. DDR3 vs DDR4, latency, SSD vd HDD among other things.
Pandas has a strict memory limit but there are options other than just increasing RAM if you need to process large datasets.
1.- Dask
Here are certain limitations in dask.

Dask cannot parallelize within individual tasks.
As a distributed-computing framework, dask enables remote execution of arbitrary code. So dask workers should be hosted within trusted network only.

A Dask tutorial:
https://medium.com/swlh/parallel-processing-in-python-using-dask-a9a01739902a
2.- Jax
3.- Feather Format
Language agnostic so it's usable in R and Python, can reduce the memory footprint of storage in general.
4.- Decreasing memory consumption natively in Pandas
Reducing the number of bits of memory to encode a column help specially when you use tree-based algorithms to process the data later. This is a script popularized by Kaggle.
import pandas as pd

def reduce_mem_usage(df, verbose=True):
   numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
   start_mem = df.memory_usage().sum() / 1024**2
   for col in df.columns:
       col_type = df[col].dtypes
       if col_type in numerics:
           c_min = df[col].min()
           c_max = df[col].max()
           if str(col_type)[:3] == 'int':
               if c_min > np.iinfo(np.int8).min and c_max <    np.iinfo(np.int8).max:
                   df[col] = df[col].astype(np.int8)
               elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:
                   df[col] = df[col].astype(np.int16)
               elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:
                   df[col] = df[col].astype(np.int32)
               elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:
                   df[col] = df[col].astype(np.int64)
           else:
               if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:
                   df[col] = df[col].astype(np.float16)
               elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:
                   df[col] = df[col].astype(np.float32)
               else:
                   df[col] = df[col].astype(np.float64)

   end_mem = df.memory_usage().sum() / 1024**2
   print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))
   print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))

   return df


calculate distance between each data point of a cluster to their respective cluster centroids

def k_mean_distance(data, cx, cy, i_centroid, cluster_labels):
        distances = [np.sqrt((x-cx)**2+(y-cy)**2) for (x, y) in data[cluster_labels == i_centroid]]
        return distances
clusters=km.fit_predict(data2D)
centroids = km.cluster_centers_

distances = []
for i, (cx, cy) in enumerate(centroids):
    mean_distance = k_mean_distance(data2D, cx, cy, i, clusters)
    distances.append(mean_distance)

print(distances)

Using this function I have solved my problem


Data Snooping, Information Leakage When Performing Feature Normalization

The normalization parameters you fitted in training are now part of your model. You fitted the model weights on the training data: the normalization step is part of your model now, and the "parameters" for that step are the mean and variance of the training data, not the test data.

To help make this more concrete, let's pretend that this model is in production. You fit your model on whatever training data you had available, and now it's running as a scoring service. You can evaluate your model accuracy over time as data you've scored gets labeled to ground truth, but you're just evaluating the model not refitting it. How do you normalize these incoming predictions? You're going to need to use the mean and variance from whatever dataset you used to fit the model. If you'd rather do something like using the last K observations to estimate mean/variance, then that's the procedure you need to be testing when you fit the model.


Understanding the concept vanishing gradient and exploding gradient problem in terms of training data

I always refer "Andrew NG" documents if available for any ML understanding. I believe that Youtube link Vanishing and Exploding Gradient, will help you in understanding the concept in a better way.

Still a brief from my side:

  DNN of K layers without an activation function, will be like
  multiplying K coefficients(weights) together. Somehow if you received
  initial coefficient value < 0, in such a case during back
  propagation till you reach to the starting layers (from Input layer
  side), it may possible you completely lost the value of gradient,
  as gradually you are multiplying smaller value i.e. vanishing a value. Similarly if initial
  coefficient value > 0, multiplying together will form a very big
  number i.e. exploding a value.


Emotion Recognition with Multi-task Learning

I suppose your question is .... "How to do multi-task learning?"

Should you take a neural network approach, for instance:
A simple example with the above diagram is to pass Columns 1-8 into the "Shared Layers".
The task specific layers would each be related to your three targets
ie. Task 1 - Arousal,
Task 2 - Dominance &,
Task 3 - Valance.
Do have a look at this review article on multi-task learning and its various types. A more accessible article with some code examples here.


is it bad to have many different measurements for the same target variable?

Note that you are solving two different problems here.

In the first problem, you want to predict the target variable given one noisy measurement.

In the second problem, you want to predict the target variable given some statistics from a group of noisy measurements.

Your results show that the second problem is easier to solve which is intuitive, since the amount of noise (variance) for average of multiple measurements is less than only one measurement (closely related to Law of Large Numbers), thus the relation in the second problem is easier to find by the model.

Therefore, if both problems are equivalent to you, go with the second problem which is easier to solve.


What can be the best dropout value and the FC layer for good accurate predictions?

In the area of Neural Networks there isn't one right value that works for all kinds of situations. You just need to repeat experiments, till you reach a satisfactory value. Ideally Dropout values range from 0 to 0.5 . I would use a dropout value of .3 or .2 in my neural networks. Also, the number of neurons in FC layer, impact the degree of fit in your network. Too small numbers, result in underfit. Too large values, result in overfit. Keep on increasing the complexity of the model and evaluate  it against a validation loss, that will inform you about increasing or decreasing the number of FC layers.


Can anybody explain Apache Kafka for me in a plain language?

As the official documentation of Kafka says:

  Apache Kafka is publish-subscribe messaging rethought as a distributed
  commit log. It follows the publish-subscribe messaging style, with speed and durability built in.

Now, let me give a dumbed down explanation of the same:

A publish-subscribe(commonly called pub-sub) architecture basically contains a publisher(who sends the messages, for example: A twitter stream) and a subscriber(who receives the messages Ex: the analyst/the user's activity stream)

A clearer explanation would be:

  In software architecture, publish–subscribe is a messaging pattern
  where senders of messages, called publishers, do not program the
  messages to be sent directly to specific receivers, called
  subscribers, but instead characterize published messages into classes
  without knowledge of which subscribers, if any, there may be.
  Similarly, subscribers express interest in one or more classes and
  only receive messages that are of interest, without knowledge of which
  publishers, if any, there are.

And keep in mind that Kafka is a distributed pub-sub messaging system, designed to scale.

Now,

  At which stage of a BigData analysis is it used?

It is basically used in the Extraction step of the ETL pipeline. It can contain high volumes and speed of data flow (we call it high throughput in technical terms).

It can store messages also, but it is a persistent store; which means that the data is not stored forever. The stored data has an expiry date.

  And what are prerequisites for learning it?

This is a tough question, as everyone who can program can get started with it, but if you really want to implement this in your architecture, then these are the pre-requisites which I can think of:

Know what is a persistent data store and where to use them
Know what a pub-sub messaging system is
Know what an extraction step is, in ETL. And also know how to deal with data with huge volume coming at you at high velocity.

Further reading:

Kafka has an excellent documentation, which can help to get started with it.


how to handle values that only appear once in a column?

Assuming those are target categories, the most common strategies are:

Drop them. Do not try to predict categories with single observations because the model will overfit.
Bundle them to together into a "dustbin" category. Then the model can learn to predict "dustbin" category membership.


Cut-Off using Frequent Pattern Mining - Spark Mllib

Like @SeanOwen pointed out its called support.

spark.mllib’s FP-growth implementation takes it as a hyper-parameter under minSupport.

It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.

Usage:

import org.apache.spark.mllib.fpm.FPGrowth

val transactions: RDD[Array[String]] = ???

val fpg = new FPGrowth().setMinSupport(0.2)

val model = fpg.run(transactions)

I hope this helps.


What type of neural network should I use to detect meteors in images?

This is not a NN solution but can you adjust the parameters to opencv's HoughLine? There should be a way to find just the straight lines. There should be a way to adjust the minimum line length value to be considered a meteors. Meteors will always appear straight. So only keep the results when the points have similar slopes and are near each other?

See this example:

import cv2
import numpy as np

img = cv2.imread('dave.jpg')
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
edges = cv2.Canny(gray,50,150,apertureSize = 3)
minLineLength = 100
maxLineGap = 10
lines = cv2.HoughLinesP(edges,1,np.pi/180,100,minLineLength,maxLineGap)
for x1,y1,x2,y2 in lines[0]:
    cv2.line(img,(x1,y1),(x2,y2),(0,255,0),2)

cv2.imwrite('houghlines5.jpg',img)

Source


How does an individual feature affect predictions in neural network classification problem?

Is there a way to quantify the likelihood in ANN similar to logistic regression wherein regression coefficients give the change in the log odds of the outcome for a one unit increase in the predictor variable?

Good question.

Yes, there is a way. The approach that can help you is called partial dependence plot (PDP), see the links below for further details and examples.

The approach is model agnostic, i.e. works well for any predictive model, powerful yet simple.

The main steps for one-dimensional partial dependence plot are as follows

Fit your model as usual
Select the predictor of interest and a set of values to be investigated (e.g. income as in the article you refer to and values of say 50k, 70k, 80k, ..., 120k)
For all observations in your dataset replace the values of your predictor with the first value from the set above (50k).
Calculate the model output for the modified dataset from the previous step and calculate the average over all observations.
Repeat steps 3-4 for the remaining values (70k, 80k, ...) and plot the values of your predictor along X axis and the corresponding averaged model predictions along Y axis.

With one-dimensional PDP described above you can easily see the marginal impact of a predictor being analysed on the model output. Furthermore, one can use similar technique to perform multi-dimensional analysis, e.g. to investigate the impact of interactions.

partial dependence plots- scikit-learn documentation

partial dependence plot - tutorial by Dans Becker on Kaggle


Building Timeseries models for stock trading having multiple stocks

This is a question about Univariate versus Multivariate Time Series analysis. Both can be valid approaches.
However, for stock price forecasting I would suggest that Multivariate will probably be better, with the important caveat that your choice of time series should have some fundamental justification. Randomly choosing to analyse stock A and B together from a pool of 'N' thousands of potential stock pairs/triplets etc. will not be as effective as choosing stock X and Y if you believe X and Y are related in some way, i.e. they are in the same industry sector.


How Hyper-linked library vs traditional library differs from each other as ML problem?

To get discussion started, here are some of my intuitions.

Instead of hierarchy, it seems that the hyper-linked problem is more about optimization of information entropy in a tree structure: giving all labels to all documents would be a bad solution and also giving shortest path possible is far from optimal (due to being too vague). It seems that the multi-class classification problem of hyper-linked libraries would benefit from some cost factor in relation to how many labels a document is allowed to have and how (non-)generic they should be.

Scale-free networks might provide such cost factor as they can normalize network through-put even when the amount of documents grows unboundedly, e.g. the concept space can be frozen with a guarantee of keeping the navigation depth as constant, but still being able to distribute the documents "evenly" by preferential attachment based on some optimization goal that would give privilege for some documents over others (a scale-free network, where new links are also made between existing nodes).

This is actually the very same problem Google solved with PageRank. Internet is a scale-free network by topology and search engine provides us a symbolic link to the original based on keyword mapping.

However, while Google is entirely free from ontologies, hyper-linked libraries need to provide some baseline structure from existing ontologies. Also, if we would consider the reality of News Agencies classifying their own content, the life-cycle of some content is not as "eternal" as with content stored by libraries. Discovery of new media products would be important part of the problem. Again, seems like Google has solved these kind of problems.


could not broadcast input array from shape (2,3) into shape (3) while using timestamp to build neural network in python

First of all, you should use separate variables for the 3D and 2D x_trains:

x_train_n = np.zeros((x_train.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")
x_test_n = np.zeros((x_test.shape[0] - num_time_steps + 1, num_time_steps, num_features), dtype="float32")

And, your outer loop should span the new x_train, i.e. x_train_n:

for i in range(len(x_train_n)):
   for timestep in range(num_time_steps):
       x_train_n[i][timestep] = x_train[i + timestep].squeeze()

Change the other loop (the one for the test case) as well.


Try an LSTM Encoder-decoder model. The encoder takes Column_B as input and Decoder gives Column X, Column_A as output. Or create a two machine learning models one for each column prediction.


Predicting Composition of Chemical Compounds

Try a one-hot encoding of the elements in your training set (H, C, I etc.) and the same for the chemical descriptors ('acid', 'oxalic' etc.). Then try feeding the data to a simple feed-forward neural network, mapping the one-hot encodings of the descriptors as your x_train and x_val to the chemical elements, which are your y_train and y_val.

For the encodings, take a look at the Scikit learn encoders.

I think such a simple approach may work, as the chemical names are logically named, so any experienced chemist knows exactly which elements to expect in a compound, given its name. Your problem doesn't require further information in the predictions, e.g. the actual chemical makeup, like $CH_3CH_2OH$ (ethanol).

Regarding a small amount of data: look into something like K-fold cross validation. Using this, you select some portion of the data to be your validation data, and train the model. You then repeat this process, selection a different portion of the data. This will help make the most of a limited dataset, although it may introduce overfitting because your model will eventually have seen all the data!

Here is a schematic of this method:

Scikit Learn has a class that implements it for you.


Spark, optimally splitting a single RDD into two

First of all let me tell you that I'm not a Spark expert; I've been using it quite a lot in the last few months, and I believe I now understand it, but I may be wrong.

So, answering your questions:

a.) they are equivalent, but not in the way you're seeing it; Spark will not optimize the graph if you are wondering, but the customMapper will still be executed twice in both cases; this is due to the fact that for spark, rdd1 and rdd2 are two completely different RDDs, and it will build the transformation graph bottom-up starting from the leafs; so Option 1 will translate to:

rdd0 = rddIn.map(lambda x: customMapper(x)).filter(lambda x: x[0]==0).cache()
rdd1 = rddIn.map(lambda x: customMapper(x)).filter(lambda x: x[0]==1).cache()

As you said, customMapper is executed twice (moreover, also rddIn will be read twice, which means that if it comes from a database, it may be even slower).

b.) there is a way, you just have to move cache() in the right place:

mappedRdd = rddIn.map(lambda x: customMapper(x)).cache()
rdd0 = mappedRdd.filter(lambda x: x[0]==0)
rdd1 = mappedRdd.filter(lambda x: x[0]==1)

By doing this, we are telling spark that it can store the partial results of mappedRdd; it will then use these partial results both for rdd1 and rdd2. From the spark point of view this is equivalent to:

mappedRdd = rddIn.map(lambda x: customMapper(x)).saveAsObjectFile('..')
# forget about everything
rdd0 = sc.objectFile('..').filter(lambda x: x[0]==0)
rdd1 = sc.objectFile('..').filter(lambda x: x[0]==1)


How do you measure performance for word prediction tasks?

Your accuracy should be this: Did the phrase existed before? It means either storing all combinations of 4-grams or searching the data on the fly.

But then the prediction algorithm that uses any combination from the previous data will be 100% accurate. I do not really understand the reason for such accuracy estimation.


Efficient dimensionality reduction for large dataset

Have you heard of Uniform Manifold Approximation and Projection (UMAP)?

  UMAP (Uniform Manifold Approximation and Projection) is a novel
  manifold learning technique for non-linear dimension reduction. UMAP is
  constructed from a theoretical framework based in Riemannian geometry
  and algebraic topology. The result is a practical scalable algorithm
  that applies to real world data. The UMAP algorithm is competitive
  with t-SNE for visualization quality, and arguably preserves more of
  the global structure with superior run time performance. Furthermore,
  UMAP as described has no computational restrictions on embedding
  dimension, making it viable as a general purpose dimension reduction
  technique for machine learning.

Check their code and original paper for list of pros and cons, it is super easy to use.

Quick Facts: UMAP can handle large datasets and is faster than  t-SNE and also supports fitting to sparse matrix data, and contrary to t-SNE, a general purpose dimension reduction technique, meaning that not only it can be used for visualisation but also for reducing the feature space for feeding into other machine learning models.

Concrete Examples: I have benchmarked the method and compared it against some other dimensionality reduction techniques benchmark notebook, if interested to have a quick look and a jump start.


Reinforcement Learning Vs Transfer Learning?

I didn't watch this lecture, but, the way I see it, reinforcement learning and transfer learning are absolutely different things. Transfer learning is about fine-tuning a model, which was trained on one data and then striving to work with another data and another task. For example if you use weights of pretrained model on imagenet and then implement it to your dataset, while your dataset consists of small amount of different species of birds images (which might be not sufficient to train for example unet from a scrath). Reinforcement learning is about how some agent should response to environment condition to receive high reward. There is an illustrative example with a drone making a delivery, when there is some range of restrictions of the environment. There are two links, which might be useful:
https://machinelearningmastery.com/transfer-learning-for-deep-learning/
https://skymind.com/wiki/deep-reinforcement-learning

I guess, I can't answer, which approach is better, because they aim to solve different challenges.


How to handle potential interactions when one-hot encoding?

You could combine these features before using one-hot encoding, and see if the performance is improved. But keep in mind, that it really depends on the problem each time.

Generally, is a good thought to combine these type of features. CatBoost, a very good gradient boosting library, create such combinations and the results are pretty good most of the time. I would give it a go if I were you.


How to extract contents by topic from a document?

pyresparser is useful for extracting information from resumes. I believe this should work in your case.
Check out the more details on the same here https://pypi.org/project/pyresparser/
Let me know if it works!


How to use machine learning to extract product info from the titles of eBay listings

Like Toros91 explained, it is a raw data format. You want to convert 5 PCS into There are 5 pieces. How does you know PCS means pieces? You have a link somewhere in your brain that tells this. You have to do the same in programming. Create a dictionary which tells the machine that wherever it encounters PCS in line, PCS means pieces (it can be a model name also, you have to take care of that). The number just before PCS is the value. All this you have to code manually. There is no shortcut here. And of course you have to use NLP to achieve your task.

Pre-Processing your raw data is the hard task, using ML algorithm on a formatted data is easy one.


Time series forecasting: prediction and forecast far from the reality

I’m by no means an expert in ARIMA models but I have experimented a bit with them so I can give you a list of some things that I found to improve my models.

I did a grid search of different ARIMA model parameters to find the best model
I also tried some methods to ensemble ARIMA models. For example, averaging the best three ARIMA models gave a better overall score than just one.
Depending on the problem you can play with what level the data is aggregated. I was interested in predicting weekly sales and so I made a model to predict daily sales and then aggregated the predictions to weekly sales. This improved the accuracy dramatically.

Looking at your autocorrelation there appear to be regular weekly patterns. Trying a SARIMA model with a 7 day cycle may result in better scores, I.e. there is a 7 day seasonality.

Finally, I also tried other model types, like facebook’s prophet package or converting it to a regression problem and using weekday, week number, month and holiday days as features in a random forest.


how to classify text based on more than one column

In order to make the model take into account the two columns and distinguish whether a word is from the title or the article, you can generate the two sets of features independently and then concatenate the two vectors of features. This way the learning algorithm can assign different weights to a particular word depending on whether it comes from the title or the article.
However there is a disadvantage in doing that: depending on how many instances, how many words etc. there are, increasing the number of features might make the task harder for the model (for instance cause overfitting). It's probably worth testing the two options:

concatenate the text from the two columns then generate the features, i.e. no distinction between the columns but simpler job for the classifier
generate the features independently then concatenate the two sets of features (as above).


how to use predictions on a single value?

Building a model pipeline is the way to go once you are done experimenting with your model. Following is a toy example -
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline

#Generating random dataset
X, y = make_classification(random_state=0)

#Splitting test and tran sets
X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    random_state=0)

#Defining a pipeline with standardscaling transformer and then svc estimator
pipe = Pipeline([('scaler', StandardScaler()), ('svc', SVC())])

# The pipeline can be used as any other estimator
# and avoids leaking the test set into the train set
pipe.fit(X_train, y_train)

#During prediction on a test or on a fresh set of data, just use the pipe object to run the whole pipeline and get predictions.
pipe.score(X_test, y_test)

You can easily create a pipeline in Sklearn for almost every possible transform available in Sklearn. Check the user guide for more details.
Another example with PCA -
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC
from sklearn.decomposition import PCA
estimators = [('reduce_dim', PCA()), ('clf', SVC())]
pipe = Pipeline(estimators)
pipe

Pipeline(steps=[('reduce_dim', PCA()), ('clf', SVC())])
#Now you can use the fit method of the pipeline object to perform feature engineering and train model, while the scoring method for predictions

For Sklearn Pipelines, you can have either a single or multiple transformation paths (which are combined together with a FeatureUnion) and then it has to end with an estimator (only a single estimator is allowed)

While all this is powerful enough for most purposes, sometimes, there are custom transformations that you will need for your data, and those transformations will not be a part of the Sklearn API. In that case you can actually use the base classes of sklearn to defined your own transformer which can easily become a part of the pipeline that you define. More information here.


How to Normalize & Scale a Single Data Point

You need to normalise the input in the same way that the training data was normalised -- however, you don't need access to this training data during predictions of new data. If you have used a MinMaxScaler for example, then you can re-use this to transform your new data point:

scaler = MinMaxScaler()
X_normalised = scaler.fit_transform(X)

# do other stuff here like train & validate your model

# now we have a new test data point from somewhere, we scale
# it using the scaler we fitted on the training data
new_test_point_normalised = scaler.transform(new_test_point)

# now we can classify it with our model!

You can serialise the scaler to disk using pickle and re-load it when predicting new data points if necessary (i.e., the model training and testing happen in two different scripts).


When it comes to dealing with many disparate kinds of data, especially when the relationships between them are unclear, I would strongly recommend a technique based on decision trees, the most popular ones today to the best of my knowledge are random forest, and extremely randomized trees.

Both have implementations in sklearn, and they are pretty straightforward to use. At a very high level, the reason that a decision tree-based approach is advantageous for multiple disparate kinds of data is because decision trees are largely independent from the specific data they are dealing with, just so long as they are capable of understanding your representation.

You'll still have to fit your data into a feature vector of some kind, but based on your example that seems like a pretty straightforward task, and if you're willing to go a little deeper on your implementation you could certainly come up with a custom tree-splitting rule without actually having to change anything in the underlying algorithm. The original paper is a pretty decent place to start if you want to give that a shot.

If you want pseudo-structural data from your text data though, I might suggest looking into doc2vec, recently developed by Google. I don't think there are any good open-source implementations now, but it's a pretty straightforward improvement on the word2vec algorithm, which has implementations in at least C and python.

Hope that helps! Let me know if you've got any more questions.


In smoothing of n-gram model in NLP, why don't we consider start and end of sentence tokens?

It depends on the definition of vocabulary (V).
Most teaching examples only include words in the vocabulary for simplicity. Start and stop of sentences tags can also included in the vocabulary.
Vocabulary can also include punctuation, or stop words can be removed from the vocabulary.


What does this linear regression summary tells us?

The global test shows that the model is significant as the p value is very low which is p-value: 8.916e-13 in alpha level of 0.05. 98.64% of the variability of Minutes  is described by Units. The effect of Units is significant in predicting Minutes. For each one unit value increase in Units, Minutes increases by 15.509.


NN is automatic feature extractor, it will find columns that you suspect are valuable, if you encode it right

Dont worry about indicating specifically these columns. Worry about optimising your DNN architecture. Since you say you have variable input lengths, one optimisation method would be utilising dynamic sequence bucketing

There is no ultimate theory (without some serious assumptions being made) that says if only X is satisfied than you can be sure Y will beat all other methods, or that it will converge. Try a couple of different approaches in all stages (models, preprocessing etc) and see what works. Just make sure you make minimum viable experiment enviroment, i.e. representative train/test set where you can quickly iterate at test different approaches


Cause of periodic jumps in loss function

Looks like you may be feeding training data in a very specific way or weights and biases reset after a specific period of time, e.g. end of epoch?
I would start with checking that

training and validation sets consist of a desired class ratio
datapoints within mini batches are being shuffled
model parameters are being correctly updated during training


how to take CSV file input in list of tuples

Technically speaking, you do not have a CSV, you have a TSV. So I would take the time to split out each row accordingly and then build your tuple.

In the line:

data=[tuple(line) for line in csv.reader(f)] return (data)

You need to replace csv.reader to handle tsv instead.


Python : How to use Multinomial Logistic Regression using SKlearn

Put the training data into two numpy arrays:

import numpy as np

# data from columns A - D
Xtrain = np.array([[1,    20,   30,   1],
                   [2,    22,   12,   33],
                   [3,    45,   65,   77],
                   [12,   43,   55,   65],
                   [11,   25,   30,   1],
                   [22,   23,   19,   31],
                   [31,   41,   11,   70],
                   [1,    48,   23,   60]])

# data from column E
ytrain = np.array([1, 2, 3, 4, 1, 2, 3, 4])

Then train a logistic regression model:

from sklearn.linear_model import LogisticRegression

lr = LogisticRegression().fit(Xtrain, ytrain)

Make predictions (on the training data):

yhat = lr.predict(Xtrain)

=> results in "1, 4, 3, 4, 1, 2, 3, 4".. so it's got 7 right and 1 wrong.

Calculate accuracy:

from sklearn.metrics import accuracy_score

accuracy_score(ytrain, yhat)

=> results in 87.5% accuracy

To make predictions for new data, just create another numpy array containing your test data and call lr.predict on it.

You might also want to look into parameter tuning to improve your score. For example the LogisticRegression class has some parameters that control regularization - tuning them with methods found in sklearn.grid_search might improve your score.


Need recommandations of using timing variables for forecasting sales

Generally it works well to include some lagged variables.

For example:

Sales last week
Sales a month ago
Inventory a month ago
etc


I am not sure I can go as deep as you'd like, but I can give the basics.
The base types in R are C-structs. Taken from Hadley Wickham's Advanced R:

Base types
Underlying every R object is a C structure (or struct) that describes
how that object is stored in memory. The struct includes the contents
of the object, the information needed for memory management, and, most
importantly for this section, a type. This is the base type of an R
object. Base types are not really an object system because only the R
core team can create new types. As a result, new base types are added
very rarely: the most recent change, in 2011, added two exotic types
that you never see in R, but are useful for diagnosing memory problems
(NEWSXP and FREESXP). Prior to that, the last type added was a special
base type for S4 objects (S4SXP) in 2005.

One layer higher (or at least different in parallel to the C base types), R itself has a few Object Oriented Systems at work. Major data containers that you likely use, such as vectors, dataframe, methods are going to be of type S3. The newer major object system defines S4 objects. There is a great overview on Hadley's webpage
Things such as removing a column if it is set to NULL (as opposed to Panda's approach, of making each column take that value) I would guess are design choices. It is likely more common to drop a column than it is to fill it with null-values - in the context of 99% of the algorithms R packages the null values would simply be ignored anyway. So for convenience; not that DF.drop() in Pandas is a major inconvenience.
This behaviour is called an idiom in this blog. They also show that assigning NULL to a vector element also deletes that element. Additionally, NULL and NA behave differently: NA is a missing value, NULL is not!
IMO you should take the time to at least skim some documentation. With the help of search engines and control-F, you rarely need to drudge through pages of docs to find what you are looking for.


How to perform upsampling (and NOT interpolation) process theoretically modelled?

If I understood your question right, you want a mathematical expression for the I/O (input/output) relationship of a signal expander (name of the block that expands (upsamples) -without interpolation filering- an input signal $x[n]$)
Below is a block diagram of signal expander by a factor of $L$:
$$ x[n] \longrightarrow \boxed{ \uparrow L } \longrightarrow x_e[n] \tag{1}$$
An expression for the expanded sequence $x_e[n]$ can be written as :
$$ x_e[n] = \begin{cases} { x[\frac{n}{L}] ~~~,~~~ n=m\cdot L ~~~,~~~ m=...,-1,0,1,... \\ ~~~ 0 ~~~~~~,~~ ~ \text{otherwise} } \end{cases} \tag{2} $$
An identical expression is also the following :
$$x_e[n] = \sum_{k=-\infty}^{\infty} x[k] \delta[n-L\cdot k] \tag{3}$$
where $\delta[n]$ is a unit sample (discrete-time impulse).
With $L=3$, an input $x[n]=[1,2,3,4]$ becomes output $x_e[n] =[1,0,0,2,0,0,3,0,0,4,0,0]$.


What is the difference between GPT blocks and Transformer Decoder blocks?

GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model:

For GPT-2, this is clarified by the authors in the paper:

There have been several lines of research studying the effects of having the layer normalization before or after the attention. For instance the "sandwich transformer" tries to study different combinations.
For GPT-3, there are further modifications on top of GPT-2, also explained in the paper:


Best practical algorithm for sentence similarity

Cosine Similarity for Vector Space could be you answer.
Or you could calculate the eigenvector of each sentences. But the Problem is, what is similarity?
"This is a tree",
"This is not a tree"
If you want to check the semantic meaning of the sentence you will need a wordvector dataset. With the wordvector dataset you will able to check the relationship between words. Example: (King - Man + woman = Queen)
Siraj Raval has a good python notebook for creating wordvector datasets.


Ensembling vs clustering in machine learning

Short answer: Ensembling and clustering are completely unrelated techniques.

Ensembling: Combine the strengths of many diverse models. Ensembles generally do not involve training models on separate sets of data -- it's the models themselves that are different. Generally the more diverse the models the better. For example, an ensemble might comprise the following models: a support vector machine (SVM), a random forest, a gradient boosting model (GBM), logistic regression, k-nearest-neighbors, and a neural network. On Kaggle, I have also seen top competitors create ensembles comprising the same type of model (e.g., random forests) but with a variety of tuning parameters (e.g., one with 100 trees, another with 250 trees, etc). Clustering could be one of the models included in an ensemble. Ensembles are very effective (although extremely complex to operationalize) and can generally improve your error about 2-5%.

Clustering: Divide the data into mutually exclusive groups with the hope that your cluster-specific models can reduce error by specializing, except in the case of fuzzy clustering where the groups need not be mutually exclusive. At financial companies that build credit risk models, these cluster-specialized models are sometimes called "sub-models" in the sense that they predict the same target variable and therefore fit under an abstract/figurative parent model. This makes model risk reporting easier in the sense that it's simpler to refer to model 1 then model 1a, 1b, 1c, 1d. Clustering in this context is generally based on business knowledge and not a formal clustering technique such as k-means or other types of centroid-based clustering. For supervised learning tasks, formal clustering is generally considered a pretty poor technique because measuring the quality of the clusters is subjective. Your clusters can also vary dramatically depending on how you initialize the centers and how many clusters you specify. In practice, clustering is used much less frequently than ensembling.


In general, its not acceptable to drop an observation just because it is an outlier.

Sometimes outliers give us very important information about data.
Removing outliers is legitimate if:

it is obvious that's incorrect data
not change the results and assumptions.

How to handle outliers? Try other models or use data transformation.


That sounds like you're suffering from overfitting, probable duo the "curse of dimensionality", which means 2 things.

When trying to measure distance between two points in high dimensional space and trying to interpret this data, the difference between the longest and the shortest distance get's less meaningful.
Trying to train an algorithm with way less samples then features, like in your case, will cause that the algorithm will memorize your samples.

I see 3 options. Getting more data, reduce your feature set or try greater values for C. C will regularize the decision boundary.

More to read:

Curse of dimensionality
n >> p


What's the easiest way to quickly plot several features simultaneously for new data?

I believe seaborn's pairplot is what you are looking for.

Seaborn is a plotting library built on top of matplotlib for exactly the reason you want. To make quick aesthetically pleasing graphs for data analysis. Seaborn also work very well with pandas DataFrames.


Time series forecast for everyday for till a distant future

Have you tried the good old Fourier Transform?
If you can take the load of math, you can try the Wavelet Transform, but first I would recommend a simple Fourier Transform.

A Guide


How to Save Model that has a TensorFlow Probability Regularizer?

One solution I found was saving the model in HD5 format. That seems to bypass the serialization of the regularizer:
vae.save("./VAE.h5")
You can also try saving the weights only: vae.save_weights("./VAE-weights")


How do I correctly build model on given data to predict target parameter?

Train data includes latitude, longitude and zipcode. Output variable is location.

It is trivial to predict location if zip and lat,lon are known.  Try removing these attributes and see if that has any impact on validation score.


Welcome to the community.
About your problem, you say that your last column inhospital_mortality is binary so your goal is to classify, at a certain time with some info about them, patients likely to die in hospital if I am right.
Assuming this is your goal, I think you can try a classification approach without reshaping your dataset to a time series format, since you are not interested in forecasting a sequence (or a point in that sequence), you have independent buckets of x measurements per patient, without temporal connection in your training history.
You can use certain lag values (the -1, 2, 3... you already have) as informative features, and proceed with a binary classifier. Other option is, with this approach and per patient, calculate some statistical values like min, max, mean... of the 8 measurements for each type of patient indicator, while keeping the same number of rows but aggregating the attributes.
Of course you can also reshape it by having a unique date-time index with a value for each attribute type as you say, but you end up with less input features per row to predict your inhospital_mortality target value.
If you are interested in actually forecasting and eventually use an LSTM as you suggest, you might want to have a look at other answers about this type of problem like this


Learning the uncertainty of a ML algorithm

Take a look on Google trust score - https://github.com/google/TrustScore
Based on the following paper - https://papers.nips.cc/paper/9547-can-you-trust-your-models-uncertainty-evaluating-predictive-uncertainty-under-dataset-shift.pdf
A relatively naive but useful approach


I am not clear on how you convert your N rows into a single row. Do you concatenate all of them? This is an option but could become problematic over longer sequence lengths.

A straightforward method could be to simply average the features across time-steps. Although you lose causal information, this has been shown to work surprisingly well for word embeddings.

Another standard approach is to use hidden markov models. Very simply, a hidden markov model assumes a latent state space which generates an output (in this case your feature) at every time-step conditioned on the previous state.
https://en.wikipedia.org/wiki/Hidden_Markov_model

Today the practice is to use sequence neural models such as LSTMs and GRUs which can hold memory over longer time-steps.
https://en.wikipedia.org/wiki/Long_short-term_memory

I suggest your read up on these methods and select the one that suits your purpose :)


What is "data scaling" regarding StandardScaler()?

I will use k-Nearest Neighbor algorithm to explain why we must do scaling as a preprocessing step in most machine learning algorithms.

Let's say you are trying to predict if I transaction is fraudulent or not, that is, you have a classification problem and you only have two features: value of transaction and time of the day. Both variables have different magnitudes, while transactions can vary from 0 to 100000000 (It is just an example), and time of the day between 0 to 24 (let's use only hours).

So, while we are computing the nearest neighbor, using euclidean distance, we will do

distance_class = sqrt(
    (new_value_transaction - old_value_transaction)**2) +
    (new_time_of_day - old_time_of_day)**2)
)

Where old is the reference to our train data and new is related to a new transactions we want to predict the class.

So now you can see that transactions will have a huge impact, for example,

new_value_transaction = $100
new_time_of_day = 10

old_value_transaction = $150
new_time_of_day = 11

class_distance = sqrt(($50)**2) + (1)**2)

Now, you have no indication that transaction value is more important than time of the day, that is why we will scale our data.

Between the alternatives, we can have a lot of different, such as MinMaxScaler, StandardScaler, RobustScaler, etc. Each of them will treat the problem different. To be honest? Always try to use at least two of them to compare results.

You can read more about in the sklearn documentation: https://scikit-learn.org/stable/modules/preprocessing.html#preprocessing

I hope you got the feeling why we should use standardization techniques. Let me know if you have any further questions.

To complement, here is a visual explanation of what I explained above.

Credit: https://www.youtube.com/watch?v=d80UD99d4-M&list=PLpQWTe-45nxL3bhyAJMEs90KF_gZmuqtm&index=10

In the video they give a better explanation. Also, I highly recommend this course, the guys are amazing.


As you can't disclose much detail, I'm forced to be a bit generic in my answer. I hope it will be helpful nevertheless. First of all, I would only consider reducing the sequences before classification (be it by using the dot product or something else) if you can make sure that you don't lose information you need for classification afterwards. So this approach is only feasible if you have some insight into the nature of the classification. To give a simple example: if the class label is just the number of vectors in your sequence, you won't be very successfull in predicting the class label from the dot product.

Hence, I would take the full sequence as an input for classification, and impose a maximum on the sequence length you want to consider. You might do this by first finding the maximum sequence length m in your training set and then turning each sequence of 50-dimensional vectors into one vector of dimension 50*m, possibly with some missing values at the end if your sequence doesn't have maximum length. You will probably want to get rid of these missing values and you may want to simply replace them by zeros.

There's two roads you can go from here:
1.) You directly apply classification methods known to be suitable for high dimensions. Try something simple that doesn't need much tuning like naive Bayes. This way you can see whether this approach is feasible without losing too much time if it isn't.
2.) You try first to reduce the dimension and understand the nature of the classification better. You may want to use something like principal components analysis or analyse correlation/association between each vector component and the class label. If you're successful, you know how to properly reduce the dimension of your input before applying classification.

If you would like to follow any of these ideas, please keep in mind that the concrete details of your data and the classification may render any of the ideas proposed above infeasible. So please be careful to check against any details you know but can't post here before trying to make sure you're not wasting your time.


Pretty late but I'm surprised this wasn't answered more.  "Cosine similarity" is a great technique to try, though simply letting users search with a hard string and then ranking by popularity isn't so bad (e.g. "dutch" brings up everything with "dutch" in it, though I would discard mid-word matches, so "ball" wouldn't return "football", but would return "ball room dancing").

I'd say in any approach a main issue will be deduplicating previous (non-standardized) skills input by users that weren't quite standardized.  You could also try replacing the candidate skills with versions that have different synonyms substituted at search time, e.g. "soccer coaching" might be stored also as "football coaching" if most of your content is from Europeans.

Sometimes extreme accuracy might not be the best goal, though... You may want to encourage users to explore new skills that they never knew existed!  Not sure what your needs are...

Whatever you settle on, it might be worth building a semi-hand-crafted test set of queries and relevant results so that you can see if the performance is terrible (Google precision and recall in the context of search results).


The infinity is due to divide by zero.
Just replace any zero value with a very small value.
This question is very common , if you search for it you will find many questions similar to it .
Remember that kl is not a metric


Spliting keras model into multiple GPU's

One graph - multiple GPUs

In order to specify specific GPU for layers, this post can be helpful. Just specify for layer the GPU device.

Multiple copies of graph - multi_gpu_model()

There is a multi_gpu_model() function in Keras which will make your training be distributed on multiple GPUs on one machine.

But, as it is stated in the documentation, this approach copies the graph on multiple GPUs and splits the batches to those multiple GPUs and later fuses them.

When using this multi_gpu_model() be aware that you might encounter a problem with saving such model, but it can be easily solved by saving the original model as stated is this answer.


How is the hidden state of a GRU initialized

If you think intuitively what h_t (that is h_t-1) shoud represent is, very freely speaking, amount of rembering from the previous steps. Since you are just starting, and want to remember everything, very intuitive thing is to set the value at 1. Than you can see in the equations that the mutiplication with h_t-1 makes no difference in the beginning.


How do ML model retain/store the learning(s)?

After generating a predictive model by fitting the data with model.fit(X,y) , "the learnings" are simply coefficients (weights ) that are multiplied by the features ( X ) to give you a certain y. In other words, in the end, what you get is a function that for an input X returns a y, you can store the model on the disk using a joblib or pickle and load it in another environment. machine learning? more like coefficients learning.

Note : This could be edited in case my answer is lacking more details


how can I solve label shape problem in tensorflow when using one-hot encoding?

Since your data are already in a one-hot encoding, you can use tf.nn.softmax_cross_entropy_with_logits(), which expects an input of shape [batch_size, num_classes] for the labels.

(The tf.nn.sparse_softmax_cross_entropy_with_logits() op expects the labels as a batch of integers, where each integer corresponds to the class ID for each example.)


Stop words are usually thought of as "the most common words in a language". However, other definitions based on different tasks are possible.

It clearly makes sense to consider 'not' as a stop word if your task is based on word frequencies (e.g. tf–idf analysis for document classification).

If you're concerned with the context (e.g. sentiment analysis) of the text it might make sense to treat negation words differently. Negation changes the so-called valence of a text. This needs to be treated carefully and is usually not trivial. One example would be the Twitter negation corpus. An explanation of the approach is given in this paper.


Consider the class to be the variable that you are sampling.

Your "reservoir sample" should still be as good as uniformly drawn from your data.

Yes, there may be fluctuations, in particular if you have small samples. If you sample a single observation, the class distribution in that sample will be 100% of one class, there is no way around that. ;-)


What does it mean to "warm-start" XGBoost?

Why not just keep training your old model?
just pass the previous xgb model (path + filename) as an argument in the train function.

bst = xgb.train(param0, dtrain, num_round, evals=[(dtrain, "training")], xgb_model=prev_xgb_model)

xgb_model=prev_xgb_model


Why does the votingClassifier in sklearn.ensemble gives higher accuracy than each of the classifiers that make it? and should i depend on it?

Basically I would say that your individual classifiers might be overfitting the training data. Ensemble voting is known to avoid overfitting, so it does not seem unlikely that the ensemble classifier behaves better than each classifier individually.


Scikit-learn decision tree in production

Persisting the model parameters is the only out of the box solution in sklearn. You should ensure that:

Your model is reproducible from data
The versions of dependencies used (incl sklearn) is locked
Your QA system checks correctness of your model

If you have these things down then the possible changes in model persistence in sklearn is a non-issue. If it happens, you just update sklearn in a controlled manner, retrain and deploy as usual.
And you avoid a whole range of other problems, for instance bugs in sklearn/numpy/whatever catching you by suprise. Or new data needs new models trained and deployed.

You can of course build your own persistence mechanism but you then need to maintain the compatibility. Better spend the effort one the above.


Most important part of feature standardization and how is standardization affected by sparsity?

No, you are misinterpreting his comments.  If you have data that has some outliers in it then the outliers will extend beyond 3 standard deviations.  Then if you standardize the data some will extend beyond the [-3,3] region.

He is simply saying that you need to remove your outliers so the outliers don't reap havoc on your stochastic gradient descent algorithm. He is NOT saying that you need to use some weird scaling algorithm.

You should standardize your data by subtracting the mean and dividing by the standard deviation, and then remove any points that extend beyond [-3,3], which are the outliers.

In stochastic gradient descent, the presence of outliers could increase the instability of the minimization and make it thrash around excessively, so its best to remove them.

If the sparseness of the data prevents removal then... Do you need to use stochastic gradient descent, or can you just use gradient descent?  Gradient descent (GD) might help to alleviate some of the problems relating to convergence.  Finally, if GD is having trouble converging, you could always do an direct solve (e.g. direct matrix inversion) rather than an iterative solve.

Hope this helps!


In the case of 1 note at a time maximum you could use categorical_crossentropy as loss and add a class for when no note is played. This is a log loss on a last softmax layer. This would turn it into a 37 one-hot-encoding representation.

In the case of having the possibility of multiple notes I would keep the 36 (or 88 for the pian) dimensional representation and have a sigmoid activation on each of the notes at the end. Then sum the binary log loss of each of the nodes and use that as the loss. Then you can treshold the nodes individually at prediction time to see if a note is on or off.


What is the correct way to compute lift in lift charts

Lift is computed by comparing performance with a random selection model. I'll explain with your example below,

assume that we didn't have any statistical/ML model for ranking/scoring the respondents.
In that case assume we did a random ordering of respondents.
A decile (10% of total population) is expected to have 10% of the respondents. In your case, there should've been (approximately) 488 respondents in 2500 cases.
But after ordering the cases by score, you are seeing 44.71% of the cases in first decile against expected 10% (in random/no model case). This gives the gain of 44.71/10 = 4.471.
For next decile, cumulatively you have covered 20% of the cases. You'd expect a random/no model scenario covers 20% of the respondents. But using scores, we covered 80% of them. That gives a cumulative lift of 80/20 = 4.


How to extract numerical data from a matplotlib scatter plot in python?

You could also show the values for each point by using matplotlibs annotate command:

import numpy as np
import matplotlib.pyplot as plt

x = np.arange(5)
y = np.random.randint(10, size=5)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.set_ylim(0,10)
plt.plot(x,y)
for i,j in zip(x,y):
    # xytext and textcoords are used to offset the labels
    ax.annotate("({},{})".format(i, j), xy=(i, j), xytext=(5, 5), textcoords='offset')
plt.show()


Should a model be re-trained if new observations are available?

When new observations are available, there are three ways to retrain your model:

Online: each time a new observation is available, you use this single data point to further train your model (e.g. load your current model and further train it by doing backpropagation with that single observation). With this method, your model learns in a sequential manner and sort of adapts locally to your data in that it will be more influenced by the recent observations than by older observations. This might be useful in situations where your model needs to dynamically adapt to new patterns in data. It is also useful when you are dealing with extremely large data sets for which training on all of it at once is impossible.
Offline: you add the new observations to your already existing data set and entirely retrain your model on this new, bigger data set. This generally leads to a better global approximation of the target function and is very popular if you have a fixed data set, or if you don't have new observations to often. However it is unpractical for large data sets.
Batch/mini batch: this is sort of a middle ground approach. With batch, you wait until you have a batch of $n$ new observations and then train your already existing model on this whole batch. It is not offline as you are not adding this batch to your preexisting data set and then retraining your model on it and it is not online as your are training your model on $n$ observations at once and not just a single one. So it's a bit of both :) Mini batch is exactly the same except that the batch size is smaller so it tends towards online learning. Actually online learning is just batch with batch size 1 and offline is batch with batch size the size of the whole data set.

Most models today will use batch/mini batch and the choice for the size of the batch depends on your application and model. Choosing the right size batch is equivalent to choosing the right frequency with which to re-train your model. If your new observation have a low variance with your existing data, I'd suggest larger batches (256-512 maybe) and if on the contrary new observations tend to vary greatly with your existing data, use small batches (8-256). At the end of the day, batch size is kind of like another hyper-parameter which you need to tune and which is specific to your data


In general, you cannot reorder like that. The example you give is a very special case that only works because softmax is based on the exponential function, which is the inverse function of the natural logarithm.


Syntax of the inputs for LightGBM binary build (without python API)

The first column in the input file is considered to be the target variable. I found this by trial and error.

EDIT

There is a parameter called label_column which can take an integer value from 0 to N, to specify which column is the target variable. (source: https://lightgbm.readthedocs.io/en/latest/Parameters.html)


Is it possible to pass dataframes between R and Python using Arrow with zero-copy?

Not currently, though hopefully very soon. https://issues.apache.org/jira/browse/ARROW-3750 is in progress and hopefully will resolve in the coming weeks.


Jupyter notebook - Converted Python dataset to R Dataframe. Got stuck in converting back from R to Python Dataframe

I find the answer for my own question. I hope this helps to others. Thanks

https://pandas.pydata.org/pandas-docs/stable/r_interface.html


Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?

The difference between these methods is the assumptions they make about the task.  Multi-class classification assumes that each document has exactly one label.  So a document can either be about sports or weather, not both.
Multi-label classification allows a document to have any combination of labels, including none.  So a document can be about only sports, only weather, sports AND weather, or neither.

You could train a multi-label classifier with data where each document has exactly one label, but there is no guarantee that the predictions made at test time will have only one label.  Also you are forcing the classifier to do more work (and potentially make more errors) by considering more possible labelings than it needs to.
Therefore, if the multi-class assumption makes sense for your problem, you are better off with a multi-class classifier.

The method that you describe for training individual binary classifiers corresponds to multi-label classification.  The binary classifiers that you use could each be trained from one-class data or two-class data.  However, this is only one of the many ways to do multi-label classification (see the wikipedia page above for more).

Unfortunately, the problem that you describe does not cleanly fit into either multi-class or multi-label classification, since you want each document to have at most one label.


Tensorflow oscillating Test and Train Accuracy?

From your results I see that the network is working good: the loss function is reducing, that means that the calculated results are approaching the expected ones.
But in facts the accuracy is not, so there should be some error in the way you calculate it.

The fact that the loss function start to oscillate at some point (~after 20000 iterations) is expected, and that is the reason why usually one lower the learning rate after some epochs, or use some other technique of annealing.


PCA reduces dimensionality. It does not change the number of observations you have. Nor does it change the order of the data. The n-th observation in your original dataset will still be the n-th observation post-PCA.
Choosing the number of components in PCA and choosing the number of clusters in K-Means are independent of each other. Both K-Means and PCA seek to "simplify/summarize" the data, but their mechanisms are deeply different. PCA looks to find a low-dimensional representation of the observation that explains a good fraction of the variance. K-Means looks to find homogeneous subgroups among the observations.

For PCA, the optimal number of components is determined visually through the scree plot or mathematically using Kaiser's criterion (drop all components with eigenvalue <1). From my experience the two don't always give the same results, but the difference is negligible.

Screeplot (source)

It does not make much sense to go beyond the 4th component. Law of diminishing marginal returns.

For K-Means: Since increasing the number of clusters will always reduce the distance from centroid to data points, increasing K will always decrease this metric, to the extreme of reaching zero when K is the same as the number of data points. If we plot the average within cluster distance to centroids vs number of clusters, we will find the "elbow" to be a good place to stop.

source


Is there an appropriate use of adjusting class weights for a balanced dataset?

Class Weight can be important even for balanced data if, for example, some class is more significant than others, so loss wrt this class should count more.
One can even think of class weights as unique extra hyper-parameters with their own effect on the outcome (either positive or negative) and treat them as such without interpretation.
Related: How does class_weight work in Decision Tree?


Since the points in each cluster are fairly spread out along the direction of the separating line, the average distance within a cluster is probably not that much smaller (as you might hope) than the average distance to points in the other cluster. A small positive value could be plausible. (Maybe look at the distribution of the points along the second principal component. A big spread could be the problem.)


Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

I am working on a similar problem where we are classifying 2 million products into about 1000 categories.  I converted the product descriptions using tf-idf vectorization, and then used SVM to run the supervised classification.  There is a lot of optimization you can do with the sklearn package in Python for natural language processing.  Additionally I struggled because I wanted to include other features beyond just text descriptions.  There is a union method in sklearn for accomplishing that.  The final model was about 90% accurate in classifying, but out of a 1000 classes there were quite a few with low accuracy rates from the test set.  This is likely due to small training sample for those classes.  So, just make sure there are enough training samples for each class when you build your model.


Why do we use +1 and -1 for marginal decision boundaries in SVM

It's important for the optimization formulation of the SVM that $y_i=\{-1,1\}$ which is why it makes sense to also output $y=\{-1,1\}$. If we look at the soft-margin linear SVM we want to minimize:

$\left[\frac{1}{n}\sum_{i=1}^n\max{(0,1-y_i(w\cdot x_i+b))}\right]+\lambda\| w\| ^2$

The $y_i$ is either +1 or -1 which flips the hyperplane in the soft-margin definition of the problem.


How to combine two different embeddings in the best way possible?

I would say that the best thing to do here is to concatenate both embeddings and use the concatenated vector as an input for your binary classification model without using the norm -> you loose way too much information, i.e.:
final_layer = concatenate [ book_1_attention, book_2_attention]

I reckon that concatenating without norm will increase the dimension (but that's potentially a problem for which you have solutions - such as regularisation)


Average of importance gain for a categorical variable

I think you are looking for information gain.

The way you would compute it for 1 variable is:

Let's say that you label variable is binary.

1) Compute percentages of label per category, for example you have three categories: "US", "UK", "Ger", if there are 5 labels that are marked 1, and 3 of the area associated with "UK", 2 with the "US" and 0 with "Ger", your percentages would be: [3/5, 2/5, 0/5].

2) Do step 1) for every label and aggregate

3) Calculate entropy for every percentage

4) Information Gain = Entropy of aggregated percentage - sum of weighted percentages. Weight is just number of instances in a label / total number. For example if we had labels = [1,1,0,0,1,1,0,0,1], entropy of percentages of label 1 would be weighted by 5/9 and entropy of percentages of label 0 would be weighted by 4/9.

Then you compute IG for every variable, and compare!

As a side note, if you are working with a lot of categorical variables, you might want to look into LightGBM or CatBoost, these algorithms allow you to specify categorical variables without creating one hot encoded vectors, and they provide feature importance on these variables.


What representation of data should I choose, pandas, numpy or tensors to train neural network model

It's hard to say for sure without knowing more about the size of your dataset, the architecture of your model, and the libraries you are using.
Changing from pandas.Dataframe to numpy.array is very unlikely to make your training faster.  Pandas dataframes are already backed by numpy arrays.
Depending on what neural network library you are using and what hardware you have available, changing from Dataframes to tensors may help.  Libraries like Tensorflow and PyTorch allow you to move computation to a GPU/TPU, which may speed up training.
The benefit of using tensor data structures from DL libraries is that they can be moved to a GPU/TPU.  They aren't inherently faster.  So if you don't have a GPU, or if you are using scikit-learn to train the model, then there's no point switching from pandas to tensors


How to simulate customer walk-ins for a given period in a fast food chain in R

You need either an initial data set or a large set of constraints to "fake" a data set.  if you have an initial data set you could make somewhat valid assumptions regarding traffic flow, purchasing behaviors and the likes.

If not, you are going to need to research these different metrics from existing companies that are like what you want data for and then I assume put together a little program with those metrics as constraints and simulate your data.  Once you have those constraints you could use simple averaging or weighted options for developing a constraint per variable.

Make sure they are semi accurate.  For example, if you were modeling company growth for a "really good idea" research growth trends for like companies and for companies you want to emulate and then scale BACK from there.

You can also always purchase a dataset:
http://datamarket.azure.com/browse/data


Unbalanced dataset on image classification, is it better to lose samples and balance it?

Instead of downsampling the dataset, you should try using the Upsampling technique. Sometime downsampling leads to loss of data. Use augmentation techniques to increase the size of your dataset.


Graphs for representing performance of machine learning classifiers

The basics are:

Precision-Recall Curve
ROC Curve

Then, you can plot other features of the model to get insights about generalization, learning process, overfitting/underfitting, like accuracy vs epochs, loss versus epochs, to name a few.


transfer learning with sentiment analysis?

I can not fully answer your questions, but would like to offer a couple of my thoughts here:
1) Transfer learning for sentiment analysis can be hard given that knowledge learned from one topic may not be not broad or general enough to perform well on the target or downstream tasks. For example, I have recently trained a neural network along with Word2Vec embedding using Twitter airline customer review data and get an prediction accuracy of 77%. However, when I use the same Word2Vec and neural network to classify some general customer review data, I got an prediction accuracy of only 35%.

2) Transfer learning in natural language processing is a hot topic and many researchers are working on it recently. 2018 sees some breakthroughs in transfer learning, i.e., Google Universal Sentence Encoder, BERT algorithm etc. I can not give you a comprehensive list here since I'm also learning. I would suggest you dive into some blog articles or even the original research articles to get a better understanding.

May it helps.


Recommender Model for Human Action in Income Protection

It looks like the inverse reinforcement learning problem defined by Stuart Russell as

Given

measurements of an agent’s behaviour over time, in a variety
of circumstances.
measurements of the sensory inputs to that agent;
a model of the physical environment (including the agent’s
body).

Determine the reward function that the agent is optimizing.

It is particularly difficult in your case because you only have observations of a tiny subset of the actual state of a customer and the environment.
I don't think it is solvable without being able to actively explore the actions or without really strong assumptions or a model of the dynamics.
If you can assume that the historical supervisors did their best to help their clients to return to work, your task simplifies to: given the historical actions up to a given point, predict the next historical action.
Even with this simpler problem, you still have to be careful with conditioning, the possible non-stationarity in time, choosing a good model (perhaps similar to what you tried), etc.


How is a splitting point chosen for continuous variables in decision trees?

In order to come up with a split point, the values are sorted, and the mid-points between adjacent values are evaluated in terms of some metric, usually information gain or gini impurity. For your example, lets say we have four examples and the values of the age variable are $(20, 29, 40, 50)$. The midpoints between the values $(24.5, 34.5, 45)$ are evaluated, and whichever split gives the best information gain (or whatever metric you're using) on the training data is used.

You can save some computation time by only checking split points that lie between examples of different classes, because only these splits can be optimal for information gain.


What is the difference between Concatenate() and concatenate() layers in Keras / TensorFlow 2.0?

Keras provides 2 kinds of API i.e. Sequential and Functional. And because of this 2 kind of API, the difference is there in them.

Concatenate is used when you are using Sequential API
concatenate is used when you are using Functional API

Just look into the following documentation:
https://keras.io/layers/merge/


Frequency delta is not necessary a sound method. A better approach would be to use a statistical test for independence, like chi square independence test which can tell you if two categorical samples does not come from the same distribution. A similar construction can be done with G statistic. See more on wikipedia.

One problem you will certainly encounter is that some categorical distribution for ngrams have a large number of categories and the test is heavily applicable. To avoid that you can group the frequencies which are very small in both distributions into a single category: other. That way you limit the number of categories but comparison remains significant.

Simply comparing differences is not enough since you do not have some threshold to use which have also a meaning, while a statistical test produces p-values which are probabilities in the end.

On the other hand, you should consider the following issue with both approaches (the one you proposed and the one I presented you): they test the individual contribution of one feature, independent of others. As such, they will not catch situations where two features have insignificant contribution, but together they have a significant one.


Does Batch Normalization make sense for a ReLU activation function?

That is known a problem with the ReLU activation functions. It is often called a "dying ReLU". Given an input over the zero boundary, the unit is now almost always closed. A closed ReLU cannot update its input parameters, a dead ReLU stays dead.

The solution is to use variants of ReLU for the activation function such as Leaky ReLU, Noisy ReLUs, or ELUs.


I personally use NetworkX and igraph where the first one is for programming in Python and the second one has interfaces for Python, R and C.

If you are not into programming much, Gephi is what you are looking for!


BertPunc (punctuation restoration with BERT)

I am also confused by the method of BertPunc. Actually I can't believe it really works. Beacause the method use BertLanguageModel to do the job, and LanguseModel's main capabiltiy is to predict the next word of a word. So also use your example,  BertPunc inputs 8 words, the output of bert should be 8 next words of the inputs. But BertPunc added a Linear layer to only get 1 output value, which is to predict punctuation.

But I think we should use bert the way like the graph below:


Interpretation of scikit-learn one class svm scores

The sample_scores values, along with a cutoff threshold value, are used to determine whether a value is an outlier or not. You should be careful if you try to compare these sample_score values to see which values may be more anomalous than others. If you are looking to eliminate outliers solely to eliminate them from your data-set, using this value alone on a outlier would not be reason enough to eliminate it.


To answer you first question about Non linear regression:

I believe your problem of choosing mapping function for non linear regression can be solved by using Support Vector Machines.

SVMs can learn non linear mapping functions in a kernel-induced feature space.
What this means is in svms , the basic idea is to map the input data X into some high dimensional feature space f using a non linear mapping (kernel) and then doing linear regression in this feature space.

To learn more about non-linear regression and kernels, you can read this.

Secondly, Regularization is a technique that is used to solve over-fitting problem. This usually happens when you use a very dense model for your training set or you train the model for far too many steps. In this case, while the accuracy on your train set is high,but it performs very poorly in case of unseen data. Hence when you add regularization, it helps reduce the cost function.

Regularization is of two types, L1 and L2. The difference lies in the power of weight-coefficients.These should be enough for your SVM based models.

To reduce overfitting induced high cost, you can also use BatchNormalization and Dropout algorithms.

Hope this helps :)


Preprocess image data to classify objects based on shape

Well, you are trying so describe the image by predominant line directions and that really sounds fair but a bit of too much work. If you want to you can try a bit of data augmentation to increase your numbers of samples (Mainly affine transformations)
Other possible useful feature would be descriptors such as:

SIFT
SURF
FAST
BRIEF

You could try to describe those shapes using other geometrical functions such as skewness (fill the inside of the shape, consider the eigenvalues to define the major axis directions on PCA)
Finally, bottles in the sample image look similar in shape and differ mainly in size, if your model uses resize it might become scale invariant and then your method will decrease too much of your shape information and give bad results.
------------------------------------------------------------------------------------
Answering the comment bellow:

This is why i tried to do it with my approach of cropping out each bottle and split them into a 2d array of tiles (e.g. 50x20)

This inserts a scale invariance to your model, that is good for processes where image-size vary either because of distance or camera resolution but: This throws alway scale information, that might not be helpful when the main discriminant on your image is scale-related.


I think that the issue depends on what you'd expect the model to learn:

If the model is supposed to "know" the users it has seen during training, i.e. exploit the user id in order to infer particular choices for a specific user, then I don't see the point in adding this kind of frequency feature: the model already "knows" what choices this user tends to make, and it's supposed to know it only from the training set. Disadvantage: for any new/unique user not seen during training, the model probably has to fall back to a generic prediction.
If the user id is not used (removed from the features), then I think it makes sense to add the frequency features. I would see this mostly as a matter of feature engineering: let's assume that there are 3 features which represent how many times so far this user has chosen option A, B and C. The counting is made from any previous data available for this user. Of course, this is assuming that the same features can be calculated at testing time (equivalently, in a production environment).

So in the second option the features exist before any question about what is the training set and what is the test set. However they introduce a potential risk of data leakage, so what I would do is to separate the training set and test set based on the users, i.e. make sure that users in the training set and test set don't overlap.


You can obtain one negative example by taking one of the 3000 customer records and pairing it with any user record that is known not to match.  In this way, you can obtain $3000$ positives and $3000 \times 5999$ negatives.  You could then train a boolean classifier on this entire training set.  This might work better than using one-class classification on just the positives.

Even better might be to use techniques for learning to rank.  If $c$ is a customer record that is known to match a user record $u$, and $u'$ is any other user record (which $c$ doesn't match), then you want your classifier to rank the pair $(c,u)$ higher than $(c,u')$.  In this way you can obtain $3000 \times 5999$ such ranking-pairs, and try to train a classifier to learn to rank, then use that to find the best match for each of the 1000 customer records.


TimeDistributed with different input / output sequence length

I have looked for an answer to this question for quite a while, and couldn't really find a clear yes/no answer.

From what I understand, TimeDistributed really only works for similar input/output lengths.
According to what Jason Brownlee writes in the comments section of his TimeDistributed tutorial, he would treat the different input/output lengths as a seq2seq/encoder-decoder problem (which he describes here).

If you find a better answer, or find a way to make TimeDistributed work with different output sizes, please let me know.

*Update (copying my comment from below, with better formatting):

Actually, I have found a different solution which does use TimeDistributed: You can trim the output sequence you get from TimeDistributed using Cropping1D layer. So that my model now looks like this:

model.add(LSTM(hidden_size, return_sequences=True, input_shape=(num_input_timepoints, 1) ,dropout=0.05)
model.add(TimeDistributed(Dense(1)))
model.add(Cropping1D(cropping=(num_input_timepoints - num_output_timepoints,0))) # cropping the end

Which, in your case, would trim the first 97 time points and would leave you with the last 3. That would fit the shape of your target data.


Using already trained classifiers in Orange

A scheme like this should work. Namely, you pass training data directly into the the learner widget and pass its trained Model into Predictions widget along with labeled test data.